[camel-main] Fixes #9280. Add filter.* and modality=media to langchain4j-ingest by JiriOndrusek · Pull Request #9290 · apache/camel-quarkus

JiriOndrusek · 2026-10-07T13:57:11Z

Builds on #9282: only the last commit is new.

filter.* options go to the sink Kamelet, which enforces them. With a parser, the id patterns are also checked before the parse.
modality=media (optional content-type) embeds each document whole, as one vector; the builder gets modality() and contentType(). Conflicting settings fail the build.
Unlike the issue's first plan, the raw-size guard before the parse stays in the extension: it reads at most max + 1 bytes, while the cap of Fix #3086: Parser actions: add maxDocumentSize, checked before the parse camel-kamelets#3087 reads the whole stream. That cap is pinned off.

Not mergeable yet: needs #9282 and the camel-kamelets 4.23.0 release with apache/camel-kamelets#3085.

Tested locally: deployment tests, JVM and native integration tests, with camel-kamelets built from apache/camel-kamelets#3085 and apache/camel-kamelets#3087.

Claude Code on behalf of @JiriOndrusek. AI-generated; please verify before merging.

🤖 Generated with Claude Code

jamesnetherton

Reviewed locally (latest commit only; the first commit is from #9282). A few comments inline. The most important is the first: the raw-size guard is skipped for media pipelines.

jamesnetherton · 2026-10-07T14:34:30Z

+     */
+    private static ProcessorDefinition<?> parseSteps(ProcessorDefinition<?> route, String name, String parser,
+            int maxDocumentSize, boolean directory, IdempotentRepository register, String includeId, String excludeId) {
+        if (parser == null) {


With modality=media there is no parser, so parseSteps returns here and the bounded raw-size read (max + 1 bytes, not trusting CamelFileLength) never runs for media pipelines, including consumer-fed ones (source.uri / Source.endpoint).

The component's own check only trusts the CamelFileLength header before calling getMandatoryBody(byte[]). So a large body sent with a forged CamelFileLength gets past max-document-size and is loaded into the heap in full. Should the size guard also apply when media is true?

Partly true: an oversized media payload is still refused, because MediaIngestService re-checks bytes.length. But that happens only after the whole body is read, and a forged CamelFileLength passes the up-front check.

maxDocumentSize is the component's option, so the fix belongs there rather than in a media copy of the guard here: CAMEL-25463. With it, a body not yet in memory will be read only one byte (media) or one character (text) past the cap, then refused. CamelFileLength will still reject early, but it will no longer decide how much is read. That will cover every user of the component, not only Camel Quarkus, and keep another copy of the size logic out of the extension.

I'll add a test here to verify it, so this PR will wait for the fix. That's no extra delay: the fix targets Camel 4.23.0, which this PR already waits for.

Why that will be enough:

A directory pipeline is already safe: the file consumer's CamelFileLength is checked before the file is read.

On a consumer-fed pipeline, a guard in the extension could not do more. Stream caching (on by default, in memory) reads a stream whole before any of our steps run. That needs a limit at the consumer (e.g. quarkus.http.limits.max-body-size) or spooling, as the component docs in the fix will say.

The extension keeps its own guard only for parser pipelines, since the parse runs before the sink.

jamesnetherton · 2026-10-07T14:34:30Z

+    }
+
+    /** Comma-separated id patterns, parsed as the component parses them; null when none remain. */
+    private static String[] patterns(String value) {


patterns() and the AntPathMatcher check before parsing re-implement the component's private parsePatterns/idAccepted. If the component's matching changes (case, a leading /, splitting, where the id comes from), the advisory filter and the authoritative one could disagree. Worth a comment linking the two, or a test that checks they agree?

True. I added a comment pointing to LangChain4jIngestProducer.parsePatterns/idAccepted ("keep them in step"). The component stays authoritative; the copy only spares the parse. Exposing them from the component would remove the copy, and I can raise that in Camel.

jamesnetherton · 2026-10-07T14:34:30Z

            }
+
+            @Override
+            public FilterRunTimeConfig filter() {


filter() returns null, so this PipelineRunTimeConfig is only partly implemented. Any later code that reads runtime.filter() will hit an NPE, and only for @Ingest pipelines. Could it return an empty FilterRunTimeConfig, or delegate to the configured pipeline's filter?

Agreed. The builder view now delegates to the configured filter.*, or returns an empty filter when there is none. IngestRoutes reads runtime.filter() for both declaration styles.

jamesnetherton · 2026-10-07T14:34:30Z

+        // filter.* comes from configuration for both declaration styles, the builder having no
+        // filter API, and the component enforces it
+        IngestRunTimeConfig.PipelineRunTimeConfig configured = runTimeConfig.pipelines().get(name);
+        String includeId = configured == null ? "" : configured.filter().includeId().orElse("");


filter.* adds the pipeline name to the runtime map. A filter property for a misspelled @Ingest name (e.g. datasheet.filter.exclude-id when the pipeline is datasheets) creates a phantom configured pipeline. Startup then fails with a missing source.directory error rather than "unknown pipeline", and the real pipeline runs without the filter. Could a name that only has filter.* set be detected and reported?

Partly. The misleading error is real, but the start fails, so the real pipeline doesn't run without the filter.
The error now names the Java-declared pipelines: Ingestion pipeline 'datasheet' has no source.directory, and no @Ingest method declares it (declared in Java: [datasheets]). Fix the name, or set ...source.directory. Test: IngestBuilderFilterTypoTest.

jamesnetherton · 2026-10-07T14:34:30Z

+            if (configured.filter().minDocumentSize() != 0) {
+                sink.put("minDocumentSize", String.valueOf(configured.filter().minDocumentSize()));
+            }
+            configured.filter().documentFilter().ifPresent(bean -> sink.put("documentFilter", "#bean:" + bean));


When filter.document-filter is unset it isn't passed, so an application-wide camel.kamelet.langchain4j-ingest-sink.documentFilter still applies to every pipeline. The docs do say not to set those properties, but binding a neutral always-true Predicate as the default would pin the option, as the PR intends.

Done. An unset filter.document-filter now goes as an always-true predicate. The IT poison set gains camel.kamelet.langchain4j-ingest-sink.documentFilter=#bean:rejectAll. documentSplitter and idempotentRepository stay unpinned, since no neutral bean exists for them.

jamesnetherton · 2026-10-07T14:34:30Z

+     * document id's file extension. The twin of the {@code content-type} configuration property.
+     */
+    public IngestPipeline contentType(String contentType) {
+        if (contentType == null || contentType.isBlank()) {


This only rejects blank values. The component drops everything after ; and treats an empty result as unset, so contentType("; codecs=opus") passes here and then silently falls back to the file extension. Could this apply the same normalization before validating?

Fixed for both paths: a value with nothing before ; is rejected by IngestPipeline.contentType() and, for content-type, at build time. Tests extended.

jamesnetherton · 2026-10-07T14:34:30Z

+        sink.put("maxSegmentSize", String.valueOf(maxSegmentSize));
+        sink.put("maxOverlapSize", String.valueOf(maxOverlapSize));
+        sink.put("embeddingBatchSize", String.valueOf(embeddingBatchSize));
+        sink.put("modality", modality);


The modality is forwarded exactly as typed ("Media", "MEDIA", both used in the tests), so this depends on the component's IngestModality enum conversion ignoring case. modality.toLowerCase(Locale.ROOT) here would remove that dependency.

Done: it is lower-cased before it goes to the Kamelet.

jamesnetherton · 2026-10-07T14:34:30Z

+----
+
+The MIME type handed to the model comes from `content-type` or, when unset, from the file extension through Camel's MIME table (wav, mp3, flac, ogg, opus, m4a, aac and aiff for audio; png, jpg, gif and webp for images; mp4, mov and webm for video; pdf); a file whose type cannot be determined fails the exchange, and `content-type` without `modality=media` is rejected like the parser and splitter conflicts.
+On a directory pipeline such a file — `notes.txt`, `Thumbs.db` — fails again on every poll, so restrict the ids with `filter.include-id`, as above: it is checked before the media type.


On a directory pipeline, a file whose media type can't be determined fails on every poll, for ever, unless filter.include-id is set. Since retrying can never succeed, could it be committed in the register as filtered/rejected (like an empty parse is) instead of only being documented?

True: the rollback releases the file's key. I'd keep it documented for now:

The extension can't tell this failure from the others without copying the component's MIME table (MediaTypes is package-private).

An oversized file is retried the same way on purpose (Langchain4jIngestTikaTest).

With a persistent register, a committed key would keep the file skipped even after a config fix.

Answering filtered for an untypable id fits better in the component. I can open a Camel issue.

jiriOndrusek-agent · 2026-10-08T12:35:55Z

Thanks for the review, @jamesnetherton. The last commit addresses it. I kept it separate for an easier re-review and will squash it before merge.

2–7 are fixed in the extension; details are in the threads.
1 belongs to the component: CAMEL-25463 (CAMEL-25463: camel-langchain4j-ingest - read a body only up to maxDocumentSize camel#27569) makes it read a body only up to maxDocumentSize. This PR only adds a test for it: Langchain4jIngestAudioTest.oversizedClipIsRefusedBeforeItIsReadWhole.
8 stays documented for now; see the thread.

That test fails until CAMEL-25463 is in the Camel snapshot, so CI will be red on it and the PR stays a draft. It now waits for:

Camel 4.23.0 with CAMEL-25463 (CAMEL-25463: camel-langchain4j-ingest - read a body only up to maxDocumentSize camel#27569)
camel-kamelets 4.23.0 with CAMEL-25027: langchain4j-ingest-sink: add modality and contentType camel-kamelets#3085 and Split HTTP test #3087
[camel-main] Fixes #9152. Compose the langchain4j-ingest pipelines from the catalog Kamelets #9282

Verified locally against a build of the fix: deployment tests, JVM and native ITs pass.

…Kamelets The extension's own route builder is replaced by a composition of the catalog Kamelets: langchain4j-ingest-file-source, tika-extract-text-action or docling-convert-action, and langchain4j-ingest-sink, always loaded from the classpath. Ingestion results stay the same apart from the edge cases listed in the 4.0 migration guide. Each Kamelet runs as its own route, named langchain4j-ingest-<name>-source, -parser or -sink, and the extension registers the four Kamelets for native itself. New dependencies: camel-kamelets, camel-quarkus-kamelet and camel-quarkus-yaml-dsl. Also: an invalid @ingest pipeline name now fails the build, and with two store or model beans the error for a Java-declared pipeline names the IngestPipeline setter. Pipelines with colliding route ids are rejected. Needs the camel-kamelets 4.23.0 release, the first with these Kamelets; until then the root pom pins 4.23.0-SNAPSHOT (marked TODO). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Builds on the Kamelet composition of apache#9152. filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time; the builder setters reject bad values. The raw-size guard before the parse stays in the extension: it reads CamelFileLength on a directory pipeline and at most max + 1 bytes otherwise. The parser actions' own maxDocumentSize (apache/camel-kamelets#3087) is pinned to 0. modality, contentType, includeId, excludeId and the parsers' maxDocumentSize are always passed ("" when unset), so app-wide camel.kamelet.<id>.* properties cannot change them. Needs the camel-kamelets 4.23.0 release including apache/camel-kamelets#3085 (modality and contentType on langchain4j-ingest-sink). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Test that a consumer-fed media stream is read only up to the cap (needs CAMEL-25463) - Pin an unset documentFilter to an always-true predicate - The builder view's filter() delegates to configuration instead of null - Name the @ingest pipelines when filter.* targets an unknown name - Reject a content-type that is only parameters, in the builder and at build time - Lower-case modality before it goes to the sink - Link the advisory id filter to the component's matcher Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jiriOndrusek-agent · 2026-10-16T10:20:06Z

Superseded by #9340 (rebased on main after #9315).

Generated with Claude Code on behalf of @JiriOndrusek

JiriOndrusek · 2026-10-16T10:20:42Z

superseded by #9340

jamesnetherton reviewed Oct 7, 2026

View reviewed changes

JiriOndrusek force-pushed the camel-main branch 2 times, most recently from 21bc352 to 82a5858 Compare October 12, 2026 07:11

jiriOndrusek-agent and others added 3 commits October 13, 2026 10:38

jiriOndrusek-agent force-pushed the fix/9280 branch from f249cfa to b23e833 Compare October 13, 2026 14:29

jiriOndrusek-agent mentioned this pull request Oct 16, 2026

Fixes #9280. Add filter.* and modality=media to langchain4j-ingest #9340

Merged

JiriOndrusek closed this Oct 16, 2026

jiriOndrusek-agent mentioned this pull request Oct 23, 2026

Action: OK to push rebased #9339/#9340 (Kamelets 4.23.0), un-draft (item 25) jiriOndrusek-agent/camel-quarkus#20

Closed

Conversation

JiriOndrusek commented Oct 7, 2026

Uh oh!

jamesnetherton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiriOndrusek-agent commented Oct 8, 2026

Uh oh!

jiriOndrusek-agent commented Oct 16, 2026

Uh oh!

JiriOndrusek commented Oct 16, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants