Fixes #9280. Add filter.* and modality=media to langchain4j-ingest by jiriOndrusek-agent · Pull Request #9340 · apache/camel-quarkus

jiriOndrusek-agent · 2026-10-16T10:19:58Z

Supersedes #9290

filter.* options go to the sink Kamelet, which enforces them. With a parser, the id patterns are also checked before the parse.
modality=media (optional content-type) embeds each document whole, as one vector; the builder gets modality() and contentType(). Conflicting settings fail the build.
Unlike the issue's first plan, the raw-size guard before the parse stays in the extension: it reads at most max + 1 bytes, while the cap of Fix #3086: Parser actions: add maxDocumentSize, checked before the parse camel-kamelets#3087 reads the whole stream. That cap is pinned off.

Generated with Claude Code on behalf of @JiriOndrusek

jamesnetherton

Reviewed locally. The fixes from the #9290 review all carried over. A few new comments inline; the first two matter most.

On the two points left open in #9290: have the Camel issues been opened (answering filtered for a media file whose type can't be determined, and exposing parsePatterns/idAccepted)? A link here would let us follow them.

This review was generated by an AI agent and may contain inaccuracies. Please verify all suggestions before applying.

Claude Code on behalf of James Netherton

jamesnetherton · 2026-10-27T07:26:47Z

+            // logged at DEBUG
            route.to(kameletUri(name, SINK_KAMELET, "sink", sink)).process(exchange -> {
                IngestResult result = exchange.getMessage().getBody(IngestResult.class);
+                if (result != null && result.outcome() == IngestResult.Outcome.FILTERED) {


On a directory pipeline a filtered file's exchange succeeds, so its key is committed in the source register. With a persistent source.idempotent-repository, a file filtered by filter.exclude-id stays skipped after the filter is removed and the app restarted, unless the file is touched. filter.* is runtime config, so changing it is expected.

That is the same problem given in #9290 as the reason not to commit an untypable media file ("a committed key would keep the file skipped even after a config fix"). Should a filtered file roll back its key instead, or should usage.adoc at least warn about it alongside the persistent register docs?

True, reproduced with a FileIdempotentRepository: the excluded file's key was persisted, and after removing the filter and restarting, the file was never ingested.
Fixed in 92c3c12. Releasing the key would re-read the file on every poll, so a named register on a directory pipeline is now wrapped (SourceRegister), which holds a filtered file's key in memory only. The file is skipped while the app runs and checked again after a restart. Test: Langchain4jIngestFilterTest.filteredFileStaysOutOfTheNamedRegister. usage.adoc updated.
This also removes my #9290 objection to answering filtered for an untypable media file.

Generated with Claude Code on behalf of @JiriOndrusek

jamesnetherton · 2026-10-27T07:26:47Z

-            if (parser == null) {
-                // text is read as UTF-8; a parser receives the raw bytes instead - the format is
-                // its business, and a charset conversion would corrupt a binary document
+            if (parser == null && !media) {


With !media added here, a media pipeline now leaves charset out, so an application-wide camel.kamelet.langchain4j-ingest-file-source.charset fills it in. That goes against the "options at their defaults are passed all the same" rule at line 253. If I read GenericFileConverter right, a charset on the exchange makes the byte conversion go through a reader, which would corrupt WAV/PNG bytes with no error. The same applies to parser pipelines, though that was there before. Can it be pinned to a neutral value, or covered by the IT poison set like documentFilter?

Right, reproduced both ways: with UTF-8, WAV and PDF files fail on every poll. With ISO-8859-1 and max-document-size set, the WAV is stored under a wrong vector and nothing is logged.
It can't be pinned: the file endpoint has no neutral charset ("" fails its validation). Fixed in 92c3c12: parser and media directory pipelines drop the charset right after the source, before anything reads the file. Text pipelines keep UTF-8 pinned.
The IT poison set now has camel.kamelet.langchain4j-ingest-file-source.charset=UTF-16. The audio, Tika and text tests pass with it.

Generated with Claude Code on behalf of @JiriOndrusek

jamesnetherton · 2026-10-27T07:26:47Z

+        if (filter.minDocumentSize() != 0) {
+            sink.put("minDocumentSize", String.valueOf(filter.minDocumentSize()));
+        }
+        sink.put("documentFilter", "#bean:" + filter.documentFilter()


A misspelled filter.document-filter bean fails inside Kamelet property binding, with an error that names the sink route rather than the pipeline or the config key. repositoryRef() already checks the bean up front with a clear "references ... but no such bean exists" message. Could this get the same check?

Done in 92c3c12, with the same message: Ingestion pipeline 'docs' references document filter 'no-such-filter' but no such bean exists. Test: IngestMissingDocumentFilterTest.

Generated with Claude Code on behalf of @JiriOndrusek

jamesnetherton · 2026-10-27T07:26:47Z

            }
+
+            // the component and IngestRoutes check these at startup; a configured pipeline fails the build instead
+            boolean media = "media".equalsIgnoreCase(pipeline.modality());


Minor: the max-segment-size/max-overlap-size/embedding-batch-size checks above still run for modality=media, although the new Javadoc says they don't apply there. A media pipeline with settings copied from a text one (e.g. overlap 600, default segment 500) fails the build over options that are ignored. Skip them when media is true?

Yes, done in 92c3c12. They are skipped for media, as the component does. Test: IngestMediaSplitterSettingsTest (overlap 600 and batch size 0 on a media pipeline, which builds and starts).

Generated with Claude Code on behalf of @JiriOndrusek

jamesnetherton · 2026-10-27T07:26:47Z

+        String excludeId = filter.excludeId().orElse("");
+        sink.put("includeId", includeId);
+        sink.put("excludeId", excludeId);
+        if (filter.minDocumentSize() != 0) {


Nit: minDocumentSize is put as "0" at line 270 and then conditionally overwritten here. A single sink.put("minDocumentSize", String.valueOf(filter.minDocumentSize())) would do.

Done in 92c3c12.

Generated with Claude Code on behalf of @JiriOndrusek

filter.include-id, filter.exclude-id, filter.min-document-size and filter.document-filter go to the sink Kamelet and are enforced by the component (CAMEL-24753), for configured and @ingest pipelines alike. With a parser, the id patterns are also checked before the parse, so an excluded document is never parsed. A missing document-filter bean fails the start naming it. On a directory pipeline, a filtered file's key is held in memory rather than in a named, possibly persistent register, so a changed filter applies to the file after a restart. modality=media, with an optional content-type, embeds each document whole through the sink's media path (CAMEL-25027); IngestPipeline gets modality() and contentType(). A configured pipeline's modality and content-type are checked at build time, while the splitter sizes and batch size, which do not apply, are not; the builder setters reject bad values. modality, contentType and the filter options replace the neutral values apache#9152 pins on the sink, so app-wide camel.kamelet.<id>.* properties still cannot change them. The file source's charset has no neutral value, so parser and media pipelines drop an app-wide one before the raw bytes are read. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

jiriOndrusek-agent · 2026-10-27T10:45:03Z

Thanks for the review. All five points are fixed in 92c3c12, squashed into the PR commit. Details are in the threads. I reproduced the register and charset issues before fixing them. The deployment tests (48) and JVM ITs (31) pass.

On the Camel issues from #9290: filed both, thanks for the nudge:

CAMEL-26341 - answer filtered for a media document whose type can't be determined;
CAMEL-26342 - expose the id-pattern matching (parsePatterns/idAccepted).

With filtered keys now held in memory, the first one no longer has the drawback I gave in #9290.

Generated with Claude Code on behalf of @JiriOndrusek

jamesnetherton

Thanks, all five points are addressed. I checked SourceRegister against the file consumer's idempotent flow (eager add/confirm, non-eager add, rollback remove) and it holds in each case. Approving pending CI.

Two optional nits, not blocking:

fileFiltered looks the register up in the registry for every filtered file; passing the SourceRegister in directly would avoid that.
The new IT checks the key stays out of the named register, but nothing covers the after-restart behaviour itself.

Please link the two Camel issues here once filed.

This review was generated by an AI agent and may contain inaccuracies. Please verify all suggestions before applying.

Claude Code on behalf of James Netherton

This was referenced Oct 16, 2026

[camel-main] Fixes #9280. Add filter.* and modality=media to langchain4j-ingest #9290

Closed

Action: triage your 53 tickets (~20 closeable) jiriOndrusek-agent/camel-quarkus#13

Closed

jiriOndrusek-agent mentioned this pull request Oct 23, 2026

Action: OK to push rebased #9339/#9340 (Kamelets 4.23.0), un-draft (item 25) jiriOndrusek-agent/camel-quarkus#20

Closed

jiriOndrusek-agent force-pushed the agent_9280-ingest-filter-media branch from 4753a66 to 5c6f2c1 Compare October 26, 2026 09:40

jiriOndrusek-agent mentioned this pull request Oct 26, 2026

Fixes #9152. Compose the langchain4j-ingest pipelines from the catalog Kamelets #9339

Merged

jiriOndrusek-agent force-pushed the agent_9280-ingest-filter-media branch from 5c6f2c1 to 28197ef Compare October 26, 2026 15:41

JiriOndrusek marked this pull request as ready for review October 26, 2026 15:50

jamesnetherton reviewed Oct 27, 2026

View reviewed changes

jiriOndrusek-agent force-pushed the agent_9280-ingest-filter-media branch from 28197ef to 92c3c12 Compare October 27, 2026 10:43

jamesnetherton approved these changes Oct 27, 2026

View reviewed changes

jamesnetherton merged commit ae4ef2f into apache:main Oct 27, 2026
8 checks passed

Conversation

jiriOndrusek-agent commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jamesnetherton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiriOndrusek-agent commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jamesnetherton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jiriOndrusek-agent commented Oct 16, 2026 •

edited

Loading

jiriOndrusek-agent commented Oct 27, 2026 •

edited

Loading