-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Native engine abstractions #20821
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Bukhtawar
merged 11 commits into
opensearch-project:main
from
bharath-techie:native-eng
Mar 25, 2026
Merged
Native engine abstractions #20821
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
8ab51e6
Native engine abstractions / skeleton flow
bharath-techie 6575f27
Moving reader manager out of engine and adding contexts + providers
bharath-techie 13221aa
Refactor CompositeEngine to use factory (#50)
Bukhtawar c010bb4
Introduce segment collector interface and simplify Providers (#51)
Bukhtawar ce7225b
Decouple IndexFileDeleter (#52)
Bukhtawar b80dc73
wiring result stream and adding tests
bharath-techie 9ba9266
separating out bridge/engine and indexing integration interfaces / a…
bharath-techie db1c08b
switching search exec engine per query
bharath-techie c098bc0
adding java docs
bharath-techie bfa6e18
fixing gradle run
bharath-techie 17d7546
addressing comments and fixing build failures
bharath-techie File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
57 changes: 0 additions & 57 deletions
57
...libs/analytics-framework/src/main/java/org/opensearch/analytics/backend/EngineBridge.java
This file was deleted.
Oops, something went wrong.
39 changes: 39 additions & 0 deletions
39
...analytics-framework/src/main/java/org/opensearch/analytics/backend/EngineResultBatch.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.analytics.backend; | ||
|
|
||
| import java.util.List; | ||
|
|
||
| /** | ||
| * Read-only view of a single record batch. Provides field names, row count, | ||
| * and positional access to field values. | ||
| * | ||
| * @opensearch.internal | ||
| */ | ||
| public interface EngineResultBatch { | ||
|
|
||
| /** | ||
| * Ordered list of field (column) names in this batch. | ||
| */ | ||
| List<String> getFieldNames(); | ||
|
|
||
| /** | ||
| * Number of rows in this batch. | ||
| */ | ||
| int getRowCount(); | ||
|
|
||
| /** | ||
| * Returns the value at the given row index for the named field. | ||
| * | ||
| * @param fieldName column name | ||
| * @param rowIndex zero-based row index | ||
| * @return the value (may be null) | ||
| */ | ||
| Object getFieldValue(String fieldName, int rowIndex); | ||
| } |
30 changes: 30 additions & 0 deletions
30
...nalytics-framework/src/main/java/org/opensearch/analytics/backend/EngineResultStream.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.analytics.backend; | ||
|
|
||
| import java.util.Iterator; | ||
|
|
||
| /** | ||
| * A closeable stream of record batches returned by engine execution. | ||
| * Callers iterate batches via the returned iterator and MUST close the stream | ||
| * when done to release native resources. | ||
| * | ||
| * @opensearch.internal | ||
| */ | ||
| public interface EngineResultStream extends AutoCloseable { | ||
|
|
||
| /** | ||
| * Returns an iterator over the record batches in this stream. | ||
| * Each call returns the same iterator instance — the stream is single-pass. | ||
| */ | ||
| Iterator<EngineResultBatch> iterator(); | ||
|
|
||
| @Override | ||
| void close(); | ||
| } |
52 changes: 52 additions & 0 deletions
52
.../analytics-framework/src/main/java/org/opensearch/analytics/backend/ExecutionContext.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.analytics.backend; | ||
|
|
||
| import org.opensearch.action.search.SearchShardTask; | ||
| import org.opensearch.index.engine.DataFormatAwareEngine; | ||
|
|
||
| /** | ||
| * Execution context carrying reader and delegation state through | ||
| * the query execution lifecycle. | ||
| * | ||
| * @opensearch.internal | ||
| */ | ||
| public class ExecutionContext { | ||
|
|
||
| private final String tableName; | ||
| private final DataFormatAwareEngine.DataFormatAwareReader reader; | ||
| private final SearchShardTask task; | ||
|
|
||
| /** | ||
| * Constructs an execution context. | ||
| * @param tableName the target table name | ||
| * @param task the search shard task | ||
| * @param reader the data-format aware reader | ||
| */ | ||
| public ExecutionContext(String tableName, SearchShardTask task, DataFormatAwareEngine.DataFormatAwareReader reader) { | ||
| this.tableName = tableName; | ||
| this.task = task; | ||
| this.reader = reader; | ||
| } | ||
|
|
||
| /** Returns the search shard task. */ | ||
| public SearchShardTask getTask() { | ||
| return task; | ||
| } | ||
|
|
||
| /** Returns the target table name. */ | ||
| public String getTableName() { | ||
| return tableName; | ||
| } | ||
|
|
||
| /** Returns the data-format aware reader. */ | ||
| public DataFormatAwareEngine.DataFormatAwareReader getReader() { | ||
| return reader; | ||
| } | ||
| } |
37 changes: 37 additions & 0 deletions
37
.../analytics-framework/src/main/java/org/opensearch/analytics/backend/SearchExecEngine.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.analytics.backend; | ||
|
|
||
| import org.opensearch.common.annotation.ExperimentalApi; | ||
|
|
||
| import java.io.Closeable; | ||
| import java.io.IOException; | ||
|
|
||
| /** | ||
| * Shard-level search execution engine interface. | ||
| * @opensearch.experimental | ||
| */ | ||
| @ExperimentalApi | ||
| public interface SearchExecEngine<T, V> extends Closeable { | ||
| /** | ||
| * Creates an execution context from a resolved plan. | ||
| * | ||
| * @param context ExecutionContext | ||
| */ | ||
| void prepare(T context); | ||
|
|
||
| /** | ||
| * Executes the context and returns a result stream. | ||
| * @param context the execution context | ||
| */ | ||
| V execute(T context) throws IOException; | ||
|
|
||
| @Override | ||
| default void close() throws IOException {} | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.