[analytics-engine] Fix planner gaps surfacing as 'JSONObject["datarows"] not found'

[mengweieric]

Problem

PPL queries on the analytics-engine force-routed path fail with JSONObject["datarows"] not found. Two distinct upstream causes:

1. mvcombine planning fails — PPL mvcombine lowers to Calcite ARRAY_AGG, but AggregateFunction did not register it. The HEP marker rule throws IllegalStateException, which the PPL renderer surfaces as an error envelope.
2. Nested-field queries fail validation — OpenSearchSchemaBuilder skipped nested-typed mappings entirely, so the Calcite row type never exposed their leaves. Parquet already writes them as flat dotted columns (e.g. skills.name), so every reference to such a leaf hits a Calcite "column not found".

Solution

• Register ARRAY_AGG as a STATE_EXPANDING aggregate (same shape as LIST); add it to DataFusion's capability set; route it through the existing PARTIAL/FINAL branch in PplAggregateCallRewriter.
• Treat nested like object in OpenSearchSchemaBuilder: recurse properties and emit dotted leaves. Matches the SQL plugin's own OpenSearchDataType.

Test plan

• Unit test added for nested-with-properties exposing dotted leaves
• Compile + spotless clean on edited modules
• CalciteMvCombineCommandIT on a force-routed AE cluster
• CalciteNewAddedCommandsIT mvexpand-edge-cases / graph-employees on the same

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 5e7037c)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

Persistent review updated to latest commit 5e7037c

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Add null check for fieldProps Add null-safety check for fieldProps before accessing it. If fieldEntry.getValue() returns null, the cast to Map<String, Object> will succeed but subsequent operations will fail with NullPointerException. sandbox/libs/analytics-api/src/main/java/org/opensearch/analytics/schema/OpenSearchSchemaBuilder.java [289-295] +if (fieldProps == null) { + continue; +} if (fieldType == null || "object".equals(fieldType) || "nested".equals(fieldType)) { Map<String, Object> nested = (Map<String, Object>) fieldProps.get("properties"); if (nested != null) { addLeafFields(builder, typeFactory, nested, fieldName); } continue; } Suggestion importance[1-10]: 7 __ Why: Valid defensive programming suggestion. If fieldEntry.getValue() returns null, the cast on line 279 succeeds but accessing fieldProps.get("type") on line 280 would throw NullPointerException. However, this appears to be an edge case that may not occur in practice with valid OpenSearch mappings.	Medium
General	Handle ARRAY_AGG distinct behavior explicitly The new ARRAY_AGG case shares logic with LIST and VALUES, but the isValues boolean only checks for "VALUES". If ARRAY_AGG requires distinct handling similar to VALUES, this logic may be incomplete. Verify that ARRAY_AGG doesn't need its own distinct flag. sandbox/plugins/analytics-backend-datafusion/src/main/java/org/opensearch/be/datafusion/PplAggregateCallRewriter.java [79-88] case "LIST", "VALUES", "ARRAY_AGG" -> { // arg0 type tells us PARTIAL (raw element) vs FINAL (already array): // PARTIAL → array_agg (with INVOCATION_DISTINCT for VALUES); rebuild // return type as ARRAY<arg0> to repair PPL's lossy STRING_ARRAY. // FINAL → list_merge / list_merge_distinct un-nests per-shard arrays. if (call.getArgList().isEmpty()) { return call; } boolean isValues = "VALUES".equalsIgnoreCase(aggregation.getName()); + boolean isArrayAgg = "ARRAY_AGG".equalsIgnoreCase(aggregation.getName()); Suggestion importance[1-10]: 5 __ Why: The suggestion asks to verify if ARRAY_AGG needs distinct handling. While adding an isArrayAgg boolean could be useful, the comment in the code states that ARRAY_AGG has the "same PARTIAL/FINAL state shape as LIST", suggesting it follows LIST's behavior rather than VALUES' distinct behavior. The suggestion is reasonable but may not be necessary based on the documented behavior.	Low

[github-actions]

❌ Gradle check result for 5e7037c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?