From 1a46f4ff68eb1a8a6b80e316869b6e46903ecd26 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20BEAUDOIN?= Date: Thu, 20 Aug 2026 12:07:46 +0200 Subject: [PATCH] fix(schema): model metric aggregation results under `aggregations` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `aggBucketsResult` only declares `buckets` and `after_key`, but half of the aggregations the schema can request never answer with buckets at all. Verified against Manticore Search 29.0.2: terms, histogram, date_histogram, range, date_range, composite -> {"buckets": [...]}, plus {"after_key": {...}} for composite min, max, sum, avg -> {"value": 30.5} median_absolute_deviation -> {"value": 10, "value_as_string": "10"} percentiles, percentile_ranks -> {"values": [{"key": 25, "value": 18, "value_as_string": "18"}, ...]} -> {"values": {"25": 18, "50": 30.5}} when `keyed` is true The request side of these was modelled in 10.1.1 (aggPercentiles, aggPercentileRanks, aggMedianAbsoluteDeviation, aggMetric); the response side was not, so the two halves no longer meet. For the Java client this is fatal rather than lossy. Its ObjectMapper is built with FAIL_ON_UNKNOWN_PROPERTIES enabled, so a metric result raises UnrecognizedPropertyException: Unrecognized field "values" (class com.manticoresearch.client.model.AggBucketsResult), not marked as ignorable (2 known properties: "buckets", "after_key") and the whole searchResponse is lost with it — not just the aggregation. Five of the six metric aggregations above fail this way on 10.2.0; only the bucket ones deserialize. Any faceted search that also asks for percentiles therefore cannot be read back by the Java client, on any server version. Add `value`, `value_as_string` and `values` to `aggBucketsResult`, and say in the description which aggregation returns which. `values` is left untyped for the same reason `buckets` already is: its shape follows the request's `keyed` flag — an array of `{key, value, value_as_string}` objects when false, a map of requested point to value when true. `value` is left untyped as well, deliberately, because its JSON type follows the aggregation. An integer aggregation returns an exact integer that can leave the range a double holds exactly — measured on 29.0.2, `sum` over three int64 rows answers 18014398526259203, and `max` answers 9007199254740993 — while `avg` answers 20.500000. `type: number` would be worse than imprecise in some clients: this generator maps an unformatted number to a 32-bit float, so the Go client would round every metric above 16777216 (`aggTDigest.compression` is `*float32` in the current out/manticoresearch-go for exactly that reason). The new test pins the exact-integer case so the decision cannot be undone by accident. `searchResponse.aggregations` carried its own copy of the pre-fix description and a buckets-only example; both now cover metric results too. Existing accessors are untouched and no schema is renamed, so this is additive. It does add fields to generated response models, which in Rust means a struct literal that names every field stops compiling — the same cost the previous release carried when `uuid` was added to `successResponse` and `updateResponse`. Verified by regenerating the Java, Python and Go clients with OpenAPI Generator 7.17.0 (the version in ./generator-versions) and running the new test against the regenerated Java output. Response bodies in the test are verbatim from Manticore Search 29.0.2. --- manticore.yml | 43 +++++- manticore_int64.yml | 43 +++++- manticore_no_nullables.yml | 43 +++++- .../AggregationResultDeserializationTest.java | 131 ++++++++++++++++++ 4 files changed, 245 insertions(+), 15 deletions(-) create mode 100644 test/java/api/AggregationResultDeserializationTest.java diff --git a/manticore.yml b/manticore.yml index 3e6e6045..b050895b 100644 --- a/manticore.yml +++ b/manticore.yml @@ -1121,17 +1121,42 @@ components: additionalProperties: true aggBucketsResult: type: object - description: Aggregation group returned under `aggregations` + description: > + One aggregation result. It is either BUCKET-shaped or METRIC-shaped, never + both: which one depends on the aggregation that was requested. properties: buckets: description: > Bucket list when `keyed` is false (array of bucket objects), or a keyed map of bucket objects when `keyed` is true. Each bucket has - `key`, `doc_count`, and optional `status`. + `key` and `doc_count`, plus `from`/`to` for `range` and `date_range`, + `key_as_string` for `date_histogram`, and an optional `status`. + Returned by `terms`, `histogram`, `date_histogram`, `range`, + `date_range` and `composite`. after_key: type: object additionalProperties: true description: Pagination cursor returned by composite aggregations + value: + description: > + Returned by `min`, `max`, `sum`, `avg` and + `median_absolute_deviation`. Untyped like `buckets` and `values`, + because the JSON type follows the aggregation: an integer one + returns an exact integer, which for `sum` or `max` over an int64 + attribute can exceed what a double holds exactly, while a float one + returns a floating-point number. `value_as_string` carries the exact + rendering when the engine sends it. + value_as_string: + type: string + description: > + String rendering of `value`, returned alongside it by + `median_absolute_deviation`. + values: + description: > + Returned by `percentiles` and `percentile_ranks`. Like `buckets`, the + shape follows the request's `keyed` flag: an array of + `{key, value, value_as_string}` objects when false, a map of requested + point to value when true. additionalProperties: true searchResponse: type: object @@ -1154,9 +1179,10 @@ components: additionalProperties: $ref: '#/components/schemas/aggBucketsResult' description: > - Aggregated search results grouped by the specified criteria. Each - named aggregation typically contains a `buckets` array (or keyed map) - of bucket objects with `key`, `doc_count`, and optional `status`. + Aggregated search results, keyed by aggregation name. A bucket + aggregation contributes a `buckets` array (or keyed map) of bucket + objects; a metric aggregation contributes a `value` or `values` + instead. See `aggBucketsResult`. example: sizes: buckets: @@ -1173,6 +1199,13 @@ components: - key: 9 doc_count: 954 status: unavailable + avg_price: + value: 30.5 + price_percentiles: + values: + - key: 50 + value: 30.5 + value_as_string: '30.5' hits: type: object properties: diff --git a/manticore_int64.yml b/manticore_int64.yml index 63efbb33..6fcfd55f 100644 --- a/manticore_int64.yml +++ b/manticore_int64.yml @@ -1121,17 +1121,42 @@ components: additionalProperties: true aggBucketsResult: type: object - description: Aggregation group returned under `aggregations` + description: > + One aggregation result. It is either BUCKET-shaped or METRIC-shaped, never + both: which one depends on the aggregation that was requested. properties: buckets: description: > Bucket list when `keyed` is false (array of bucket objects), or a keyed map of bucket objects when `keyed` is true. Each bucket has - `key`, `doc_count`, and optional `status`. + `key` and `doc_count`, plus `from`/`to` for `range` and `date_range`, + `key_as_string` for `date_histogram`, and an optional `status`. + Returned by `terms`, `histogram`, `date_histogram`, `range`, + `date_range` and `composite`. after_key: type: object additionalProperties: true description: Pagination cursor returned by composite aggregations + value: + description: > + Returned by `min`, `max`, `sum`, `avg` and + `median_absolute_deviation`. Untyped like `buckets` and `values`, + because the JSON type follows the aggregation: an integer one + returns an exact integer, which for `sum` or `max` over an int64 + attribute can exceed what a double holds exactly, while a float one + returns a floating-point number. `value_as_string` carries the exact + rendering when the engine sends it. + value_as_string: + type: string + description: > + String rendering of `value`, returned alongside it by + `median_absolute_deviation`. + values: + description: > + Returned by `percentiles` and `percentile_ranks`. Like `buckets`, the + shape follows the request's `keyed` flag: an array of + `{key, value, value_as_string}` objects when false, a map of requested + point to value when true. additionalProperties: true searchResponse: type: object @@ -1154,9 +1179,10 @@ components: additionalProperties: $ref: '#/components/schemas/aggBucketsResult' description: > - Aggregated search results grouped by the specified criteria. Each - named aggregation typically contains a `buckets` array (or keyed map) - of bucket objects with `key`, `doc_count`, and optional `status`. + Aggregated search results, keyed by aggregation name. A bucket + aggregation contributes a `buckets` array (or keyed map) of bucket + objects; a metric aggregation contributes a `value` or `values` + instead. See `aggBucketsResult`. example: sizes: buckets: @@ -1173,6 +1199,13 @@ components: - key: 9 doc_count: 954 status: unavailable + avg_price: + value: 30.5 + price_percentiles: + values: + - key: 50 + value: 30.5 + value_as_string: '30.5' hits: type: object properties: diff --git a/manticore_no_nullables.yml b/manticore_no_nullables.yml index 0f5c54a9..6e32de2f 100644 --- a/manticore_no_nullables.yml +++ b/manticore_no_nullables.yml @@ -1119,17 +1119,42 @@ components: additionalProperties: true aggBucketsResult: type: object - description: Aggregation group returned under `aggregations` + description: > + One aggregation result. It is either BUCKET-shaped or METRIC-shaped, never + both: which one depends on the aggregation that was requested. properties: buckets: description: > Bucket list when `keyed` is false (array of bucket objects), or a keyed map of bucket objects when `keyed` is true. Each bucket has - `key`, `doc_count`, and optional `status`. + `key` and `doc_count`, plus `from`/`to` for `range` and `date_range`, + `key_as_string` for `date_histogram`, and an optional `status`. + Returned by `terms`, `histogram`, `date_histogram`, `range`, + `date_range` and `composite`. after_key: type: object additionalProperties: true description: Pagination cursor returned by composite aggregations + value: + description: > + Returned by `min`, `max`, `sum`, `avg` and + `median_absolute_deviation`. Untyped like `buckets` and `values`, + because the JSON type follows the aggregation: an integer one + returns an exact integer, which for `sum` or `max` over an int64 + attribute can exceed what a double holds exactly, while a float one + returns a floating-point number. `value_as_string` carries the exact + rendering when the engine sends it. + value_as_string: + type: string + description: > + String rendering of `value`, returned alongside it by + `median_absolute_deviation`. + values: + description: > + Returned by `percentiles` and `percentile_ranks`. Like `buckets`, the + shape follows the request's `keyed` flag: an array of + `{key, value, value_as_string}` objects when false, a map of requested + point to value when true. additionalProperties: true searchResponse: type: object @@ -1152,9 +1177,10 @@ components: additionalProperties: $ref: '#/components/schemas/aggBucketsResult' description: > - Aggregated search results grouped by the specified criteria. Each - named aggregation typically contains a `buckets` array (or keyed map) - of bucket objects with `key`, `doc_count`, and optional `status`. + Aggregated search results, keyed by aggregation name. A bucket + aggregation contributes a `buckets` array (or keyed map) of bucket + objects; a metric aggregation contributes a `value` or `values` + instead. See `aggBucketsResult`. example: sizes: buckets: @@ -1171,6 +1197,13 @@ components: - key: 9 doc_count: 954 status: unavailable + avg_price: + value: 30.5 + price_percentiles: + values: + - key: 50 + value: 30.5 + value_as_string: '30.5' hits: type: object properties: diff --git a/test/java/api/AggregationResultDeserializationTest.java b/test/java/api/AggregationResultDeserializationTest.java new file mode 100644 index 00000000..a774f12d --- /dev/null +++ b/test/java/api/AggregationResultDeserializationTest.java @@ -0,0 +1,131 @@ +package com.manticoresearch.client.api; + +import com.manticoresearch.client.JSON; +import com.manticoresearch.client.model.AggBucketsResult; +import com.manticoresearch.client.model.SearchResponse; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; + +/** + * `aggregations` carries two kinds of result, and the client must read both. + * + * A BUCKET aggregation (`terms`, `histogram`, `date_histogram`, `range`, `date_range`, + * `composite`) answers with `buckets`; a METRIC aggregation (`min`, `max`, `sum`, `avg`, + * `median_absolute_deviation`, `percentiles`, `percentile_ranks`) answers with `value` or + * `values` and no `buckets` at all. + * + * The bodies below are verbatim responses from Manticore Search 29.0.2. A field absent from the + * schema is unreadable at best; on the client as published, whose mapper rejects unknown + * properties, a metric result costs the caller the whole search response rather than that one + * aggregation. + */ +public class AggregationResultDeserializationTest { + + private static final String HEAD = + "{\"took\":0,\"timed_out\":false," + + "\"hits\":{\"total\":5,\"total_relation\":\"eq\",\"hits\":[]}," + + "\"aggregations\":{"; + + private SearchResponse deserialize(String aggregations) throws Exception { + return JSON.getDefault().getMapper().readValue(HEAD + aggregations + "}}", SearchResponse.class); + } + + private AggBucketsResult aggregation(String body) throws Exception { + SearchResponse response = deserialize("\"a\":" + body); + assertNotNull(response.getAggregations(), "the response carries no aggregations at all"); + AggBucketsResult result = response.getAggregations().get("a"); + assertNotNull(result, "aggregation \"a\" is missing from the response"); + return result; + } + + @Test + public void percentilesAndPercentileRanksReturnValuesAsAnArray() throws Exception { + AggBucketsResult result = aggregation( + "{\"values\":[{\"key\":25,\"value\":18,\"value_as_string\":\"18\"}," + + "{\"key\":50,\"value\":30.5,\"value_as_string\":\"30.5\"}]}"); + + List values = assertInstanceOf(List.class, result.getValues()); + assertEquals(2, values.size()); + Map first = assertInstanceOf(Map.class, values.get(0)); + assertEquals(25, first.get("key")); + assertEquals("18", first.get("value_as_string")); + assertNull(result.getBuckets(), "a metric result carries no buckets"); + } + + @Test + public void keyedPercentilesReturnValuesAsAMap() throws Exception { + AggBucketsResult result = aggregation("{\"values\":{\"25\":18,\"50\":30.5}}"); + + Map values = assertInstanceOf(Map.class, result.getValues()); + assertEquals(18, values.get("25")); + } + + @Test + public void singleMetricsReturnAValue() throws Exception { + AggBucketsResult result = aggregation("{\"value\":30.5}"); + + assertEquals(30.5, result.getValue()); + assertNull(result.getValueAsString()); + } + + @Test + public void medianAbsoluteDeviationAlsoReturnsValueAsString() throws Exception { + AggBucketsResult result = aggregation("{\"value\":10,\"value_as_string\":\"10\"}"); + + assertEquals(10, result.getValue()); + assertEquals("10", result.getValueAsString()); + } + + /** + * An integer aggregation returns an exact integer, and `sum` or `max` over an int64 attribute + * can leave the range a double represents exactly. Measured on Manticore Search 29.0.2: + * `sum` over three int64 rows answered 18014398526259203, which is 2^54 + 3. Typing `value` as + * a number would round it here, and would cost far more in the clients whose generator maps an + * unformatted number to a 32-bit float. + */ + @Test + public void integerMetricsKeepTheirExactValue() throws Exception { + AggBucketsResult result = aggregation("{\"value\":18014398526259203}"); + + assertEquals(18014398526259203L, result.getValue()); + } + + @Test + public void bucketResultsStillDeserialize() throws Exception { + AggBucketsResult result = aggregation( + "{\"buckets\":[{\"key\":\"x\",\"doc_count\":2},{\"key\":\"y\",\"doc_count\":2}]}"); + + assertEquals(2, assertInstanceOf(List.class, result.getBuckets()).size()); + assertNull(result.getValue(), "a bucket result carries no metric value"); + } + + @Test + public void compositeBucketsKeepTheirAfterKey() throws Exception { + AggBucketsResult result = aggregation( + "{\"after_key\":{\"c\":\"y\"},\"buckets\":[{\"key\":{\"c\":\"x\"},\"doc_count\":2}]}"); + + assertEquals(Collections.singletonMap("c", "y"), result.getAfterKey()); + } + + @Test + public void bucketAndMetricAggregationsCoexistInOneResponse() throws Exception { + SearchResponse response = deserialize( + "\"by_cat\":{\"buckets\":[{\"key\":\"x\",\"doc_count\":2}]}," + + "\"avg_price\":{\"value\":30.5}," + + "\"price_percentiles\":{\"values\":[{\"key\":50,\"value\":30.5," + + "\"value_as_string\":\"30.5\"}]}"); + + assertEquals(3, response.getAggregations().size()); + assertNotNull(response.getAggregations().get("by_cat").getBuckets()); + assertEquals(30.5, response.getAggregations().get("avg_price").getValue()); + assertNotNull(response.getAggregations().get("price_percentiles").getValues()); + } +}