Skip to content

fix(dataset): detect dimensions across sparse object-row datasets #21699

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
plainheart merged 2 commits into from
Jul 23, 2026
+96 −6
Merged

fix(dataset): detect dimensions across sparse object-row datasets #21699

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
db7241a
fix(data): detect sparse object keys. close #11322
Jul 17, 2026
b6ace83
perf(data): avoid object key scan allocations. close #11322
Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions src/data/Source.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

import {
isTypedArray, HashMap, clone, createHashMap, isArray, isObject, isArrayLike,
hasOwn, assert, each, map, isNumber, isString, keys
hasOwn, assert, each, map, isNumber, isString
} from 'zrender/src/core/util';
import {
SourceFormat, SeriesLayoutBy, DimensionDefinition,
Expand Down Expand Up @@ -401,12 +401,23 @@ function determineSourceDimensions(
}

function objectRowsCollectDimensions(data: OptionSourceDataObjectRows): DimensionDefinitionLoose[] {
let firstIndex = 0;
let obj;
while (firstIndex < data.length && !(obj = data[firstIndex++])) {} // jshint ignore: line
if (obj) {
return keys(obj);
const dimensionNameMap = createHashMap<true, DimensionName>();
const dimensionsDefine: DimensionDefinitionLoose[] = [];

for (let i = 0; i < data.length; i++) {
const obj = data[i];
if (!obj) {
continue;
}
for (const name in obj) {
if (hasOwn(obj, name) && !dimensionNameMap.get(name)) {
dimensionNameMap.set(name, true);
dimensionsDefine.push(name);
}
}
}
Comment on lines +404 to 418

@plainheart plainheart Jul 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this issue. I agree that collecting keys from all object rows is necessary to handle sparse fields correctly, but it also adds an O(total number of object keys) pre-scan during source initialization when dimensions is not explicitly provided.

Could you please add a small performance check to verify that this does not introduce a noticeable overhead for large datasets? It would be useful to compare:

  • the previous behavior (reading only the first non-empty object)
  • the new full-scan behavior
  • the path where dataset.dimensions is explicitly provided

For example, testing with around 100k rows would be helpful, covering both stable object shapes and sparse keys that appear only in later rows. Measuring setOption or source creation time should be sufficient. This would help us assess the practical impact of the additional scan.

Also, the current implementation creates a keys(obj) array and uses a callback for every row. If this shows up in the benchmark, could we avoid the intermediate allocation and callback overhead with something like:

for (const name in obj) {
    if (hasOwn(obj, name) && !dimensionNameMap.get(name)) {
        dimensionNameMap.set(name, true);
        dimensionsDefine.push(name);
    }
}

This preserves the intended semantics, collecting every field while retaining first-seen order, while potentially reducing allocations and per-row overhead.

@rahul37wallst-sudo rahul37wallst-sudo Jul 22, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. I ran a local benchmark with 100k object rows on Windows / Node 22.18.0, using two warmups and nine measured runs.

Median source-creation times:

  • Stable shape: previous-like first-row path 0.07 ms; full scan 4.92 ms; explicit dataset.dimensions 0.02 ms.
  • Sparse shape with a key appearing only in the final row: previous-like path 0.02 ms; full scan 5.25 ms; explicit dimensions 0.01 ms.

I also isolated the collector implementation:

  • keys(obj) plus callback: 38.47 ms stable / 34.86 ms sparse.
  • for...in plus hasOwn: 3.10 ms stable / 3.67 ms sparse.

I updated the implementation to the allocation-free for...in loop in commit b6ace83a5. The full scan remains O(total object keys), as required to discover late sparse fields, but the measured source-initialization overhead was about 5 ms for 100k three-field rows. Supplying dataset.dimensions continues to bypass the scan.

The focused regression tests, npm run lint, npm run checktype, and git diff --check all pass.


return dimensionsDefine.length ? dimensionsDefine : void 0;
}

// Consider dimensions defined like ['A', 'price', 'B', 'price', 'C', 'price'],
Expand Down
14 changes: 14 additions & 0 deletions test/ut/spec/data/SeriesData.test.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

65 changes: 65 additions & 0 deletions test/ut/spec/series/bar.test.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.