Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 68 additions & 12 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,47 @@ For setup and compatibility notes, see [Coding agents](CONTRIBUTING.md#coding-ag

Apache IoTDB is a time series database for IoT data. It uses a distributed architecture with ConfigNodes (metadata/coordination) and DataNodes (storage/query). Data is stored in TsFile columnar format (separate repo: https://github.com/apache/tsfile). Current version is 2.0.11-SNAPSHOT.

## Branch Relationship and Backports

- `master` is the primary development branch for the 2.x line. `dev/1.3` maintains the 1.3.x
line. They have diverged in supported features, implementation, APIs, and dependencies
(including TsFile); `dev/1.3` is not a mirror of `master`.
- A request to cherry-pick (CP) a `master` PR to `dev/1.3` authorizes only that PR's changes and
the minimum compatibility adaptations needed to preserve their logic on the target branch.
**The backport must be logically and behaviorally equivalent to the original PR. Do not make
additional changes or use the backport as an opportunity to improve the surrounding code.**

### Cherry-picking from master to dev/1.3

1. Read the source PR's final diff, relevant surrounding code, and regression tests. Identify
the exact source commit(s) and target base revision before editing. Base the backport branch
on the requested `dev/1.3` revision, and follow that branch's applicable instructions and
build/test configuration.
2. Prefer `git cherry-pick -x` for the requested commits. When conflicts or branch differences
require manual adaptation, preserve source traceability in the commit/PR description and
resolve each change by its intended behavior. Do not blindly choose one side of a conflict
or copy whole files from `master` over target-branch implementations.
3. Preserve the source PR's conditions, execution order, return values, exceptions/status codes,
side effects, boundary cases, and concurrency/resource-lifecycle guarantees. Adapt package
names, method signatures, or APIs only where the target requires it, using equivalent target
facilities. A clean cherry-pick or successful compilation alone does not prove equivalence.
4. Keep the diff limited to the requested backport. Do not add unrelated fixes, refactors,
optimizations, renames, formatting changes, dependency upgrades, or configuration changes.
Preserve unrelated `dev/1.3` behavior and keep formatter changes limited to the affected code.
Do not silently cherry-pick prerequisite PRs or import newer `master` features to make the
patch compile.
5. If a source change has no target counterpart, a prerequisite is missing, or equivalence cannot
be established, identify the exact gap and the minimum work needed to resolve it. Obtain
direction before expanding scope or dropping part of the source change; do not substitute
a weaker fix or describe an incomplete backport as equivalent.
6. Port the relevant regression tests with equivalent scenarios and assertions, adapting only
to the target test framework. Verify the original failure case and affected boundary/error
paths on `dev/1.3`; do not weaken assertions or skip failing cases to make the backport pass.
7. Review the complete diff against the target base before finishing: account for every source
change (including any already present on the target), and map every target hunk to a source
change or a necessary compatibility adaptation. Report the source PR/commit(s), target base,
adaptations and why they preserve semantics, and actual validation results/limitations.

## Build Commands

```bash
Expand Down Expand Up @@ -56,12 +97,30 @@ mvn spotless:apply
mvn spotless:check
```

## Integration Tests
## Module-specific Instructions

Before editing files or running build/test commands for the areas below, read and follow the
corresponding instructions. For changes spanning multiple areas, read each applicable file,
including when the task starts from the repository root.

| Area | Instructions | Module-specific guidance |
| --- | --- | --- |
| AINode | [iotdb-core/ainode/AGENTS.md](iotdb-core/ainode/AGENTS.md) | Python environment, generated sources, formatting, tests, and binary packaging |
| Tree-model SQL grammar | [iotdb-core/antlr/AGENTS.md](iotdb-core/antlr/AGENTS.md) | Lexer keywords, identifier compatibility, and grammar verification |
| Table-model SQL grammar | [iotdb-core/relational-grammar/AGENTS.md](iotdb-core/relational-grammar/AGENTS.md) | Reserved/non-reserved keywords, identifier compatibility, and grammar verification |
| Shared node models and infrastructure | [iotdb-core/node-commons/AGENTS.md](iotdb-core/node-commons/AGENTS.md) | Shared paths, schema, partition/auth/configuration models, and RPC/service abstractions |
| Reusable query computation | [iotdb-core/calc-commons/AGENTS.md](iotdb-core/calc-commons/AGENTS.md) | Reusable operators, column transformations, aggregation, joins, and sorting |
| Distribution and assembly descriptors | [distribution/AGENTS.md](distribution/AGENTS.md) | Artifact identities, package contents, Edge configuration, and packaging verification |
| Launchers, environment scripts, and operational tools | [scripts/AGENTS.md](scripts/AGENTS.md) | Platform compatibility, process ownership, cleanup behavior, and script verification |
| Integration tests | [integration-test/AGENTS.md](integration-test/AGENTS.md) | IT authoring, test profiles, targeted verification, and Edge/IDE runs |

Read both commons guides before adding or moving shared code between these modules. Logic tied to
a concrete node's storage, lifecycle, or execution scheduling belongs in that node's module.

Before editing or running integration tests, read and follow
[integration-test/AGENTS.md](integration-test/AGENTS.md). It contains the IT authoring rules,
formatting and build commands, test profiles, targeted verification, and Edge/IDE instructions.
Keep IT-specific rules and commands in that module-level file.
For changes to `EdgeNode` or Edge configuration defaults outside these directories, also read the
distribution and scripts instructions. For SQL parsing or keyword changes in DataNode code, also
read the relevant grammar instructions. Keep shared rules in this file and specialized rules in
the corresponding module file; link to existing guidance instead of copying it.

## Code Style

Expand All @@ -80,12 +139,10 @@ Keep IT-specific rules and commands in that module-level file.

### Edge Distribution

- `EdgeNode` (`iotdb-core/confignode/src/main/java/org/apache/iotdb/edge/EdgeNode.java`) starts the ConfigNode and DataNode in one JVM for resource-constrained single-node deployments.
- `distribution/src/assembly/edge.xml` produces the additional `apache-iotdb-<version>-edge-bin.zip` artifact. Keep its `edge-bin` assembly id unique so it cannot replace an existing attached artifact.
- Edge-specific system defaults live under `iotdb-core/node-commons/src/assembly/resources/conf/edge/`. Standard node, all-in-one, and integration-test assemblies must continue to exclude `edge/**`.
- Edge launchers must reuse the configuration-aware port checks in `scripts/conf/iotdb-common.sh`. Stop scripts must match both the `EdgeNode` main class and the exact `IOTDB_HOME`; never fall back to a global process-name kill.
- Package node-independent tools that work with the combined process. Exclude scripts that require standalone node launchers or environment files, including destructive, daemon-management, and health-check scripts.
- After packaging changes, run a clean distribution build, check that every existing distribution artifact is still produced, and inspect the Edge zip contents and startup/shutdown behavior.
`EdgeNode` (`iotdb-core/confignode/src/main/java/org/apache/iotdb/edge/EdgeNode.java`) starts the
ConfigNode and DataNode in one JVM for resource-constrained single-node deployments. Its package
layout and operational requirements are maintained in the distribution and scripts instructions
listed above.

### Dual Data Model

Expand Down Expand Up @@ -164,7 +221,6 @@ The project uses compile-time i18n via the `build-helper-maven-plugin`. The prop
### Code Style

- **Always run `mvn spotless:apply` after editing Java files**: Spotless runs `spotless:check` automatically during the `compile` phase. Format violations cause an immediate BUILD FAILURE. Make it a habit to run `mvn spotless:apply -pl <module>` right after editing, not at the end.
- **Always run `black` and `isort` after editing Python files under `iotdb-core/ainode/`**: The AINode Code Style Check CI runs `black --check .` and `isort --check-only --profile black .` on that directory. Run `cd iotdb-core/ainode && black . && isort --profile black .` before committing. Requires `pip install black==25.1.0 isort==6.0.1`.
- **Gson version compatibility**: `JsonObject.isEmpty()` / `JsonArray.isEmpty()` may not be available in the Gson version used by this project. Use `size() > 0` instead and add a comment explaining why.

## Git Commit
Expand Down
78 changes: 78 additions & 0 deletions distribution/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
<!--

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->

# Distribution Guidelines

These instructions apply to `distribution/` and supplement the
[repository instructions](../AGENTS.md).

## Assembly ownership and artifact identity

- Treat [pom.xml](pom.xml) and `src/assembly/` as the sources of package layout. Edit the
original configuration or script source, not staged files or archives under `target/`.
- Preserve the default `all-bin`, `library-udf-bin`, `library-pipe-bin`, and `edge-bin`
assemblies. `with-ainode` and `with-cpp` add their own distributions; keep profile descriptors
additive so enabling one does not replace the default descriptor list.
- Assembly IDs determine attached artifact classifiers. Keep each ID unique, especially
`edge-bin`, and check final archive/base-directory names and SHA-512 checksum includes when
adding or renaming an artifact. Preserve existing outputs for every affected profile.
- Keep LICENSE, NOTICE, and bundled dependency licenses in the resulting archives; reuse
`src/assembly/common-files.xml` and the AINode-specific component where applicable.
- Preserve executable file modes and Maven filtering for shared environment scripts, including
`iotdb-common.sh` and `windows/iotdb-common.bat`. Check includes/excludes against both Unix
and Windows paths. Read [scripts/AGENTS.md](../scripts/AGENTS.md) when changing script contents
or availability, and [AINode instructions](../iotdb-core/ainode/AGENTS.md) for AINode packaging.

## Edge package boundaries

- [src/assembly/edge.xml](src/assembly/edge.xml) produces
`apache-iotdb-<version>-edge-bin.zip` for the combined ConfigNode/DataNode process.
- Edge system defaults come from
`iotdb-core/node-commons/src/assembly/resources/conf/edge/`; Edge logging configuration comes
from `distribution/src/assembly/resources/conf-edge/`. Keep these separate from standard
configuration. Standard node, all-in-one, and integration-test packages must not acquire
Edge defaults; preserve `edge/**` exclusions wherever shared configuration directories are
copied recursively.
- Package the Edge start/stop pair and compatible CLI/tools. Exclude tools requiring standalone
node launchers or environment files, including standalone daemon, destroy, and health-check
scripts. The explicit Edge daemon/destroy replacements are intentional exceptions; retain
their inclusion without restoring the standalone tools.
- Check the standalone node and integration-test assembly descriptors as well when changing
shared configuration or script selection; a correct Edge archive does not establish that
other packages are unaffected.

## Packaging verification

For changes to assembly descriptors, packaged resources, or artifact wiring:

1. Run the clean distribution build from the
[root build instructions](../AGENTS.md#build-commands). Enable any affected optional profile
as well; an old archive in `target/` is not evidence of a successful new build.
2. Inspect the produced archives and SHA-512 files under `distribution/target/`. Verify all
expected artifacts for the active profiles, archive roots, configuration, dependency jars,
licenses, Unix executable modes, and Windows entries. Confirm unwanted tools and Edge-only
defaults are absent from the other packages.
3. Inspect the Edge zip and verify startup/shutdown from a freshly extracted package using an
isolated installation. For Edge package selection changes, also run the script/assembly
checks described in [scripts/AGENTS.md](../scripts/AGENTS.md#verification).
4. For Edge runtime changes, follow the build and Tree/Table read-write verification in
[integration-test/AGENTS.md](../integration-test/AGENTS.md#edge-tests). Keep the IT commands
there rather than duplicating them here.
16 changes: 16 additions & 0 deletions integration-test/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,22 @@ These instructions apply to `integration-test/` and supplement the repository-le
- Assert the behavior under test and its recovery result. Only tolerate expected cleanup errors
in narrowly scoped cleanup code; do not swallow failures from the operation being verified.

### Result ordering across DataNodes

- When row order is not part of the query's contract and may differ across DataNodes, use
`TestUtils.resultSetEqualTestUnordered(sql, expectedHeader, expectedRetArray)` for tree-model
SQL cases. When the test already owns a result set (including table-model queries), use
`TestUtils.assertResultSetEqualUnordered(resultSet, expectedHeader, expectedRows)` and close
the original result set after the assertion. Reuse these helpers instead of writing per-node
comparison loops in individual tests.
- These helpers check every node independently, including headers, values, and duplicate counts.
The default `ClusterTestResultSet` compares nodes at the same cursor position; collecting or
sorting rows after reading through that wrapper cannot avoid row-order mismatches. Do not
replace the helpers with set equality or single-node checks that weaken these assertions.
- Keep ordered assertions when SQL guarantees the final result order. For `LIMIT`/`OFFSET`,
ensure the selected rows are deterministic before applying unordered equality; sorting cannot
repair a nondeterministic subset. Add sufficient `ORDER BY` tie-breakers when needed.

## Verification

This module is excluded from the default build. Run the commands below from the repository root,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
package org.apache.iotdb.db.it.aggregation;

import org.apache.iotdb.it.env.EnvFactory;
import org.apache.iotdb.it.env.cluster.node.DataNodeWrapper;
import org.apache.iotdb.it.framework.IoTDBTestRunner;
import org.apache.iotdb.itbase.category.ClusterIT;
import org.apache.iotdb.itbase.category.LocalStandaloneIT;
Expand All @@ -32,20 +31,10 @@
import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;

import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Set;

import static org.apache.iotdb.db.it.utils.TestUtils.assertTestFail;
import static org.apache.iotdb.db.it.utils.TestUtils.prepareData;
import static org.apache.iotdb.db.it.utils.TestUtils.resultSetEqualTest;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.apache.iotdb.db.it.utils.TestUtils.resultSetEqualTestUnordered;

@RunWith(IoTDBTestRunner.class)
@Category({LocalStandaloneIT.class, ClusterIT.class})
Expand Down Expand Up @@ -225,7 +214,7 @@ public void testTimeDurationSingleColumnOnOneDataRegion() {
}

@Test
public void testTimeDurationManyColumnOnManyDataRegion() throws SQLException {
public void testTimeDurationManyColumnOnManyDataRegion() {
// normal
String[] expectedHeader =
new String[] {
Expand Down Expand Up @@ -357,27 +346,10 @@ public void testTimeDurationManyColumnOnManyDataRegion() throws SQLException {
// group by tags
expectedHeader = new String[] {"city", "time_duration(s1)"};
retArray = new String[] {"Beijing,1677570933,", "Nanjing,1677570938,"};
// GROUP BY TAGS does not guarantee row order, so check each DataNode independently.
for (DataNodeWrapper dataNode : EnvFactory.getEnv().getDataNodeWrapperList()) {
try (Connection connection = EnvFactory.getEnv().getConnection(dataNode);
Statement statement = connection.createStatement();
ResultSet resultSet =
statement.executeQuery(
"select time_duration(s1) from root.db1.**,root.db.** group by tags(city)")) {
ResultSetMetaData metaData = resultSet.getMetaData();
assertEquals(expectedHeader.length, metaData.getColumnCount());
for (int i = 0; i < expectedHeader.length; i++) {
assertEquals(expectedHeader[i], metaData.getColumnName(i + 1));
}
Set<String> remainingRows = new HashSet<>(Arrays.asList(retArray));
while (resultSet.next()) {
String row =
resultSet.getString("city") + "," + resultSet.getString("time_duration(s1)") + ",";
assertTrue(dataNode.getIpAndPortString() + ": " + row, remainingRows.remove(row));
}
assertTrue(dataNode.getIpAndPortString() + ": " + remainingRows, remainingRows.isEmpty());
}
}
resultSetEqualTestUnordered(
"select time_duration(s1) from root.db1.**,root.db.** group by tags(city)",
expectedHeader,
retArray);

// group by time
expectedHeader =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
Expand Down Expand Up @@ -280,6 +281,26 @@ public static void resultSetEqualTest(
resultSetEqualTest(sql, expectedHeader, expectedRetArray, null);
}

/**
* Executes a tree-model query and checks unordered rows on every read node, including duplicate
* multiplicity. Use only when the query's final result does not require a particular row order or
* select a nondeterministic subset of rows.
*/
public static void resultSetEqualTestUnordered(
String sql, String[] expectedHeader, String[] expectedRetArray) {
try (Connection connection = EnvFactory.getEnv().getConnection();
Statement statement = connection.createStatement()) {
connection.setClientInfo("time_zone", "+00:00");
try (ResultSet resultSet = statement.executeQuery(sql)) {
assertResultSetEqualUnordered(
resultSet, String.join(",", expectedHeader) + ",", Arrays.asList(expectedRetArray));
}
} catch (SQLException e) {
e.printStackTrace();
fail(e.getMessage());
}
}

public static void tableResultSetEqualTest(
String sql, String[] expectedHeader, String[] expectedRetArray, String database) {
tableResultSetEqualTest(
Expand Down
Loading
Loading