Skip to content

Add portable IoTDB ODBC driver with Session TLS and application compatibility tests - #18878

Open
hongzhi-gao wants to merge 17 commits into
apache:masterfrom
hongzhi-gao:feat/odbc-client
Open

hongzhi-gao wants to merge 17 commits into
apache:masterfrom
hongzhi-gao:feat/odbc-client

Conversation

@hongzhi-gao

@hongzhi-gao hongzhi-gao commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Description

Add a portable ODBC client for IoTDB's tree and table models, backed by the in-tree C++ Session SDK. The default transport is Session RPC, including verified TLS and mutual TLS; the existing HTTP REST transport remains available through RESTFUL=1.

Driver behavior and application compatibility

The driver provides the ODBC 3.8 ABI, forward-only/read-only cursors, column and rowset binding, explicit/implicit descriptors, prepared input parameters, parameter arrays, data-at-execution input, diagnostics, and capability discovery. Unsupported transactions, stored procedures, output parameters, bookmarks and asynchronous execution return explicit SQLSTATEs and are not advertised as supported.

The implementation includes the following compatibility fixes:

  • Export Unicode W entry points for connection strings, SQL, catalog arguments, descriptions, descriptors and diagnostics. STRING/TEXT metadata uses SQL_WVARCHAR/SQL_WLONGVARCHAR, so .NET and MSDASQL/ADO request UTF-16 data instead of interpreting UTF-8 with the Windows ANSI code page.
  • Build fixed, correctly typed SQLTables/SQLColumns results from every source batch. Catalog/schema/type enumeration, empty results, wildcard/escape matching, per-table column ordinals, DATE/BOOLEAN metadata and synthetic tree-model time columns remain consistent across fetches. Pattern arguments are matched locally rather than inserted into metadata SQL.
  • Preserve explicit time projections and dotted aliases in table queries. Correct DATE sizes, legacy numeric C-type binding used by ADO, empty-string reads, type-info cursor reuse, and type descriptions for empty/all-NULL catalog results.
  • Parse REST metadata with empty expressions, retain declared REST types, convert DATE/BLOB values, and keep non-query responses separate from result sets.
  • Propagate Session batch-read failures as errors and close exhausted Session operations once.

TLS configuration accepts SSL, SSLCA, SSLCERT and SSLKEY in connection strings, DSNs and Windows setup. CA verification and server identity checks are enabled; incomplete client credentials, certificate options with SSL disabled and TLS options combined with REST are rejected. Connection strings support braced values and escaped closing braces.

SQL_C_NUMERIC output currently supports exact integer/epoch values with precision/scale handling; FLOAT/DOUBLE conversion returns 07006 without changing the output. The README documents defaults and supported scale limits. Disconnect/free centrally drops connection-owned statement and descriptor resources before closing the Session; stale cleanup calls are checked against the live-handle registry.

Native build and packaging

with-odbc enables the Maven module; CMake/CPack can also build it directly. Release scope is Windows x64 and Linux x86_64 with a glibc 2.28 baseline. The driver links Session, Thrift, OpenSSL and curl statically, ships dependency licenses/NOTICE and build provenance, and does not require a separate C++ SDK, OpenSSL installation or VC redistributable. OS ODBC manager/installer libraries remain prerequisites.

The shared C++ build adds scoped static-link/CRT options for ODBC while retaining standalone SDK defaults. Linux releases use the shared manylinux/GCC baseline and check imported libraries and maximum GLIBC symbol versions. Windows packages check the PE architecture and imported DLLs. Windows source builds use valid generated batch-file line endings and also reject negative process exit codes. SDK Windows packaging uses Boost source headers rather than toolchain-specific binary installers.

GNU/Linux binaries incorporate GCC runtime object code under GPL-3.0-or-later WITH GCC-exception-3.1. No GCC runtime source or GPL license text is vendored in this source tree. Only GNU/Linux install/packaging obtains the upstream notices, verifies pinned SHA-256 hashes, and includes them under licenses/gcc with a binary LICENSE disclosure. Windows/MSVC packages contain neither notice. The README documents the ASF GCC Runtime Library exception policy and explicitly leaves the required inclusion review to the project PMC.

Validation

Current head 112f064 integrates upstream master a7b88b1, including the deletion-test flush-control fix; master Unit-Test CI passed all four jobs. The merge leaves ODBC/C++ SDK source and native workflows unchanged. Fork package/server CI and upstream ODBC CI both passed all eight jobs for this updated baseline, including real-server TLS/mTLS and Windows .NET/ADO consumers. All current upstream PR checks have completed: 54 passed, two optional analysis jobs (codecov/sonar) were skipped, and none failed or remain pending. The successful checks include C++ ARM packaging, Java tests, and AINode integration. The completed d227562 runs below verify the same native driver before this server-baseline update.

  • Clean full distribution build on JDK 17: mvn clean package -pl distribution -am -DskipTests.
  • Maven ODBC package lifecycle, CMake Release build, all four CTest suites, Spotless, Windows dependency checks, CPack ZIP/SHA-512 generation, and local manylinux 2.28 packaging. Maven RAT checks pass for both the parent client module and the ODBC module without a special ODBC license exclusion.
  • Extracted Windows release against a freshly built disposable 1C1D server: tree/table reads and writes, parameter arrays, small batch boundaries, REST, plain/TLS/mTLS connections, and negative certificate/identity cases.
  • Real Windows System.Data.Odbc and MSDASQL/ADO consumers: Unicode SQL/data, parameterized inserts, long text/emoji, NULL/empty strings, DATE/BOOLEAN, catalog browsing for both models, empty searches, and database changes. Tests use unique disposable databases and user DSNs, preserve normal registry-parent behavior, and stop only processes they launched.
  • Public-ABI HTTP-fixture regressions for catalog schemas/enumeration/patterns, Unicode lengths, REST types, empty strings and explicit time projection.
  • Latest fork ODBC package and integration CI and the upstream PR ODBC workflow both passed all eight jobs for d227562 after the nine review fixes, including extracted GNU/Linux/Windows packages against real servers in plain/TLS/mTLS modes, Windows .NET/ADO consumers, and clean runtime loading on AlmaLinux 8 and Ubuntu 22.04/24.04. Local Windows/Linux CTest 4/4, both CPack ZIPs, Linux glibc 2.28/dependency checks, Spotless/RAT, and real-server Windows .NET/ADO consumers also passed.
  • C++ SDK package CI: all eight package variants passed, covering Linux x86_64/aarch64, macOS x86_64/arm64, and VS2017/2019/2022/2026. This run uses commit ef42f10; the C++ SDK source and its packaging workflow/scripts are unchanged between that commit and the PR head.

Office verification scope: .NET and Office's MSDASQL/ADO data path were exercised. A Microsoft Excel/Power Query UI run has not been performed: the local Excel.Application registration points to WPS, and hosted CI runners do not include Excel. The README includes 64-bit Excel installation/import/refresh instructions and records this distinction. TIMESTAMP query columns retain epoch-integer semantics; DATE uses SQL_TYPE_DATE.

This PR has:

  • been self-reviewed.
  • added documentation for new features, supported behavior and verification scope.
  • included Apache headers and packaged dependency license/notice information.
  • added comments explaining non-obvious compatibility/build decisions.
  • added unit/public-ABI regression tests.
  • added integration and application-consumer tests.
  • been tested in disposable IoTDB 1C1D servers.

Key packages: iotdb-client/odbc, shared native build configuration under iotdb-client/client-cpp/cmake, and native package workflows/scripts.

Comment thread iotdb-client/odbc/licenses/GPL-3.0.txt Outdated

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the following correctness and robustness issues in the ODBC implementation:

  1. [P1] SQLDescribeCol can dereference a null DataTypePtr (driver.cpp:1664)

    DataTypePtr is optional, but the column-size path unconditionally evaluates *dataType. For example, SQLDescribeCol(stmt, 1, nullptr, 0, nullptr, nullptr, &columnSize, nullptr, nullptr) causes an access violation. Please obtain the type in a local variable whenever the column size is requested, or avoid dereferencing the optional pointer.

  2. [P1] SQLColAttribute writes the SQL type through the wrong pointer type (driver.cpp:1078)

    numericAttribute is an SQLLEN*, but it is reinterpreted as an SQLSMALLINT* before calling getColType. On 64-bit builds this returns negative type codes as large positive values (for example, SQL_WVARCHAR becomes 65527 instead of -9). Please write to an SQLSMALLINT local and then assign that value to the SQLLEN output.

  3. [P1/P2] SQL_C_NUMERIC silently truncates floating-point values (ODBCField.cpp:761-770)

    The FLOAT/DOUBLE conversion stores only the integer part, forces scale to zero, and still returns success. A value such as 1.5 is therefore returned as 1; setting a descriptor scale does not preserve the fractional part either. Please honor the requested precision/scale or reject unsupported conversions, and report the required fractional-truncation diagnostic (01S07) when applicable.

  4. [P1/P2] INT64_MIN cannot be converted correctly to SQL_C_NUMERIC (ODBCField.cpp:738)

    std::abs(INT64_MIN) cannot be represented as an int64_t; the current loop consequently leaves the numeric value as zero while returning success. Please calculate the magnitude in an unsigned type or handle INT64_MIN explicitly.

  5. [P2] Disconnect/free can leave child handles with dangling connection pointers (driver.cpp:1707, driver.cpp:2809, StatementHandle.h:104-108)

    SQLDisconnect only closes the session, and SQLFreeHandle(SQL_HANDLE_DBC, ...) deletes the connection while statements and implicit descriptors still retain raw pointers to it. A sequence such as allocating a statement, disconnecting, freeing the DBC, and then freeing the statement makes the statement destructor access freed memory. Please centrally close/delete/detach all child statements and descriptors before disconnecting or freeing the connection.

  6. [P2] SQLColAttribute(SQL_DESC_LABEL) does not support length-only calls (driver.cpp:1037-1044)

    getColName is called only when characterAttribute is non-null. A valid length-only call with CharacterAttributePtr == nullptr therefore leaves StringLengthPtr at zero instead of returning the label length. Please call getColName unconditionally for SQL_DESC_LABEL.

  7. [P2] A negative bufferLength can cause an invalid memset (driver.cpp:1020)

    The output buffer is cleared before validating bufferLength. Passing -1 with a non-null character buffer converts that value to a huge size_t for memset and causes an access violation. Please validate negative lengths first and return HY090 without touching the buffer.

  8. [P2] Column-name truncation returns SQL_SUCCESS (driver.cpp:471-485)

    getColName logs truncation but always returns SQL_SUCCESS. ODBC callers need SQL_SUCCESS_WITH_INFO plus the 01004 truncation diagnostic so that SQLDescribeCol and SQLColAttribute can report the shortened result correctly.

  9. [P2] Tree-model SQL_IDENTIFIER_QUOTE_CHAR returns a space (driver.cpp:3948-3955)

    The comment says the tree model uses backticks, but the implementation returns " ". This makes metadata-driven SQL generators quote tree identifiers incorrectly. Please return ` for the tree model.

I reproduced the crashes and metadata/value mismatches with direct ABI probes against the Windows driver artifact, and the existing ODBC compatibility and parameter tests do not cover these cases.

@hongzhi-gao

hongzhi-gao commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

@Caideyipi Addressed the nine items from your review in d227562:

  1. Optional SQLDescribeCol type pointer: resolve the type into a local whenever either type or column size is requested. A size-only request now works with DataTypePtr == nullptr, including the Unicode wrapper.
  2. SQLColAttribute type width/sign: obtain an SQLSMALLINT local and assign it to the full SQLLEN output. Regression cases include SQL_WVARCHAR, SQL_BIT, and SQL_BIGINT.
  3. FLOAT/DOUBLE to SQL_C_NUMERIC: removed the integer-part conversion. This conversion now explicitly returns SQL_ERROR/07006 and leaves the output untouched, including when an ARD scale is requested. Approximate values remain available through SQL_C_FLOAT/SQL_C_DOUBLE. Integer-to-numeric conversion honors bound-column and SQLGetData(SQL_ARD_TYPE) precision/scale, reports 22003 for overflow, and SQL_SUCCESS_WITH_INFO/01S07 when a negative scale discards digits. Direct SQLGetData(SQL_C_NUMERIC) uses the documented driver defaults (precision 38, scale 0).
  4. INT64_MIN: calculate its magnitude using unsigned subtraction, then populate the little-endian numeric bytes. Tests verify both 64-bit extrema and INT32_MIN without a floating-point intermediate.
  5. Child handle lifetime: centrally release statements/result sets and all implicit/explicit descriptors before closing the Session or destroying the DBC. Track live handle addresses so subsequent SQLFreeHandle cleanup calls return SQL_INVALID_HANDLE without reading freed memory. Tests cover shared explicit descriptors, implicit descriptors, disconnect-then-free-parent-then-free-child, and direct parent destruction.
  6. Label length probes: call getColName for SQL_DESC_LABEL even with a null output buffer; A/W length-only calls are covered.
  7. Negative buffer lengths: validate before clearing/copying output, return HY090, and preserve the caller's sentinel buffer.
  8. Column-name truncation: return SQL_SUCCESS_WITH_INFO and 01004; SQLDescribeCol still fills its other requested outputs before returning the warning. A/W paths and zero/short buffers are covered.
  9. Tree identifier quoting: return a backtick for tree-model SQL_IDENTIFIER_QUOTE_CHAR; table-model double quotes remain covered.

Validation completed locally: Windows and manylinux CTest 4/4 each, including new unit regressions and direct exported-ABI probes against the built DLL/shared library; Windows/Linux CPack; Linux GLIBC <= 2.28/dependency checks; Spotless and RAT. A real disposable server also passed the plain-mode Session checks plus Windows System.Data.Odbc and MSDASQL/ADO consumers. This is not a claim of an Excel/Power Query UI run.

The fork package/TLS/application CI and upstream PR ODBC workflow now both pass all eight jobs for d227562, including extracted-package real-server TLS/mTLS and Windows .NET/ADO tests.

The earlier DataNode baseline failures are now fixed upstream by a7b88b1 (master Unit-Test run: all four jobs passed). Integrated that baseline in merge commit 112f064; the ODBC/C++ SDK sources and native workflows are unchanged by this merge. The unchanged driver passed local Windows CTest 4/4 again. Fork ODBC CI and upstream ODBC CI now both pass all eight jobs for 112f064 against the updated server baseline, including extracted-package TLS/mTLS and Windows .NET/ADO consumers. All current upstream PR checks have now completed: 54 passed and two optional analysis jobs (codecov/sonar) were skipped, with no failures or pending jobs. This includes the fresh ARM C++ build, Java tests, and AINode integration checks. No unrelated Java fixes were authored in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants