Skip to content

Commit da725ca

Browse files
lfranckelfrancke
committed
docs: rewrite the README in a tighter voice
Cuts the prose by a third without dropping any facts. The length came from stacking two or three clauses of justification behind every claim, plus three facts stated twice: the deprecated ODBC 2.x rationale (What you get and Conformance), "the compiler names what is still missing" (What you get and Writing a driver), and UTF-16 conversion. Limits becomes a table, which is what that section always was. One sentence per line throughout, and `-` bullets to match every other document in the repo.
1 parent dddfeb0 commit da725ca

1 file changed

Lines changed: 97 additions & 179 deletions

File tree

‎README.md‎

Lines changed: 97 additions & 179 deletions
Original file line numberDiff line numberDiff line change
@@ -19,95 +19,61 @@
1919

2020
## What is this?
2121

22-
ODBC is the standard way desktop tools talk to a database. Excel, Tableau,
23-
Power BI and Python's pyodbc all speak it. Each database needs its own *driver*,
24-
a shared library the tool loads that translates those standard calls into
25-
whatever the database actually speaks.
26-
27-
Writing one is a large job, and most of it has nothing to do with your database.
28-
The driver has to hand out and validate handles, convert every string to and
29-
from UTF-16, report errors in the exact format the standard demands, copy values
30-
into buffers the application supplied, and not crash when the application lies
31-
about how big those buffers are.
32-
33-
`stackable-odbc-core` is that shared part, written once. What you supply is the
34-
part that really is about your database: how to connect and authenticate, how
35-
to run a query and read rows back, how your database's types map onto ODBC's,
36-
and how to answer the catalog questions. For a networked database that is a
37-
client library in its own right, and writing one is not trivial. It is just
38-
not ODBC work. One macro then generates the C entry points the standard
39-
requires.
40-
41-
This is a library rather than a driver you can load on its own. A working driver
42-
is this crate plus a backend, and
43-
[stackable-odbc-sqlite](https://github.com/stackabletech/stackable-odbc-sqlite)
44-
is the smallest complete example of one.
22+
This is a reusable crate to help develop ODBC drivers in Rust.
23+
24+
If you don't know what ODBC is, then this repo/crate will probably not be of interest to you.
25+
26+
There are other libraries for ODBC in Rust like [odbc-sys](https://crates.io/crates/odbc-sys) but they all focus on ODBC clients.
27+
This library is exclusively meant to fill the gap to write actual _ODBC drivers_.
28+
Writing a driver is obviously very specific to your target datastore but there is a lot of common stuff that this library handles:
29+
30+
- Hands out and validates handles
31+
- Converts every string to and from UTF-16
32+
- Reports errors in the exact format the standard demands
33+
- Copies values into buffers the application supplied (and handles errors gracefully)
34+
- ...and more
35+
36+
`stackable-odbc-core` is that shared part.
37+
You just need to supply the part that is really about your database:
38+
39+
- How to connect and authenticate
40+
- How to run a query, read rows back, map types onto ODBC's and so on...
41+
42+
A macro then generates all the necessary C plumbing the standard requires.
43+
44+
This is a library rather than a driver you can load on its own.
45+
A working driver is this crate plus a backend, and [stackable-odbc-sqlite](https://github.com/stackabletech/stackable-odbc-sqlite) is the smallest complete example of one.
4546

4647
## What you get
4748

48-
- **A database backend is two traits and one macro.** Implement `Backend` and
49-
`StatementBackend`, then call `forward_ffi!`. The compiler names everything
50-
still missing, so there is no list to work through by hand. Core holds no
51-
database-specific code, so a driver never forks or patches it.
52-
53-
- **A handle is a ticket number, not a memory address.** ODBC hands the
54-
application a `SQLHANDLE` that refers to a connection or a running query. The
55-
obvious implementation is a raw pointer, and then an application that frees a
56-
handle twice, or uses one after freeing it, corrupts the driver's memory.
57-
That is undefined behaviour, so the program may crash, or may quietly return a
58-
wrong answer.
59-
60-
Here a handle is a slot number plus a counter. The driver looks it up in its
61-
own table and never follows the pointer the application passed. Freeing bumps
62-
that slot's counter, so every ticket still referring to it stops matching.
63-
Use-after-free and double-free become a clean "invalid handle" error rather
64-
than memory corruption.
65-
66-
- **Two threads can share one connection safely.** The standard requires it,
67-
because "drivers must therefore support safe, multithread access to this
68-
information", and many drivers leave it to the Driver Manager instead. Each
69-
connection here has one lock, shared with every query started on it, so a call
70-
touching both a query and its connection takes a single lock. That leaves no
71-
lock ordering to get wrong, which is the usual way a driver deadlocks.
72-
`SQLCancel` takes no lock at all, because cancelling a slow query must not
73-
wait for the query it is cancelling.
74-
75-
- **The query timeout covers waiting for rows, not just sending the query.** An
76-
application sets `SQL_ATTR_QUERY_TIMEOUT` to say "give up after N seconds".
77-
Most drivers run that clock only while the query is being submitted, but a
78-
database can answer with the column names immediately and then take much
79-
longer to produce the first row. A timer covering only submission bounds
80-
nothing, so this one runs during `SQLFetch` as well.
81-
82-
- **Core builds the catalog answers.** For "what tables exist?" and its
83-
relatives, the standard dictates the exact columns, their order, and how the
84-
rows are sorted. A backend returns ordinary Rust structs with named fields,
85-
and core puts the columns in order, sorts the rows and normalises identifier
86-
case. You cannot get the column order or count wrong because you never write
87-
them, and a column added to one of those result sets is a change in core
88-
alone.
89-
90-
- **Value conversion is already done.** When an application supplies a parameter
91-
as text and asks for it to be treated as a number, the standard has three
92-
large tables saying exactly what each conversion does, down to which warning
93-
to raise when precision is lost. All three are implemented: character, binary
94-
and numeric, including the interval rows and the optional `01S07` warning for
95-
fractional seconds that were rounded away.
96-
97-
- **Windows is a first-class target.** Its Driver Manager is stricter than
98-
unixODBC and it fails quietly, so missing one requirement stops a feature
99-
working with no error to explain why. The known traps are handled: answering
100-
the version query it makes before connecting, reporting the complete function
101-
list it uses to build its dispatch table, and not exporting the deprecated
102-
ODBC 2.x functions, because exporting one replaces the Driver Manager's own
103-
better implementation with yours.
104-
105-
- **Checked by more than unit tests.** Three tools cover what ordinary tests
106-
cannot. Miri runs the code in an interpreter that detects undefined behaviour
107-
and leaked handles, loom re-runs the locking code under every thread
108-
interleaving rather than the one that happened to occur, and cargo-fuzz throws
109-
random input at the buffer-copying code under AddressSanitizer. All three run
110-
on every pull request, alongside the unit tests on Linux and Windows.
49+
- **We handle (sic!) the Handles.**
50+
If you don't know what a Handle is in ODBC-land you're lucky.
51+
A driver gets handed various Handles (e.g. `SQLHANDLE`) which are basically just pointers to memory holding its state.
52+
Ours is a slot number plus a counter looked up in our own table, so the pointer the application passed is never followed and a double free is a clean error instead of memory corruption.
53+
54+
- **Two threads can share one connection safely.**
55+
The standard requires it and many drivers leave it to the Driver Manager instead.
56+
This crate handles it correctly by using one lock per connection, shared with its queries, so there is no lock ordering left to get wrong.
57+
`SQLCancel` takes no lock at all, because cancelling a slow query must not wait for the query it is cancelling.
58+
59+
- **The query timeout covers waiting for rows, not just sending the query.**
60+
A database can answer with the column names immediately and take much longer to produce the first row, so `SQL_ATTR_QUERY_TIMEOUT` runs during `SQLFetch` too.
61+
62+
- **Core builds the catalog answers.**
63+
Return ordinary Rust structs with named fields and core puts the columns in the order the standard dictates, sorts the rows and normalises identifier case.
64+
You cannot get the column order or count wrong.
65+
66+
- **Value conversion is already done.**
67+
All three of the standard's conversion tables are implemented: character, binary and numeric, down to the interval rows and the optional `01S07` warning for rounded-away fractional seconds.
68+
69+
- **Windows is a first-class target.**
70+
Its Driver Manager is stricter than unixODBC and it fails quietly.
71+
Getting this correct is annoying.
72+
The known traps are handled and [AGENTS.md](https://github.com/stackabletech/stackable-odbc-core/blob/main/AGENTS.md) has the checklist.
73+
74+
- **Checked by more than unit tests.**
75+
Miri catches undefined behaviour and leaked handles, loom re-runs the locking code under every thread interleaving, and cargo-fuzz throws random input at the buffer copies under AddressSanitizer.
76+
All three run on every pull request, alongside the unit tests on Linux and Windows.
11177

11278
## Writing a driver
11379

@@ -116,124 +82,76 @@ cargo new --lib stackable-odbc-xyz
11682
cargo add stackable-odbc-core
11783
```
11884

119-
Implement `Backend` and `StatementBackend` for your database, then generate the
120-
C ABI in `lib.rs`:
85+
Implement `Backend` and `StatementBackend` for your database, then generate the C ABI in `lib.rs`:
12186

12287
```rust,ignore
12388
stackable_odbc_core::forward_ffi!(crate::backend::XyzBackend);
12489
```
12590

126-
That one line expands to every exported `SQL*` entry point, plus `ConfigDSNW` on
127-
Windows, each forwarding to the generic implementation in this crate.
128-
129-
`Backend` has four associated types and a body of required methods, but most of
130-
them are one-line capability declarations such as `supports_catalogs`,
131-
`identifier_case` and `sql_conformance`, each answering a single question about
132-
your database. They are required rather than defaulted on purpose: any default
133-
core supplied would be a claim about your database that nobody ever checked, and
134-
a wrong one is invisible, because the driver would confidently tell applications
135-
something untrue and nothing would complain. `StatementBackend` is the opposite,
136-
with one associated type and no required methods, so you override only what your
137-
backend supports.
138-
139-
In practice you do not look the list up. Write the four associated types, run
140-
`cargo check`, and the compiler names what is still missing.
141-
142-
Two traits and one macro bound the surface, not the effort. A backend for a
143-
real database is a real client. Authentication, sessions, type mapping, catalog
144-
queries and error mapping are all yours, and in both existing drivers that adds
145-
up to a substantial crate. What core takes off your hands is the ODBC half: the
146-
handle table, the UTF-16, the diagnostics format, the buffer copying and the
147-
conversion tables. That half is identical for every database, and it is the
148-
half where a mistake corrupts memory rather than returning a wrong answer.
149-
150-
[AGENTS.md](https://github.com/stackabletech/stackable-odbc-core/blob/main/AGENTS.md)
151-
has the full walkthrough: how a call flows through the layers, what each
152-
capability method means, the catalog and descriptor rules, and the Windows
153-
Driver Manager checklist.
91+
That one line expands to every exported `SQL*` entry point, plus `ConfigDSNW` on Windows.
92+
93+
Most of `Backend` is one-line capability declarations such as `supports_catalogs` and `identifier_case`.
94+
None of them are defaulted, because a default would be a claim about your database that nobody ever checked, and a wrong one is invisible.
95+
`StatementBackend` is the opposite and has no required methods, so you override only what your backend supports.
96+
97+
Don't look the list up.
98+
Write the four associated types, run `cargo check`, and the compiler names what is still missing.
99+
100+
[AGENTS.md](https://github.com/stackabletech/stackable-odbc-core/blob/main/AGENTS.md) has the full walkthrough: how a call flows through the layers, what each capability method means, and the catalog, descriptor and Windows rules.
154101

155102
## Conformance
156103

157-
This implements ODBC 3.80 at the `SQL_OIC_CORE` level, the base of the
158-
standard's three interface-conformance levels and the one an application may
159-
assume of any driver. All four handle types can be allocated and freed, and all
160-
five descriptor functions work. Descriptors are the standard's own way of
161-
describing a bound column or parameter, and one can be shared between queries on
162-
a connection.
163-
164-
This is a Unicode driver: every function that takes or returns a string is
165-
exported only in its wide (`W`-suffixed) form e.g. `SQLConnectW`. The
166-
Driver Manager translates for ANSI applications, so they keep working and the
167-
driver never carries a second set of entry points. Functions with no strings
168-
in their signature, such as `SQLFetch`, have one spelling and are exported
169-
unsuffixed.
170-
171-
`CORE_EXPORTED_FUNCTIONS` in `src/function_id.rs` is the authoritative list of
172-
what is exported, and a guard test pins every entry to a symbol that exists. The
173-
deprecated ODBC 2.x functions are left out, because the Driver Manager already
174-
emulates them on top of the modern ones and usually does it better than a driver
175-
would, so exporting your own version switches that off rather than adding
176-
anything. `SQLExtendedFetch` is the exception the Driver Manager does not map,
177-
so core exports it.
104+
ODBC 3.80 at the `SQL_OIC_CORE` level, which is the base of the standard's three interface-conformance levels and the one an application may assume of any driver.
105+
All four handle types and all five descriptor functions work, and a descriptor can be shared between queries on a connection.
106+
107+
This is a Unicode driver, so anything taking or returning a string is exported only in its wide form (`SQLConnectW`) and the Driver Manager translates for ANSI applications.
108+
Functions with no strings in their signature, such as `SQLFetch`, have one spelling and are exported unsuffixed.
109+
110+
`CORE_EXPORTED_FUNCTIONS` in `src/function_id.rs` is the authoritative list, pinned by a guard test.
111+
The deprecated ODBC 2.x functions are absent on purpose: the Driver Manager already emulates them on top of the modern ones.
112+
`SQLExtendedFetch` is the one the Driver Manager does not map, so core exports it.
178113

179114
## Limits
180115

181-
Each of these is reported to the application as unsupported rather than quietly
182-
ignored, so a tool can react instead of trusting a wrong answer.
183-
184-
- **Results are read front to back only** (`SQL_SO_FORWARD_ONLY`), so there is
185-
no jumping to a row and no going backwards. `SQLFetchScroll` accepts
186-
`SQL_FETCH_NEXT` and rejects every other direction with `HY106`.
187-
- **One row at a time.** There are no block cursors, so
188-
`SQL_ATTR_ROW_ARRAY_SIZE` is fixed at 1. Asking for more returns 1 with an
189-
`01S02` warning, and `SQL_GD_BLOCK` is never reported.
190-
- **No bookmarks**, which are saved row positions an application can return to
191-
later, and no automatic population of parameter metadata, so
192-
`SQL_ATTR_AUTO_IPD` stays `SQL_FALSE`.
193-
- **No async.** Every call runs to completion before returning:
194-
`SQL_ASYNC_MODE` is reported as `SQL_AM_NONE`, and turning on
195-
`SQL_ATTR_ASYNC_ENABLE` is refused rather than ignored. This is about the
196-
calling thread, not the shape of the results. Rows still arrive one
197-
`SQLFetch` at a time, the query timeout still bounds a slow query, and
198-
`SQLCancel` still interrupts one. `Backend` is synchronous too, so a driver
199-
built on an async client library bridges to it internally, for example with
200-
a current-thread tokio runtime and `block_on`.
116+
Each of these limits is actually reported back to an application that tries to use one of these features so they can react to it.
117+
118+
| Not supported | What the application sees |
119+
|---|---|
120+
| Scrollable cursors | `SQL_SO_FORWARD_ONLY`; `SQLFetchScroll` takes `SQL_FETCH_NEXT` and rejects the rest with `HY106` |
121+
| Block cursors | `SQL_ATTR_ROW_ARRAY_SIZE` fixed at 1, returning 1 with an `01S02` warning; `SQL_GD_BLOCK` never reported |
122+
| Bookmarks, automatic parameter metadata | `SQL_ATTR_AUTO_IPD` stays `SQL_FALSE` |
123+
| Async | `SQL_AM_NONE`; `SQL_ATTR_ASYNC_ENABLE` is refused, not ignored |
124+
125+
Async here means the calling thread, not the shape of the results.
126+
Rows still arrive one `SQLFetch` at a time, the query timeout still bounds a slow query, and `SQLCancel` still interrupts one.
127+
`Backend` is synchronous too, so a driver built on an async client library bridges to it internally, for example with a current-thread tokio runtime and `block_on`.
201128

202129
## Drivers built on this crate
203130

204-
Each driver is a separate crate supplying only its `Backend` and
205-
`StatementBackend` implementation.
131+
Each is a separate crate supplying only its `Backend` and `StatementBackend` implementation.
206132

207-
- [stackable-odbc-trino](https://github.com/stackabletech/stackable-odbc-trino),
208-
an ODBC driver for [Trino](https://trino.io/).
209-
- [stackable-odbc-sqlite](https://github.com/stackabletech/stackable-odbc-sqlite),
210-
a SQLite driver, used as a worked example and as the test driver for the
211-
framework itself.
133+
- [stackable-odbc-trino](https://github.com/stackabletech/stackable-odbc-trino), an ODBC driver for [Trino](https://trino.io/).
134+
- [stackable-odbc-sqlite](https://github.com/stackabletech/stackable-odbc-sqlite), a SQLite driver, used as a worked example and as the test driver for the framework itself.
212135

213136
## Resources
214137

215-
- [ODBC API reference](https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/odbc-api-reference?view=sql-server-ver16),
216-
the authoritative specification. It is the most detailed source and still not
217-
an easy read.
218-
- [Header files](https://github.com/microsoft/ODBC-Specification/blob/master/Windows/inc/sql.h)
219-
for the unreleased ODBC 4 standard, mostly valid for the older ones too.
220-
- [odbc-sys](https://github.com/pacman82/odbc-sys), the ODBC type definitions
221-
this crate builds on.
138+
- [ODBC API reference](https://learn.microsoft.com/en-us/sql/odbc/reference/syntax/odbc-api-reference?view=sql-server-ver16), the authoritative specification.
139+
It is the most detailed source and still not an easy read.
140+
- [Header files](https://github.com/microsoft/ODBC-Specification/blob/master/Windows/inc/sql.h) for the unreleased ODBC 4 standard, mostly valid for the older ones too.
141+
- [odbc-sys](https://github.com/pacman82/odbc-sys), the ODBC type definitions this crate builds on. Thank you!
222142

223143
## Getting help
224144

225-
- [GitHub Discussions](https://github.com/orgs/stackabletech/discussions) for
226-
questions
145+
- [GitHub Discussions](https://github.com/orgs/stackabletech/discussions) for questions
227146
- [Discord](https://discord.gg/7kZ3BNnCAF) to talk to us
228-
- [Issues](https://github.com/stackabletech/stackable-odbc-core/issues) for
229-
bugs, and [SECURITY.md](SECURITY.md) for anything security-related
147+
- [Issues](https://github.com/stackabletech/stackable-odbc-core/issues) for bugs, and [SECURITY.md](SECURITY.md) for anything security-related
230148

231149
## Contributing
232150

233-
See [CONTRIBUTING.md](CONTRIBUTING.md) for building from source, running the
234-
tests, and how the repository is laid out. [CHANGELOG.md](CHANGELOG.md) records
235-
what changed in each release.
151+
See [CONTRIBUTING.md](CONTRIBUTING.md) for building from source, running the tests, and how the repository is laid out.
152+
[CHANGELOG.md](CHANGELOG.md) records what changed in each release.
236153

237154
## License
238155

239-
Apache-2.0. See [LICENSE](./LICENSE) and [NOTICE](./NOTICE).
156+
Apache-2.0.
157+
See [LICENSE](./LICENSE) and [NOTICE](./NOTICE).

0 commit comments

Comments
 (0)