diff --git a/.github/workflows/pr-workflow.yaml b/.github/workflows/pr-workflow.yaml index cdb0436a10..f7e171797e 100644 --- a/.github/workflows/pr-workflow.yaml +++ b/.github/workflows/pr-workflow.yaml @@ -53,13 +53,29 @@ jobs: go-version-file: 'go.mod' # Unconditional: a released migration becoming permanent is exactly what # this guards, and that happens on the push to main, not on the PR. - - name: Verify immutable PostgreSQL migrations - run: hack/verify/postgresql-migrations.sh + - name: Verify immutable SQL migrations + run: hack/verify/sql-migrations.sh # gotestsum wraps go test to emit JUnit; the go test arguments are unchanged. - name: unit tests run: hack/run-tool.sh gotestsum --junitfile "${ARTIFACTS}/unit.xml" --jsonfile "${ARTIFACTS}/unit.json" --format standard-verbose -- -race -v ./... + # The unit tests above run the store-backed control plane suites on + # PostgreSQL; rerun them on MySQL. atemy's own tests already ran above. + - name: MySQL control plane tests + env: + ATE_TEST_STORE_BACKEND: mysql + run: hack/run-tool.sh gotestsum --junitfile "${ARTIFACTS}/unit-mysql.xml" + --jsonfile "${ARTIFACTS}/unit-mysql.json" --format standard-verbose + -- -race -v ./cmd/ateapi/internal/controlapi/... ./cmd/ateapi/internal/workerservice/... + # PlanetScale serves MySQL through Vitess, which rejects or rewrites some + # statements plain MySQL accepts. The vitess build tag keeps the large + # amd64-only vttestserver image out of the default test run. + - name: Vitess store tests + run: hack/run-tool.sh gotestsum --junitfile "${ARTIFACTS}/unit-vitess.xml" + --jsonfile "${ARTIFACTS}/unit-vitess.json" --format standard-verbose + -- -race -v -tags vitess -run '^TestVitess$' ./cmd/ateapi/internal/store/atemy/ + && grep -q '"Action":"pass".*"Test":"TestVitess"' "${ARTIFACTS}/unit-vitess.json" # tools/apitool has its own module so we need to run it explicitly. # run-tool.sh resolves the repo root itself, so it works from the subdir. - name: apitool tests diff --git a/AGENTS.md b/AGENTS.md index 036414703c..3c3af89fbf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -94,6 +94,7 @@ See the [metric registry](docs/observability.md#the-metric-registry) section of 3. Run `make verify` locally before requesting a code review to catch common issues like missed copyright headers or formatting drift. 4. For end-to-end tests involving the actual infrastructure, ensure you have a running cluster (setup via `hack/ate-dev-env.sh.example` and `go run ./tools/setup-gcp bootstrap`). 5. A test that skips when a precondition is missing — Docker, a cluster, an artifact directory — must **fail** on it in CI, because a skip and a pass are the same exit code. Resolve strictness through a named predicate rather than an inline `os.Getenv("CI")`; `cmd/ateapi/internal/store/dockerenv.Required()` is the reference implementation. Prove the strict branch red before merging: a guard only ever observed passing is not known to guard anything. See `docs/dev/best-practices/ci-fail-closed.md`. +6. The store has a PostgreSQL backend (`cmd/ateapi/internal/store/atepg`) and a MySQL backend (`cmd/ateapi/internal/store/atemy`). Make every store or schema change in both, following `docs/dev/schema-evolution.md`. Store-backed suites use PostgreSQL by default; run them with `ATE_TEST_STORE_BACKEND=mysql` too, as CI does for `./cmd/ateapi/internal/controlapi/...` and `./cmd/ateapi/internal/workerservice/...`. ## Security Considerations diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7668a5d809..be6e14da8d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -86,7 +86,7 @@ for this purpose. All code changes should be accompanied by tests. We will not merge code that does not have tests, and we will not merge code that causes tests to fail. -Follow the [PostgreSQL schema evolution rules](docs/dev/postgresql-schema-evolution.md) for each application schema change. +Follow the [schema evolution rules](docs/dev/schema-evolution.md) for each application schema change. Make each change in both the PostgreSQL and MySQL migrations. Each pull request title becomes its line in the [release notes](docs/dev/release-notes.md). diff --git a/README.md b/README.md index 2be4d5ce92..9dd2542f8d 100644 --- a/README.md +++ b/README.md @@ -192,6 +192,7 @@ go run ./tools/setup-gcp create bucket To run the PostgreSQL store backend on Cloud SQL — with IAM database authentication and no passwords — see [tools/setup-gcp/cloud-sql.md](tools/setup-gcp/cloud-sql.md). +To use an external MySQL 8.0+ or PlanetScale database instead, see [docs/mysql.md](docs/mysql.md). Similarly, you can deploy or cleanup specific Agent Substrate components using the installation script. See `./hack/install-ate.sh --help` for all options. ```bash diff --git a/_LICENSES/filippo.io/edwards25519/LICENSE b/_LICENSES/filippo.io/edwards25519/LICENSE new file mode 100644 index 0000000000..6a66aea5ea --- /dev/null +++ b/_LICENSES/filippo.io/edwards25519/LICENSE @@ -0,0 +1,27 @@ +Copyright (c) 2009 The Go Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/_LICENSES/github.com/go-sql-driver/mysql/.github/CONTRIBUTING.md b/_LICENSES/github.com/go-sql-driver/mysql/.github/CONTRIBUTING.md new file mode 100644 index 0000000000..8fe16bcb49 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/.github/CONTRIBUTING.md @@ -0,0 +1,23 @@ +# Contributing Guidelines + +## Reporting Issues + +Before creating a new Issue, please check first if a similar Issue [already exists](https://github.com/go-sql-driver/mysql/issues?state=open) or was [recently closed](https://github.com/go-sql-driver/mysql/issues?direction=desc&page=1&sort=updated&state=closed). + +## Contributing Code + +By contributing to this project, you share your code under the Mozilla Public License 2, as specified in the LICENSE file. +Don't forget to add yourself to the AUTHORS file. + +### Code Review + +Everyone is invited to review and comment on pull requests. +If it looks fine to you, comment with "LGTM" (Looks good to me). + +If changes are required, notice the reviewers with "PTAL" (Please take another look) after committing the fixes. + +Before merging the Pull Request, at least one [team member](https://github.com/go-sql-driver?tab=members) must have commented with "LGTM". + +## Development Ideas + +If you are looking for ideas for code contributions, please check our [Development Ideas](https://github.com/go-sql-driver/mysql/wiki/Development-Ideas) Wiki page. diff --git a/_LICENSES/github.com/go-sql-driver/mysql/.github/ISSUE_TEMPLATE.md b/_LICENSES/github.com/go-sql-driver/mysql/.github/ISSUE_TEMPLATE.md new file mode 100644 index 0000000000..d9771f1ddc --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/.github/ISSUE_TEMPLATE.md @@ -0,0 +1,21 @@ +### Issue description +Tell us what should happen and what happens instead + +### Example code +```go +If possible, please enter some example code here to reproduce the issue. +``` + +### Error log +``` +If you have an error log, please paste it here. +``` + +### Configuration +*Driver version (or git SHA):* + +*Go version:* run `go version` in your console + +*Server version:* E.g. MySQL 5.6, MariaDB 10.0.20 + +*Server OS:* E.g. Debian 8.1 (Jessie), Windows 10 diff --git a/_LICENSES/github.com/go-sql-driver/mysql/.github/PULL_REQUEST_TEMPLATE.md b/_LICENSES/github.com/go-sql-driver/mysql/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000000..6f5c7ebeb7 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,9 @@ +### Description +Please explain the changes you made here. + +### Checklist +- [ ] Code compiles correctly +- [ ] Created tests which fail without the change (if possible) +- [ ] All tests passing +- [ ] Extended the README / documentation, if necessary +- [ ] Added myself / the copyright holder to the AUTHORS file diff --git a/_LICENSES/github.com/go-sql-driver/mysql/.github/dependabot.yml b/_LICENSES/github.com/go-sql-driver/mysql/.github/dependabot.yml new file mode 100644 index 0000000000..5e513f2760 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/.github/dependabot.yml @@ -0,0 +1,24 @@ +# To get started with Dependabot version updates, you'll need to specify which +# package ecosystems to update and where the package manifests are located. +# Please see the documentation for all configuration options: +# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file + +version: 2 +updates: + - package-ecosystem: "gomod" + directory: "/" + schedule: + interval: "weekly" + groups: + all-dependencies: + patterns: + - "*" + + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + groups: + all-dependencies: + patterns: + - "*" diff --git a/_LICENSES/github.com/go-sql-driver/mysql/.github/workflows/test.yml b/_LICENSES/github.com/go-sql-driver/mysql/.github/workflows/test.yml new file mode 100644 index 0000000000..592b644bcb --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/.github/workflows/test.yml @@ -0,0 +1,118 @@ +name: test +on: + pull_request: + push: + workflow_dispatch: + +env: + MYSQL_TEST_USER: gotest + MYSQL_TEST_PASS: secret + MYSQL_TEST_ADDR: 127.0.0.1:3306 + MYSQL_TEST_CONCURRENT: 1 + +jobs: + lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: dominikh/staticcheck-action@9716614d4101e79b4340dd97b10e54d68234e431 # v1.4.1 + + list: + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.set-matrix.outputs.matrix }} + steps: + - name: list + id: set-matrix + run: | + import json + import os + go = [ + # Keep the most recent production release at the top + '1.26', + # Older production releases + '1.25', + '1.24', + ] + mysql = [ + '9.0', + '8.4', # LTS + '8.0', + '5.7', + 'mariadb-12.3', # LTS + 'mariadb-11.8', # LTS + 'mariadb-11.4', # LTS + 'mariadb-10.11', # LTS + 'mariadb-10.6', # LTS + 'mariadb-10.5', # LTS + ] + + includes = [] + # Go versions compatibility check + for v in go[1:]: + includes.append({'os': 'ubuntu-latest', 'go': v, 'mysql': mysql[0]}) + + matrix = { + # OS vs MySQL versions + 'os': [ 'ubuntu-latest', 'macos-latest', 'windows-latest' ], + 'go': [ go[0] ], + 'mysql': mysql, + + 'include': includes + } + output = json.dumps(matrix, separators=(',', ':')) + with open(os.environ["GITHUB_OUTPUT"], 'a', encoding="utf-8") as f: + print(f"matrix={output}", file=f) + shell: python + test: + needs: list + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.list.outputs.matrix) }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ matrix.go }} + - uses: shogo82148/actions-setup-mysql@62da9377d83991fce27b6ed2a397d3306121e41d # v1.53.1 + with: + mysql-version: ${{ matrix.mysql }} + user: ${{ env.MYSQL_TEST_USER }} + password: ${{ env.MYSQL_TEST_PASS }} + my-cnf: | + innodb_log_file_size=256MB + innodb_buffer_pool_size=512MB + max_allowed_packet=48MB + ; TestConcurrent fails if max_connections is too large + max_connections=50 + local_infile=1 + performance_schema=on + - name: setup database + run: | + mysql --user 'root' --host '127.0.0.1' -e 'create database gotest;' + + - name: test + run: | + go test -v '-race' '-covermode=atomic' '-coverprofile=coverage.out' -parallel 10 + + - name: benchmark + run: | + go test -run '^$' -bench . + + - name: Send coverage + uses: shogo82148/actions-goveralls@8781f5dd05b691c4dd042d5e859c11c73e0104fa # v1.11.1 + with: + path-to-profile: coverage.out + flag-name: ${{ runner.os }}-Go-${{ matrix.go }}-DB-${{ matrix.mysql }} + parallel: true + + # notifies that all test jobs are finished. + finish: + needs: test + if: always() + runs-on: ubuntu-latest + steps: + - uses: shogo82148/actions-goveralls@8781f5dd05b691c4dd042d5e859c11c73e0104fa # v1.11.1 + with: + parallel-finished: true diff --git a/_LICENSES/github.com/go-sql-driver/mysql/.gitignore b/_LICENSES/github.com/go-sql-driver/mysql/.gitignore new file mode 100644 index 0000000000..2de28da166 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/.gitignore @@ -0,0 +1,9 @@ +.DS_Store +.DS_Store? +._* +.Spotlight-V100 +.Trashes +Icon? +ehthumbs.db +Thumbs.db +.idea diff --git a/_LICENSES/github.com/go-sql-driver/mysql/AUTHORS b/_LICENSES/github.com/go-sql-driver/mysql/AUTHORS new file mode 100644 index 0000000000..37c96208a5 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/AUTHORS @@ -0,0 +1,160 @@ +# This is the official list of Go-MySQL-Driver authors for copyright purposes. + +# If you are submitting a patch, please add your name or the name of the +# organization which holds the copyright to this list in alphabetical order. + +# Names should be added to this file as +# Name +# The email address is not required for organizations. +# Please keep the list sorted. + + +# Individual Persons + +Aaron Hopkins +Achille Roussel +Aidan +Alex Snast +Alexey Palazhchenko +Andrew Reid +Animesh Ray +Ariel Mashraki +Arne Hormann +Artur Melanchyk +Asta Xie +B Lamarche +Bes Dollma +Bogdan Constantinescu +Brad Higgins +Brian Hendriks +Bulat Gaifullin +Caine Jette +Carlos Nieto +Chris Kirkland +Chris Moos +Craig Wilson +Daemonxiao <735462752 at qq.com> +Daniel Montoya +Daniel Nichter +Daniël van Eeden +Dave Protasowski +Demouth +Diego Dupin +Dirkjan Bussink +DisposaBoy +Egor Smolyakov +Erwan Martin +Evan Elias +Evan Shaw +Frederick Mayle +Gustavo Kristic +Gusted +Hajime Nakagami +Hanno Braun +Henri Yandell +Hirotaka Yamamoto +Huyiguang +ICHINOSE Shogo +Ilia Cimpoes +INADA Naoki +Jacek Szwec +Jakub Adamus +James Harr +Janek Vedock +Jason Ng +Jean-Yves Pellé +Jeff Hodges +Jeffrey Charles +Jennifer Purevsuren +Jerome Meyer +Jiabin Zhang +Jiajia Zhong +Jian Zhen +Joe Mann +Joshua Prunier +Julien Lefevre +Julien Schmidt +Justin Li +Justin Nuß +Kamil Dziedzic +Kei Kamikawa +Kevin Malachowski +Kieron Woodhouse +Lance Tian +Lennart Rudolph +Leonardo YongUk Kim +Linh Tran Tuan +Lion Yang +Luca Looz +Lucas Liu +Luke Scott +Lunny Xiao +Maciej Zimnoch +Michael Woolnough +Minh Quang +Morgan Tocker +Nao Yokotsuka +Nathanial Murphy +Nicola Peduzzi +Oliver Bone +Olivier Mengué +oscarzhao +Paul Bonser +Paulius Lozys +Peter Schultz +Phil Porada +Rebecca Chin +Reed Allman +Richard Wilkes +Robert Russell +Runrioter Wung +Samantha Frank +Santhosh Kumar Tekuri +Sho Iizuka +Sho Ikeda +Shuode Li +Simon J Mudd +Soroush Pour +Stan Putrya +Stanley Gunawan +Steven Hartland +Tan Jinhua <312841925 at qq.com> +Tetsuro Aoki +Thomas Wodarek +Tim Ruffles +Tom Jenkinson +Vladimir Kovpak +Vladyslav Zhelezniak +Xiangyu Hu +Xiaobing Jiang +Xiuming Chen +Xuehong Chan +Yoshito Ohata +Zhang Xiang +Zhenye Xie +Zhixin Wen +Ziheng Lyu + +# Organizations + +Barracuda Networks, Inc. +Block, Inc. +Counting Ltd. +Defined Networking Inc. +DigitalOcean Inc. +Dolthub Inc. +dyves labs AG +Facebook Inc. +GitHub Inc. +Google Inc. +InfoSum Ltd. +Keybase Inc. +Microsoft Corp. +Multiplay Ltd. +Percona LLC +PingCAP Inc. +Pivotal Inc. +Shattered Silicon Ltd. +Stripe Inc. +ThousandEyes +Zendesk Inc. diff --git a/_LICENSES/github.com/go-sql-driver/mysql/CHANGELOG.md b/_LICENSES/github.com/go-sql-driver/mysql/CHANGELOG.md new file mode 100644 index 0000000000..72a35516dc --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/CHANGELOG.md @@ -0,0 +1,383 @@ +# Changelog + +## v1.10.1 (2026-09-02) + +* Fix `Config.FormatDSN()` dropping `Addr` when `Net` is empty. + It now uses the default `tcp` network so configs with only `Addr` round-trip correctly. (#1770) + +* Fix typed-nil `json.RawMessage` with `interpolateParams=true` being interpolated as an empty string. + It is now interpolated as SQL `NULL`, matching server-side prepared statements. (#1782) + +* Add MariaDB 11.8 and 12.3 to the test matrix. (#1774) + +## v1.10.0 (2026-04-28) + +* Fix `getSystemVar("max_allowed_packet")` potentially returned wrong value. (#1754) + This affects only when `maxAllowedPacket=0` is set. + +* Bump filippo.io/edwards25519 from 1.1.1 to 1.2.0. (#1756) + While older versions have reported CVEs, they do not affect go-mysql. + +* Update Go versions to 1.24-1.26. (#1763) + +* Enhance interpolateParams to correctly handle placeholders. (#1732) + The question mark (?) within strings and comments will no longer be treated as a placeholder. + + +## v1.9.3 (2025-06-13) + +* `tx.Commit()` and `tx.Rollback()` returned `ErrInvalidConn` always. + Now they return cached real error if present. (#1690) + +* Optimize reading small result sets to fix a performance regression + introduced by compression protocol support. (`#1707`) +* Fix `db.Ping()` on compressed connection. (#1723) + + +## v1.9.2 (2025-04-07) + +v1.9.2 is a re-release of v1.9.1 due to a release process issue; no changes were made to the content. + + +## v1.9.1 (2025-03-21) + +### Major Changes + +* Add Charset() option. (#1679) + +### Bugfixes + +* go.mod: fix go version format (#1682) +* Fix FormatDSN missing ConnectionAttributes (#1619) + +## v1.9.0 (2025-02-18) + +### Major Changes + +- Implement zlib compression. (#1487) +- Supported Go version is updated to Go 1.21+. (#1639) +- Add support for VECTOR type introduced in MySQL 9.0. (#1609) +- Config object can have custom dial function. (#1527) + +### Bugfixes + +- Fix auth errors when username/password are too long. (#1625) +- Check if MySQL supports CLIENT_CONNECT_ATTRS before sending client attributes. (#1640) +- Fix auth switch request handling. (#1666) + +### Other changes + +- Add "filename:line" prefix to log in go-mysql. Custom loggers now show it. (#1589) +- Improve error handling. It reduces the "busy buffer" errors. (#1595, #1601, #1641) +- Use `strconv.Atoi` to parse max_allowed_packet. (#1661) +- `rejectReadOnly` option now handles ER_READ_ONLY_MODE (1290) error too. (#1660) + + +## Version 1.8.1 (2024-03-26) + +Bugfixes: + +- fix race condition when context is canceled in [#1562](https://github.com/go-sql-driver/mysql/pull/1562) and [#1570](https://github.com/go-sql-driver/mysql/pull/1570) + +## Version 1.8.0 (2024-03-09) + +Major Changes: + +- Use `SET NAMES charset COLLATE collation`. by @methane in [#1437](https://github.com/go-sql-driver/mysql/pull/1437) + - Older go-mysql-driver used `collation_id` in the handshake packet. But it caused collation mismatch in some situation. + - If you don't specify charset nor collation, go-mysql-driver sends `SET NAMES utf8mb4` for new connection. This uses server's default collation for utf8mb4. + - If you specify charset, go-mysql-driver sends `SET NAMES `. This uses the server's default collation for ``. + - If you specify collation and/or charset, go-mysql-driver sends `SET NAMES charset COLLATE collation`. +- PathEscape dbname in DSN. by @methane in [#1432](https://github.com/go-sql-driver/mysql/pull/1432) + - This is backward incompatible in rare case. Check your DSN. +- Drop Go 1.13-17 support by @methane in [#1420](https://github.com/go-sql-driver/mysql/pull/1420) + - Use Go 1.18+ +- Parse numbers on text protocol too by @methane in [#1452](https://github.com/go-sql-driver/mysql/pull/1452) + - When text protocol is used, go-mysql-driver passed bare `[]byte` to database/sql for avoid unnecessary allocation and conversion. + - If user specified `*any` to `Scan()`, database/sql passed the `[]byte` into the target variable. + - This confused users because most user doesn't know when text/binary protocol used. + - go-mysql-driver 1.8 converts integer/float values into int64/double even in text protocol. This doesn't increase allocation compared to `[]byte` and conversion cost is negatable. +- New options start using the Functional Option Pattern to avoid increasing technical debt in the Config object. Future version may introduce Functional Option for existing options, but not for now. + - Make TimeTruncate functional option by @methane in [1552](https://github.com/go-sql-driver/mysql/pull/1552) + - Add BeforeConnect callback to configuration object by @ItalyPaleAle in [#1469](https://github.com/go-sql-driver/mysql/pull/1469) + + +Other changes: + +- Adding DeregisterDialContext to prevent memory leaks with dialers we don't need anymore by @jypelle in https://github.com/go-sql-driver/mysql/pull/1422 +- Make logger configurable per connection by @frozenbonito in https://github.com/go-sql-driver/mysql/pull/1408 +- Fix ColumnType.DatabaseTypeName for mediumint unsigned by @evanelias in https://github.com/go-sql-driver/mysql/pull/1428 +- Add connection attributes by @Daemonxiao in https://github.com/go-sql-driver/mysql/pull/1389 +- Stop `ColumnTypeScanType()` from returning `sql.RawBytes` by @methane in https://github.com/go-sql-driver/mysql/pull/1424 +- Exec() now provides access to status of multiple statements. by @mherr-google in https://github.com/go-sql-driver/mysql/pull/1309 +- Allow to change (or disable) the default driver name for registration by @dolmen in https://github.com/go-sql-driver/mysql/pull/1499 +- Add default connection attribute '_server_host' by @oblitorum in https://github.com/go-sql-driver/mysql/pull/1506 +- QueryUnescape DSN ConnectionAttribute value by @zhangyangyu in https://github.com/go-sql-driver/mysql/pull/1470 +- Add client_ed25519 authentication by @Gusted in https://github.com/go-sql-driver/mysql/pull/1518 + +## Version 1.7.1 (2023-04-25) + +Changes: + + - bump actions/checkout@v3 and actions/setup-go@v3 (#1375) + - Add go1.20 and mariadb10.11 to the testing matrix (#1403) + - Increase default maxAllowedPacket size. (#1411) + +Bugfixes: + + - Use SET syntax as specified in the MySQL documentation (#1402) + + +## Version 1.7 (2022-11-29) + +Changes: + + - Drop support of Go 1.12 (#1211) + - Refactoring `(*textRows).readRow` in a more clear way (#1230) + - util: Reduce boundary check in escape functions. (#1316) + - enhancement for mysqlConn handleAuthResult (#1250) + +New Features: + + - support Is comparison on MySQLError (#1210) + - return unsigned in database type name when necessary (#1238) + - Add API to express like a --ssl-mode=PREFERRED MySQL client (#1370) + - Add SQLState to MySQLError (#1321) + +Bugfixes: + + - Fix parsing 0 year. (#1257) + + +## Version 1.6 (2021-04-01) + +Changes: + + - Migrate the CI service from travis-ci to GitHub Actions (#1176, #1183, #1190) + - `NullTime` is deprecated (#960, #1144) + - Reduce allocations when building SET command (#1111) + - Performance improvement for time formatting (#1118) + - Performance improvement for time parsing (#1098, #1113) + +New Features: + + - Implement `driver.Validator` interface (#1106, #1174) + - Support returning `uint64` from `Valuer` in `ConvertValue` (#1143) + - Add `json.RawMessage` for converter and prepared statement (#1059) + - Interpolate `json.RawMessage` as `string` (#1058) + - Implements `CheckNamedValue` (#1090) + +Bugfixes: + + - Stop rounding times (#1121, #1172) + - Put zero filler into the SSL handshake packet (#1066) + - Fix checking cancelled connections back into the connection pool (#1095) + - Fix remove last 0 byte for mysql_old_password when password is empty (#1133) + + +## Version 1.5 (2020-01-07) + +Changes: + + - Dropped support Go 1.9 and lower (#823, #829, #886, #1016, #1017) + - Improve buffer handling (#890) + - Document potentially insecure TLS configs (#901) + - Use a double-buffering scheme to prevent data races (#943) + - Pass uint64 values without converting them to string (#838, #955) + - Update collations and make utf8mb4 default (#877, #1054) + - Make NullTime compatible with sql.NullTime in Go 1.13+ (#995) + - Removed CloudSQL support (#993, #1007) + - Add Go Module support (#1003) + +New Features: + + - Implement support of optional TLS (#900) + - Check connection liveness (#934, #964, #997, #1048, #1051, #1052) + - Implement Connector Interface (#941, #958, #1020, #1035) + +Bugfixes: + + - Mark connections as bad on error during ping (#875) + - Mark connections as bad on error during dial (#867) + - Fix connection leak caused by rapid context cancellation (#1024) + - Mark connections as bad on error during Conn.Prepare (#1030) + + +## Version 1.4.1 (2018-11-14) + +Bugfixes: + + - Fix TIME format for binary columns (#818) + - Fix handling of empty auth plugin names (#835) + - Fix caching_sha2_password with empty password (#826) + - Fix canceled context broke mysqlConn (#862) + - Fix OldAuthSwitchRequest support (#870) + - Fix Auth Response packet for cleartext password (#887) + +## Version 1.4 (2018-06-03) + +Changes: + + - Documentation fixes (#530, #535, #567) + - Refactoring (#575, #579, #580, #581, #603, #615, #704) + - Cache column names (#444) + - Sort the DSN parameters in DSNs generated from a config (#637) + - Allow native password authentication by default (#644) + - Use the default port if it is missing in the DSN (#668) + - Removed the `strict` mode (#676) + - Do not query `max_allowed_packet` by default (#680) + - Dropped support Go 1.6 and lower (#696) + - Updated `ConvertValue()` to match the database/sql/driver implementation (#760) + - Document the usage of `0000-00-00T00:00:00` as the time.Time zero value (#783) + - Improved the compatibility of the authentication system (#807) + +New Features: + + - Multi-Results support (#537) + - `rejectReadOnly` DSN option (#604) + - `context.Context` support (#608, #612, #627, #761) + - Transaction isolation level support (#619, #744) + - Read-Only transactions support (#618, #634) + - `NewConfig` function which initializes a config with default values (#679) + - Implemented the `ColumnType` interfaces (#667, #724) + - Support for custom string types in `ConvertValue` (#623) + - Implemented `NamedValueChecker`, improving support for uint64 with high bit set (#690, #709, #710) + - `caching_sha2_password` authentication plugin support (#794, #800, #801, #802) + - Implemented `driver.SessionResetter` (#779) + - `sha256_password` authentication plugin support (#808) + +Bugfixes: + + - Use the DSN hostname as TLS default ServerName if `tls=true` (#564, #718) + - Fixed LOAD LOCAL DATA INFILE for empty files (#590) + - Removed columns definition cache since it sometimes cached invalid data (#592) + - Don't mutate registered TLS configs (#600) + - Make RegisterTLSConfig concurrency-safe (#613) + - Handle missing auth data in the handshake packet correctly (#646) + - Do not retry queries when data was written to avoid data corruption (#302, #736) + - Cache the connection pointer for error handling before invalidating it (#678) + - Fixed imports for appengine/cloudsql (#700) + - Fix sending STMT_LONG_DATA for 0 byte data (#734) + - Set correct capacity for []bytes read from length-encoded strings (#766) + - Make RegisterDial concurrency-safe (#773) + + +## Version 1.3 (2016-12-01) + +Changes: + + - Go 1.1 is no longer supported + - Use decimals fields in MySQL to format time types (#249) + - Buffer optimizations (#269) + - TLS ServerName defaults to the host (#283) + - Refactoring (#400, #410, #437) + - Adjusted documentation for second generation CloudSQL (#485) + - Documented DSN system var quoting rules (#502) + - Made statement.Close() calls idempotent to avoid errors in Go 1.6+ (#512) + +New Features: + + - Enable microsecond resolution on TIME, DATETIME and TIMESTAMP (#249) + - Support for returning table alias on Columns() (#289, #359, #382) + - Placeholder interpolation, can be activated with the DSN parameter `interpolateParams=true` (#309, #318, #490) + - Support for uint64 parameters with high bit set (#332, #345) + - Cleartext authentication plugin support (#327) + - Exported ParseDSN function and the Config struct (#403, #419, #429) + - Read / Write timeouts (#401) + - Support for JSON field type (#414) + - Support for multi-statements and multi-results (#411, #431) + - DSN parameter to set the driver-side max_allowed_packet value manually (#489) + - Native password authentication plugin support (#494, #524) + +Bugfixes: + + - Fixed handling of queries without columns and rows (#255) + - Fixed a panic when SetKeepAlive() failed (#298) + - Handle ERR packets while reading rows (#321) + - Fixed reading NULL length-encoded integers in MySQL 5.6+ (#349) + - Fixed absolute paths support in LOAD LOCAL DATA INFILE (#356) + - Actually zero out bytes in handshake response (#378) + - Fixed race condition in registering LOAD DATA INFILE handler (#383) + - Fixed tests with MySQL 5.7.9+ (#380) + - QueryUnescape TLS config names (#397) + - Fixed "broken pipe" error by writing to closed socket (#390) + - Fixed LOAD LOCAL DATA INFILE buffering (#424) + - Fixed parsing of floats into float64 when placeholders are used (#434) + - Fixed DSN tests with Go 1.7+ (#459) + - Handle ERR packets while waiting for EOF (#473) + - Invalidate connection on error while discarding additional results (#513) + - Allow terminating packets of length 0 (#516) + + +## Version 1.2 (2014-06-03) + +Changes: + + - We switched back to a "rolling release". `go get` installs the current master branch again + - Version v1 of the driver will not be maintained anymore. Go 1.0 is no longer supported by this driver + - Exported errors to allow easy checking from application code + - Enabled TCP Keepalives on TCP connections + - Optimized INFILE handling (better buffer size calculation, lazy init, ...) + - The DSN parser also checks for a missing separating slash + - Faster binary date / datetime to string formatting + - Also exported the MySQLWarning type + - mysqlConn.Close returns the first error encountered instead of ignoring all errors + - writePacket() automatically writes the packet size to the header + - readPacket() uses an iterative approach instead of the recursive approach to merge split packets + +New Features: + + - `RegisterDial` allows the usage of a custom dial function to establish the network connection + - Setting the connection collation is possible with the `collation` DSN parameter. This parameter should be preferred over the `charset` parameter + - Logging of critical errors is configurable with `SetLogger` + - Google CloudSQL support + +Bugfixes: + + - Allow more than 32 parameters in prepared statements + - Various old_password fixes + - Fixed TestConcurrent test to pass Go's race detection + - Fixed appendLengthEncodedInteger for large numbers + - Renamed readLengthEnodedString to readLengthEncodedString and skipLengthEnodedString to skipLengthEncodedString (fixed typo) + + +## Version 1.1 (2013-11-02) + +Changes: + + - Go-MySQL-Driver now requires Go 1.1 + - Connections now use the collation `utf8_general_ci` by default. Adding `&charset=UTF8` to the DSN should not be necessary anymore + - Made closing rows and connections error tolerant. This allows for example deferring rows.Close() without checking for errors + - `[]byte(nil)` is now treated as a NULL value. Before, it was treated like an empty string / `[]byte("")` + - DSN parameter values must now be url.QueryEscape'ed. This allows text values to contain special characters, such as '&'. + - Use the IO buffer also for writing. This results in zero allocations (by the driver) for most queries + - Optimized the buffer for reading + - stmt.Query now caches column metadata + - New Logo + - Changed the copyright header to include all contributors + - Improved the LOAD INFILE documentation + - The driver struct is now exported to make the driver directly accessible + - Refactored the driver tests + - Added more benchmarks and moved all to a separate file + - Other small refactoring + +New Features: + + - Added *old_passwords* support: Required in some cases, but must be enabled by adding `allowOldPasswords=true` to the DSN since it is insecure + - Added a `clientFoundRows` parameter: Return the number of matching rows instead of the number of rows changed on UPDATEs + - Added TLS/SSL support: Use a TLS/SSL encrypted connection to the server. Custom TLS configs can be registered and used + +Bugfixes: + + - Fixed MySQL 4.1 support: MySQL 4.1 sends packets with lengths which differ from the specification + - Convert to DB timezone when inserting `time.Time` + - Split packets (more than 16MB) are now merged correctly + - Fixed false positive `io.EOF` errors when the data was fully read + - Avoid panics on reuse of closed connections + - Fixed empty string producing false nil values + - Fixed sign byte for positive TIME fields + + +## Version 1.0 (2013-05-14) + +Initial Release diff --git a/_LICENSES/github.com/go-sql-driver/mysql/LICENSE b/_LICENSES/github.com/go-sql-driver/mysql/LICENSE new file mode 100644 index 0000000000..14e2f777f6 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/LICENSE @@ -0,0 +1,373 @@ +Mozilla Public License Version 2.0 +================================== + +1. Definitions +-------------- + +1.1. "Contributor" + means each individual or legal entity that creates, contributes to + the creation of, or owns Covered Software. + +1.2. "Contributor Version" + means the combination of the Contributions of others (if any) used + by a Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + means Source Code Form to which the initial Contributor has attached + the notice in Exhibit A, the Executable Form of such Source Code + Form, and Modifications of such Source Code Form, in each case + including portions thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + (a) that the initial Contributor has attached the notice described + in Exhibit B to the Covered Software; or + + (b) that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the + terms of a Secondary License. + +1.6. "Executable Form" + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + means a work that combines Covered Software with other material, in + a separate file or files, that is not Covered Software. + +1.8. "License" + means this document. + +1.9. "Licensable" + means having the right to grant, to the maximum extent possible, + whether at the time of the initial grant or subsequently, any and + all of the rights conveyed by this License. + +1.10. "Modifications" + means any of the following: + + (a) any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered + Software; or + + (b) any new file in Source Code Form that contains any Covered + Software. + +1.11. "Patent Claims" of a Contributor + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the + License, by the making, using, selling, offering for sale, having + made, import, or transfer of either its Contributions or its + Contributor Version. + +1.12. "Secondary License" + means either the GNU General Public License, Version 2.0, the GNU + Lesser General Public License, Version 2.1, the GNU Affero General + Public License, Version 3.0, or any later versions of those + licenses. + +1.13. "Source Code Form" + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that + controls, is controlled by, or is under common control with You. For + purposes of this definition, "control" means (a) the power, direct + or indirect, to cause the direction or management of such entity, + whether by contract or otherwise, or (b) ownership of more than + fifty percent (50%) of the outstanding shares or beneficial + ownership of such entity. + +2. License Grants and Conditions +-------------------------------- + +2.1. Grants + +Each Contributor hereby grants You a world-wide, royalty-free, +non-exclusive license: + +(a) under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + +(b) under Patent Claims of such Contributor to make, use, sell, offer + for sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + +The licenses granted in Section 2.1 with respect to any Contribution +become effective for each Contribution on the date the Contributor first +distributes such Contribution. + +2.3. Limitations on Grant Scope + +The licenses granted in this Section 2 are the only rights granted under +this License. No additional rights or licenses will be implied from the +distribution or licensing of Covered Software under this License. +Notwithstanding Section 2.1(b) above, no patent license is granted by a +Contributor: + +(a) for any code that a Contributor has removed from Covered Software; + or + +(b) for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + +(c) under Patent Claims infringed by Covered Software in the absence of + its Contributions. + +This License does not grant any rights in the trademarks, service marks, +or logos of any Contributor (except as may be necessary to comply with +the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + +No Contributor makes additional grants as a result of Your choice to +distribute the Covered Software under a subsequent version of this +License (see Section 10.2) or under the terms of a Secondary License (if +permitted under the terms of Section 3.3). + +2.5. Representation + +Each Contributor represents that the Contributor believes its +Contributions are its original creation(s) or it has sufficient rights +to grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + +This License is not intended to limit any rights You have under +applicable copyright doctrines of fair use, fair dealing, or other +equivalents. + +2.7. Conditions + +Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted +in Section 2.1. + +3. Responsibilities +------------------- + +3.1. Distribution of Source Form + +All distribution of Covered Software in Source Code Form, including any +Modifications that You create or to which You contribute, must be under +the terms of this License. You must inform recipients that the Source +Code Form of the Covered Software is governed by the terms of this +License, and how they can obtain a copy of this License. You may not +attempt to alter or restrict the recipients' rights in the Source Code +Form. + +3.2. Distribution of Executable Form + +If You distribute Covered Software in Executable Form then: + +(a) such Covered Software must also be made available in Source Code + Form, as described in Section 3.1, and You must inform recipients of + the Executable Form how they can obtain a copy of such Source Code + Form by reasonable means in a timely manner, at a charge no more + than the cost of distribution to the recipient; and + +(b) You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter + the recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + +You may create and distribute a Larger Work under terms of Your choice, +provided that You also comply with the requirements of this License for +the Covered Software. If the Larger Work is a combination of Covered +Software with a work governed by one or more Secondary Licenses, and the +Covered Software is not Incompatible With Secondary Licenses, this +License permits You to additionally distribute such Covered Software +under the terms of such Secondary License(s), so that the recipient of +the Larger Work may, at their option, further distribute the Covered +Software under the terms of either this License or such Secondary +License(s). + +3.4. Notices + +You may not remove or alter the substance of any license notices +(including copyright notices, patent notices, disclaimers of warranty, +or limitations of liability) contained within the Source Code Form of +the Covered Software, except that You may alter any license notices to +the extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + +You may choose to offer, and to charge a fee for, warranty, support, +indemnity or liability obligations to one or more recipients of Covered +Software. However, You may do so only on Your own behalf, and not on +behalf of any Contributor. You must make it absolutely clear that any +such warranty, support, indemnity, or liability obligation is offered by +You alone, and You hereby agree to indemnify every Contributor for any +liability incurred by such Contributor as a result of warranty, support, +indemnity or liability terms You offer. You may include additional +disclaimers of warranty and limitations of liability specific to any +jurisdiction. + +4. Inability to Comply Due to Statute or Regulation +--------------------------------------------------- + +If it is impossible for You to comply with any of the terms of this +License with respect to some or all of the Covered Software due to +statute, judicial order, or regulation then You must: (a) comply with +the terms of this License to the maximum extent possible; and (b) +describe the limitations and the code they affect. Such description must +be placed in a text file included with all distributions of the Covered +Software under this License. Except to the extent prohibited by statute +or regulation, such description must be sufficiently detailed for a +recipient of ordinary skill to be able to understand it. + +5. Termination +-------------- + +5.1. The rights granted under this License will terminate automatically +if You fail to comply with any of its terms. However, if You become +compliant, then the rights granted under this License from a particular +Contributor are reinstated (a) provisionally, unless and until such +Contributor explicitly and finally terminates Your grants, and (b) on an +ongoing basis, if such Contributor fails to notify You of the +non-compliance by some reasonable means prior to 60 days after You have +come back into compliance. Moreover, Your grants from a particular +Contributor are reinstated on an ongoing basis if such Contributor +notifies You of the non-compliance by some reasonable means, this is the +first time You have received notice of non-compliance with this License +from such Contributor, and You become compliant prior to 30 days after +Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent +infringement claim (excluding declaratory judgment actions, +counter-claims, and cross-claims) alleging that a Contributor Version +directly or indirectly infringes any patent, then the rights granted to +You by any and all Contributors for the Covered Software under Section +2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all +end user license agreements (excluding distributors and resellers) which +have been validly granted by You or Your distributors under this License +prior to termination shall survive termination. + +************************************************************************ +* * +* 6. Disclaimer of Warranty * +* ------------------------- * +* * +* Covered Software is provided under this License on an "as is" * +* basis, without warranty of any kind, either expressed, implied, or * +* statutory, including, without limitation, warranties that the * +* Covered Software is free of defects, merchantable, fit for a * +* particular purpose or non-infringing. The entire risk as to the * +* quality and performance of the Covered Software is with You. * +* Should any Covered Software prove defective in any respect, You * +* (not any Contributor) assume the cost of any necessary servicing, * +* repair, or correction. This disclaimer of warranty constitutes an * +* essential part of this License. No use of any Covered Software is * +* authorized under this License except under this disclaimer. * +* * +************************************************************************ + +************************************************************************ +* * +* 7. Limitation of Liability * +* -------------------------- * +* * +* Under no circumstances and under no legal theory, whether tort * +* (including negligence), contract, or otherwise, shall any * +* Contributor, or anyone who distributes Covered Software as * +* permitted above, be liable to You for any direct, indirect, * +* special, incidental, or consequential damages of any character * +* including, without limitation, damages for lost profits, loss of * +* goodwill, work stoppage, computer failure or malfunction, or any * +* and all other commercial damages or losses, even if such party * +* shall have been informed of the possibility of such damages. This * +* limitation of liability shall not apply to liability for death or * +* personal injury resulting from such party's negligence to the * +* extent applicable law prohibits such limitation. Some * +* jurisdictions do not allow the exclusion or limitation of * +* incidental or consequential damages, so this exclusion and * +* limitation may not apply to You. * +* * +************************************************************************ + +8. Litigation +------------- + +Any litigation relating to this License may be brought only in the +courts of a jurisdiction where the defendant maintains its principal +place of business and such litigation shall be governed by laws of that +jurisdiction, without reference to its conflict-of-law provisions. +Nothing in this Section shall prevent a party's ability to bring +cross-claims or counter-claims. + +9. Miscellaneous +---------------- + +This License represents the complete agreement concerning the subject +matter hereof. If any provision of this License is held to be +unenforceable, such provision shall be reformed only to the extent +necessary to make it enforceable. Any law or regulation which provides +that the language of a contract shall be construed against the drafter +shall not be used to construe this License against a Contributor. + +10. Versions of the License +--------------------------- + +10.1. New Versions + +Mozilla Foundation is the license steward. Except as provided in Section +10.3, no one other than the license steward has the right to modify or +publish new versions of this License. Each version will be given a +distinguishing version number. + +10.2. Effect of New Versions + +You may distribute the Covered Software under the terms of the version +of the License under which You originally received the Covered Software, +or under the terms of any subsequent version published by the license +steward. + +10.3. Modified Versions + +If you create software not governed by this License, and you want to +create a new license for such software, you may create and use a +modified version of this License if you rename the license and remove +any references to the name of the license steward (except to note that +such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary +Licenses + +If You choose to distribute Source Code Form that is Incompatible With +Secondary Licenses under the terms of this version of the License, the +notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice +------------------------------------------- + + This Source Code Form is subject to the terms of the Mozilla Public + License, v. 2.0. If a copy of the MPL was not distributed with this + file, You can obtain one at http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular +file, then You may include the notice in a location (such as a LICENSE +file in a relevant directory) where a recipient would be likely to look +for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice +--------------------------------------------------------- + + This Source Code Form is "Incompatible With Secondary Licenses", as + defined by the Mozilla Public License, v. 2.0. diff --git a/_LICENSES/github.com/go-sql-driver/mysql/README.md b/_LICENSES/github.com/go-sql-driver/mysql/README.md new file mode 100644 index 0000000000..ccbe6d0787 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/README.md @@ -0,0 +1,601 @@ +# Go-MySQL-Driver + +[![DeepWiki](https://img.shields.io/badge/DeepWiki-go--sql--driver%2Fmysql-blue.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McDcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==)](https://deepwiki.com/go-sql-driver/mysql) + + +A MySQL-Driver for Go's [database/sql](https://golang.org/pkg/database/sql/) package + +![Go-MySQL-Driver logo](https://raw.github.com/wiki/go-sql-driver/mysql/gomysql_m.png "Golang Gopher holding the MySQL Dolphin") + +--------------------------------------- + * [Features](#features) + * [Requirements](#requirements) + * [Installation](#installation) + * [Usage](#usage) + * [DSN (Data Source Name)](#dsn-data-source-name) + * [Password](#password) + * [Protocol](#protocol) + * [Address](#address) + * [Parameters](#parameters) + * [Examples](#examples) + * [Connection pool and timeouts](#connection-pool-and-timeouts) + * [context.Context Support](#contextcontext-support) + * [ColumnType Support](#columntype-support) + * [LOAD DATA LOCAL INFILE support](#load-data-local-infile-support) + * [time.Time support](#timetime-support) + * [Unicode support](#unicode-support) + * [Testing / Development](#testing--development) + * [License](#license) + +--------------------------------------- + +## Features + * Lightweight and [fast](https://github.com/go-sql-driver/sql-benchmark "golang MySQL-Driver performance") + * Native Go implementation. No C-bindings, just pure Go + * Connections over TCP/IPv4, TCP/IPv6, Unix domain sockets or [custom protocols](https://godoc.org/github.com/go-sql-driver/mysql#DialFunc) + * Automatic handling of broken connections + * Automatic Connection Pooling *(by database/sql package)* + * Supports queries larger than 16MB + * Full [`sql.RawBytes`](https://golang.org/pkg/database/sql/#RawBytes) support. + * Intelligent `LONG DATA` handling in prepared statements + * Secure `LOAD DATA LOCAL INFILE` support with file allowlisting and `io.Reader` support + * Optional `time.Time` parsing + * Optional placeholder interpolation + * Supports zlib compression. + +## Requirements + +* Go 1.24 or higher. We aim to support the 3 latest versions of Go. +* MySQL (5.7+) and MariaDB (10.5+) are supported by maintainers. +* [TiDB](https://github.com/pingcap/tidb) is supported by PingCAP. + * Do not ask questions about TiDB in our issue tracker or forum. + * [Document](https://docs.pingcap.com/tidb/v6.1/dev-guide-sample-application-golang) + * [Forum](https://ask.pingcap.com/) +* go-mysql would work with Percona Server, Google CloudSQL or Sphinx (2.2.3+). + * Maintainers won't support them. Do not expect issues are investigated and resolved by maintainers. + * Investigate issues yourself and please send a pull request to fix it. + +--------------------------------------- + +## Installation +Simple install the package to your [$GOPATH](https://github.com/golang/go/wiki/GOPATH "GOPATH") with the [go tool](https://golang.org/cmd/go/ "go command") from shell: +```bash +go get -u github.com/go-sql-driver/mysql +``` +Make sure [Git is installed](https://git-scm.com/downloads) on your machine and in your system's `PATH`. + +## Usage +_Go MySQL Driver_ is an implementation of Go's `database/sql/driver` interface. You only need to import the driver and can use the full [`database/sql`](https://golang.org/pkg/database/sql/) API then. + +Use `mysql` as `driverName` and a valid [DSN](#dsn-data-source-name) as `dataSourceName`: + +```go +import ( + "database/sql" + "time" + + _ "github.com/go-sql-driver/mysql" +) + +// ... + +db, err := sql.Open("mysql", "user:password@/dbname") +if err != nil { + panic(err) +} +// See "Important settings" section. +db.SetConnMaxLifetime(time.Minute * 3) +db.SetMaxOpenConns(10) +db.SetMaxIdleConns(10) +``` + +[Examples are available in our Wiki](https://github.com/go-sql-driver/mysql/wiki/Examples "Go-MySQL-Driver Examples"). + +### Important settings + +`db.SetConnMaxLifetime()` is required to ensure connections are closed by the driver safely before connection is closed by MySQL server, OS, or other middlewares. Since some middlewares close idle connections by 5 minutes, we recommend timeout shorter than 5 minutes. This setting helps load balancing and changing system variables too. + +`db.SetMaxOpenConns()` is highly recommended to limit the number of connection used by the application. There is no recommended limit number because it depends on application and MySQL server. + +`db.SetMaxIdleConns()` is recommended to be set same to `db.SetMaxOpenConns()`. When it is smaller than `SetMaxOpenConns()`, connections can be opened and closed much more frequently than you expect. Idle connections can be closed by the `db.SetConnMaxLifetime()`. If you want to close idle connections more rapidly, you can use `db.SetConnMaxIdleTime()` since Go 1.15. + + +### DSN (Data Source Name) + +The Data Source Name has a common format, like e.g. [PEAR DB](http://pear.php.net/manual/en/package.database.db.intro-dsn.php) uses it, but without type-prefix (optional parts marked by squared brackets): +``` +[username[:password]@][protocol[(address)]]/dbname[?param1=value1&...¶mN=valueN] +``` + +A DSN in its fullest form: +``` +username:password@protocol(address)/dbname?param=value +``` + +Except for the databasename, all values are optional. So the minimal DSN is: +``` +/dbname +``` + +If you do not want to preselect a database, leave `dbname` empty: +``` +/ +``` +This has the same effect as an empty DSN string: +``` + +``` + +`dbname` is escaped by [PathEscape()](https://pkg.go.dev/net/url#PathEscape) since v1.8.0. If your database name is `dbname/withslash`, it becomes: + +``` +/dbname%2Fwithslash +``` + +Alternatively, [Config.FormatDSN](https://godoc.org/github.com/go-sql-driver/mysql#Config.FormatDSN) can be used to create a DSN string by filling a struct. + +#### Password +Passwords can consist of any character. Escaping is **not** necessary. + +#### Protocol +See [net.Dial](https://golang.org/pkg/net/#Dial) for more information which networks are available. +In general you should use a Unix domain socket if available and TCP otherwise for best performance. + +#### Address +For TCP and UDP networks, addresses have the form `host[:port]`. +If `port` is omitted, the default port will be used. +If `host` is a literal IPv6 address, it must be enclosed in square brackets. +The functions [net.JoinHostPort](https://golang.org/pkg/net/#JoinHostPort) and [net.SplitHostPort](https://golang.org/pkg/net/#SplitHostPort) manipulate addresses in this form. + +For Unix domain sockets the address is the absolute path to the MySQL-Server-socket, e.g. `/var/run/mysqld/mysqld.sock` or `/tmp/mysql.sock`. + +#### Parameters +*Parameters are case-sensitive!* + +Notice that any of `true`, `TRUE`, `True` or `1` is accepted to stand for a true boolean value. Not surprisingly, false can be specified as any of: `false`, `FALSE`, `False` or `0`. + +##### `allowAllFiles` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`allowAllFiles=true` disables the file allowlist for `LOAD DATA LOCAL INFILE` and allows *all* files. +[*Might be insecure!*](https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-local) + +##### `allowCleartextPasswords` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`allowCleartextPasswords=true` allows using the [cleartext client side plugin](https://dev.mysql.com/doc/en/cleartext-pluggable-authentication.html) if required by an account, such as one defined with the [PAM authentication plugin](http://dev.mysql.com/doc/en/pam-authentication-plugin.html). Sending passwords in clear text may be a security problem in some configurations. To avoid problems if there is any possibility that the password would be intercepted, clients should connect to MySQL Server using a method that protects the password. Possibilities include [TLS / SSL](#tls), IPsec, or a private network. + + +##### `allowFallbackToPlaintext` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`allowFallbackToPlaintext=true` acts like a `--ssl-mode=PREFERRED` MySQL client as described in [Command Options for Connecting to the Server](https://dev.mysql.com/doc/refman/5.7/en/connection-options.html#option_general_ssl-mode) + +##### `allowNativePasswords` + +``` +Type: bool +Valid Values: true, false +Default: true +``` +`allowNativePasswords=false` disallows the usage of MySQL native password method. + +##### `allowOldPasswords` + +``` +Type: bool +Valid Values: true, false +Default: false +``` +`allowOldPasswords=true` allows the usage of the insecure old password method. This should be avoided, but is necessary in some cases. See also [the old_passwords wiki page](https://github.com/go-sql-driver/mysql/wiki/old_passwords). + +##### `charset` + +``` +Type: string +Valid Values: +Default: none +``` + +Sets the charset used for client-server interaction (`"SET NAMES "`). If multiple charsets are set (separated by a comma), the following charset is used if setting the charset fails. This enables for example support for `utf8mb4` ([introduced in MySQL 5.5.3](http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html)) with fallback to `utf8` for older servers (`charset=utf8mb4,utf8`). + +See also [Unicode Support](#unicode-support). + +##### `checkConnLiveness` + +``` +Type: bool +Valid Values: true, false +Default: true +``` + +On supported platforms connections retrieved from the connection pool are checked for liveness before using them. If the check fails, the respective connection is marked as bad and the query retried with another connection. +`checkConnLiveness=false` disables this liveness check of connections. + +##### `collation` + +``` +Type: string +Valid Values: +Default: utf8mb4_general_ci +``` + +Sets the collation used for client-server interaction on connection. In contrast to `charset`, `collation` does not issue additional queries. If the specified collation is unavailable on the target server, the connection will fail. + +A list of valid charsets for a server is retrievable with `SHOW COLLATION`. + +The default collation (`utf8mb4_general_ci`) is supported from MySQL 5.5. You should use an older collation (e.g. `utf8_general_ci`) for older MySQL. + +Collations for charset "ucs2", "utf16", "utf16le", and "utf32" can not be used ([ref](https://dev.mysql.com/doc/refman/5.7/en/charset-connection.html#charset-connection-impermissible-client-charset)). + +See also [Unicode Support](#unicode-support). + +##### `clientFoundRows` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`clientFoundRows=true` causes an UPDATE to return the number of matching rows instead of the number of rows changed. + +##### `columnsWithAlias` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +When `columnsWithAlias` is true, calls to `sql.Rows.Columns()` will return the table alias and the column name separated by a dot. For example: + +``` +SELECT u.id FROM users as u +``` + +will return `u.id` instead of just `id` if `columnsWithAlias=true`. + +##### `compress` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +Toggles zlib compression. false by default. + +##### `interpolateParams` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +If `interpolateParams` is true, placeholders (`?`) in calls to `db.Query()` and `db.Exec()` are interpolated into a single query string with given parameters. This reduces the number of roundtrips, since the driver has to prepare a statement, execute it with given parameters and close the statement again with `interpolateParams=false`. + +*This can not be used together with the multibyte encodings BIG5, CP932, GB2312, GBK or SJIS. These are rejected as they may [introduce a SQL injection vulnerability](http://stackoverflow.com/a/12118602/3430118)!* + +##### `loc` + +``` +Type: string +Valid Values: +Default: UTC +``` + +Sets the location for time.Time values (when using `parseTime=true`). *"Local"* sets the system's location. See [time.LoadLocation](https://golang.org/pkg/time/#LoadLocation) for details. + +Note that this sets the location for time.Time values but does not change MySQL's [time_zone setting](https://dev.mysql.com/doc/refman/5.5/en/time-zone-support.html). For that see the [time_zone system variable](#system-variables), which can also be set as a DSN parameter. + +Please keep in mind, that param values must be [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape)'ed. Alternatively you can manually replace the `/` with `%2F`. For example `US/Pacific` would be `loc=US%2FPacific`. + +##### `timeTruncate` + +``` +Type: duration +Default: 0 +``` + +[Truncate time values](https://pkg.go.dev/time#Duration.Truncate) in query arguments to the specified duration. The value must be a decimal number with a unit suffix (*"ns"*, *"us"*, *"ms"*, etc...), such as "1us", "1ms", or "10ns". + +> [!NOTE] +> `time.Time` arguments are sent with up to nanosecond precision, so a value from `time.Now()` usually has more fractional-second digits than a `DATETIME(N)` or `TIMESTAMP(N)` column stores. On MariaDB, comparing such a value against an indexed column can prevent an index range scan, turning it into a full index scan. Truncating to the column's precision (`1us` for `DATETIME(6)`) avoids this. Only arguments sent to the server are truncated; values read from the server are not affected. + +##### `maxAllowedPacket` +``` +Type: decimal number +Default: 64*1024*1024 +``` + +Max packet size allowed in bytes. The default value is 64 MiB and should be adjusted to match the server settings. `maxAllowedPacket=0` can be used to automatically fetch the `max_allowed_packet` variable from server *on every connection*. + +##### `multiStatements` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +Allow multiple statements in one query. This can be used to bach multiple queries. Use [Rows.NextResultSet()](https://pkg.go.dev/database/sql#Rows.NextResultSet) to get result of the second and subsequent queries. + +When `multiStatements` is used, `?` parameters must only be used in the first statement. [interpolateParams](#interpolateparams) can be used to avoid this limitation unless prepared statement is used explicitly. + +It's possible to access the last inserted ID and number of affected rows for multiple statements by using `sql.Conn.Raw()` and the `mysql.Result`. For example: + +```go +conn, _ := db.Conn(ctx) +conn.Raw(func(conn any) error { + ex := conn.(driver.Execer) + res, err := ex.Exec(` + UPDATE point SET x = 1 WHERE y = 2; + UPDATE point SET x = 2 WHERE y = 3; + `, nil) + // Both slices have 2 elements. + log.Print(res.(mysql.Result).AllRowsAffected()) + log.Print(res.(mysql.Result).AllLastInsertIds()) +}) +``` + +##### `parseTime` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`parseTime=true` changes the output type of `DATE` and `DATETIME` values to `time.Time` instead of `[]byte` / `string` +The date or datetime like `0000-00-00 00:00:00` is converted into zero value of `time.Time`. + + +##### `readTimeout` + +``` +Type: duration +Default: 0 +``` + +I/O read timeout. The value must be a decimal number with a unit suffix (*"ms"*, *"s"*, *"m"*, *"h"*), such as *"30s"*, *"0.5m"* or *"1m30s"*. + +##### `rejectReadOnly` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + + +`rejectReadOnly=true` causes the driver to reject read-only connections. This +is for a possible race condition during an automatic failover, where the mysql +client gets connected to a read-only replica after the failover. + +Note that this should be a fairly rare case, as an automatic failover normally +happens when the primary is down, and the race condition shouldn't happen +unless it comes back up online as soon as the failover is kicked off. On the +other hand, when this happens, a MySQL application can get stuck on a +read-only connection until restarted. It is however fairly easy to reproduce, +for example, using a manual failover on AWS Aurora's MySQL-compatible cluster. + +If you are not relying on read-only transactions to reject writes that aren't +supposed to happen, setting this on some MySQL providers (such as AWS Aurora) +is safer for failovers. + +Note that ERROR 1290 can be returned for a `read-only` server and this option will +cause a retry for that error. However the same error number is used for some +other cases. You should ensure your application will never cause an ERROR 1290 +except for `read-only` mode when enabling this option. + + +##### `serverPubKey` + +``` +Type: string +Valid Values: +Default: none +``` + +Server public keys can be registered with [`mysql.RegisterServerPubKey`](https://godoc.org/github.com/go-sql-driver/mysql#RegisterServerPubKey), which can then be used by the assigned name in the DSN. +Public keys are used to transmit encrypted data, e.g. for authentication. +If the server's public key is known, it should be set manually to avoid expensive and potentially insecure transmissions of the public key from the server to the client each time it is required. + + +##### `timeout` + +``` +Type: duration +Default: OS default +``` + +Timeout for establishing connections, aka dial timeout. The value must be a decimal number with a unit suffix (*"ms"*, *"s"*, *"m"*, *"h"*), such as *"30s"*, *"0.5m"* or *"1m30s"*. + + +##### `tls` + +``` +Type: bool / string +Valid Values: true, false, skip-verify, preferred, +Default: false +``` + +`tls=true` enables TLS / SSL encrypted connection to the server. Use `skip-verify` if you want to use a self-signed or invalid certificate (server side) or use `preferred` to use TLS only when advertised by the server. This is similar to `skip-verify`, but additionally allows a fallback to a connection which is not encrypted. Neither `skip-verify` nor `preferred` add any reliable security. You can use a custom TLS config after registering it with [`mysql.RegisterTLSConfig`](https://godoc.org/github.com/go-sql-driver/mysql#RegisterTLSConfig). + + +##### `writeTimeout` + +``` +Type: duration +Default: 0 +``` + +I/O write timeout. The value must be a decimal number with a unit suffix (*"ms"*, *"s"*, *"m"*, *"h"*), such as *"30s"*, *"0.5m"* or *"1m30s"*. + +##### `connectionAttributes` + +``` +Type: comma-delimited string of user-defined "key:value" pairs +Valid Values: (:,:,...) +Default: none +``` + +[Connection attributes](https://dev.mysql.com/doc/refman/8.0/en/performance-schema-connection-attribute-tables.html) are key-value pairs that application programs can pass to the server at connect time. + +##### System Variables + +Any other parameters are interpreted as system variables: + * `=`: `SET =` + * `=`: `SET =` + * `=%27%27`: `SET =''` + +Rules: +* The values for string variables must be quoted with `'`. +* The values must also be [url.QueryEscape](http://golang.org/pkg/net/url/#QueryEscape)'ed! + (which implies values of string variables must be wrapped with `%27`). + +Examples: + * `autocommit=1`: `SET autocommit=1` + * [`time_zone=%27Europe%2FParis%27`](https://dev.mysql.com/doc/refman/5.5/en/time-zone-support.html): `SET time_zone='Europe/Paris'` + * [`transaction_isolation=%27REPEATABLE-READ%27`](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_transaction_isolation): `SET transaction_isolation='REPEATABLE-READ'` + + +#### Examples +``` +user@unix(/path/to/socket)/dbname +``` + +``` +root:pw@unix(/tmp/mysql.sock)/myDatabase?loc=Local +``` + +``` +user:password@tcp(localhost:5555)/dbname?tls=skip-verify&autocommit=true +``` + +Treat warnings as errors by setting the system variable [`sql_mode`](https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html): +``` +user:password@/dbname?sql_mode=TRADITIONAL +``` + +TCP via IPv6: +``` +user:password@tcp([de:ad:be:ef::ca:fe]:80)/dbname?timeout=90s&collation=utf8mb4_unicode_ci +``` + +TCP on a remote host, e.g. Amazon RDS: +``` +id:password@tcp(your-amazonaws-uri.com:3306)/dbname +``` + +Google Cloud SQL on App Engine: +``` +user:password@unix(/cloudsql/project-id:region-name:instance-name)/dbname +``` + +TCP using default port (3306) on localhost: +``` +user:password@tcp/dbname?charset=utf8mb4,utf8&sys_var=esc%40ped +``` + +Use the default protocol (tcp) and host (localhost:3306): +``` +user:password@/dbname +``` + +No Database preselected: +``` +user:password@/ +``` + + +### Connection pool and timeouts +The connection pool is managed by Go's database/sql package. For details on how to configure the size of the pool and how long connections stay in the pool see `*DB.SetMaxOpenConns`, `*DB.SetMaxIdleConns`, and `*DB.SetConnMaxLifetime` in the [database/sql documentation](https://golang.org/pkg/database/sql/). The read, write, and dial timeouts for each individual connection are configured with the DSN parameters [`readTimeout`](#readtimeout), [`writeTimeout`](#writetimeout), and [`timeout`](#timeout), respectively. + +## `ColumnType` Support +This driver supports the [`ColumnType` interface](https://golang.org/pkg/database/sql/#ColumnType) introduced in Go 1.8, with the exception of [`ColumnType.Length()`](https://golang.org/pkg/database/sql/#ColumnType.Length), which is currently not supported. All Unsigned database type names will be returned `UNSIGNED ` with `INT`, `TINYINT`, `SMALLINT`, `MEDIUMINT`, `BIGINT`. + +## `context.Context` Support +Go 1.8 added `database/sql` support for `context.Context`. This driver supports query timeouts and cancellation via contexts. +See [context support in the database/sql package](https://golang.org/doc/go1.8#database_sql) for more details. + +> [!IMPORTANT] +> The `QueryContext`, `ExecContext`, etc. variants provided by `database/sql` will cause the connection to be closed if the provided context is cancelled or timed out before the result is received by the driver. + + +### `LOAD DATA LOCAL INFILE` support +For this feature you need direct access to the package. Therefore you must change the import path (no `_`): +```go +import "github.com/go-sql-driver/mysql" +``` + +Files must be explicitly allowed by registering them with `mysql.RegisterLocalFile(filepath)` (recommended) or the allowlist check must be deactivated by using the DSN parameter `allowAllFiles=true` ([*Might be insecure!*](https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-local)). + +To use a `io.Reader` a handler function must be registered with `mysql.RegisterReaderHandler(name, handler)` which returns a `io.Reader` or `io.ReadCloser`. The Reader is available with the filepath `Reader::` then. Choose different names for different handlers and `DeregisterReaderHandler` when you don't need it anymore. + +See the [godoc of Go-MySQL-Driver](https://godoc.org/github.com/go-sql-driver/mysql "golang mysql driver documentation") for details. + + +### `time.Time` support +The default internal output type of MySQL `DATE` and `DATETIME` values is `[]byte` which allows you to scan the value into a `[]byte`, `string` or `sql.RawBytes` variable in your program. + +However, many want to scan MySQL `DATE` and `DATETIME` values into `time.Time` variables, which is the logical equivalent in Go to `DATE` and `DATETIME` in MySQL. You can do that by changing the internal output type from `[]byte` to `time.Time` with the DSN parameter `parseTime=true`. You can set the default [`time.Time` location](https://golang.org/pkg/time/#Location) with the `loc` DSN parameter. + +**Caution:** As of Go 1.1, this makes `time.Time` the only variable type you can scan `DATE` and `DATETIME` values into. This breaks for example [`sql.RawBytes` support](https://github.com/go-sql-driver/mysql/wiki/Examples#rawbytes). + + +### Unicode support +Since version 1.5 Go-MySQL-Driver automatically uses the collation ` utf8mb4_general_ci` by default. + +Other charsets / collations can be set using the [`charset`](#charset) or [`collation`](#collation) DSN parameter. + +- When only the `charset` is specified, the `SET NAMES ` query is sent and the server's default collation is used. +- When both the `charset` and `collation` are specified, the `SET NAMES COLLATE ` query is sent. +- When only the `collation` is specified, the collation is specified in the protocol handshake and the `SET NAMES` query is not sent. This can save one roundtrip, but note that the server may ignore the specified collation silently and use the server's default charset/collation instead. + +See http://dev.mysql.com/doc/refman/8.0/en/charset-unicode.html for more details on MySQL's Unicode support. + +## Testing / Development +To run the driver tests you may need to adjust the configuration. See the [Testing Wiki-Page](https://github.com/go-sql-driver/mysql/wiki/Testing "Testing") for details. + +Go-MySQL-Driver is not feature-complete yet. Your help is very appreciated. +If you want to contribute, you can work on an [open issue](https://github.com/go-sql-driver/mysql/issues?state=open) or review a [pull request](https://github.com/go-sql-driver/mysql/pulls). + +See the [Contribution Guidelines](https://github.com/go-sql-driver/mysql/blob/master/.github/CONTRIBUTING.md) for details. + +--------------------------------------- + +## License +Go-MySQL-Driver is licensed under the [Mozilla Public License Version 2.0](https://raw.github.com/go-sql-driver/mysql/master/LICENSE) + +Mozilla summarizes the license scope as follows: +> MPL: The copyleft applies to any files containing MPLed code. + + +That means: + * You can **use** the **unchanged** source code both in private and commercially. + * When distributing, you **must publish** the source code of any **changed files** licensed under the MPL 2.0 under a) the MPL 2.0 itself or b) a compatible license (e.g. GPL 3.0 or Apache License 2.0). + * You **needn't publish** the source code of your library as long as the files licensed under the MPL 2.0 are **unchanged**. + +Please read the [MPL 2.0 FAQ](https://www.mozilla.org/en-US/MPL/2.0/FAQ/) if you have further questions regarding the license. + +You can read the full terms here: [LICENSE](https://raw.github.com/go-sql-driver/mysql/master/LICENSE). + +![Go Gopher and MySQL Dolphin](https://raw.github.com/wiki/go-sql-driver/mysql/go-mysql-driver_m.jpg "Golang Gopher transporting the MySQL Dolphin in a wheelbarrow") diff --git a/_LICENSES/github.com/go-sql-driver/mysql/auth.go b/_LICENSES/github.com/go-sql-driver/mysql/auth.go new file mode 100644 index 0000000000..610044fc16 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/auth.go @@ -0,0 +1,484 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2018 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "crypto/rand" + "crypto/rsa" + "crypto/sha1" + "crypto/sha256" + "crypto/sha512" + "crypto/x509" + "encoding/pem" + "fmt" + "sync" + + "filippo.io/edwards25519" +) + +// server pub keys registry +var ( + serverPubKeyLock sync.RWMutex + serverPubKeyRegistry map[string]*rsa.PublicKey +) + +// RegisterServerPubKey registers a server RSA public key which can be used to +// send data in a secure manner to the server without receiving the public key +// in a potentially insecure way from the server first. +// Registered keys can afterwards be used adding serverPubKey= to the DSN. +// +// Note: The provided rsa.PublicKey instance is exclusively owned by the driver +// after registering it and may not be modified. +// +// data, err := os.ReadFile("mykey.pem") +// if err != nil { +// log.Fatal(err) +// } +// +// block, _ := pem.Decode(data) +// if block == nil || block.Type != "PUBLIC KEY" { +// log.Fatal("failed to decode PEM block containing public key") +// } +// +// pub, err := x509.ParsePKIXPublicKey(block.Bytes) +// if err != nil { +// log.Fatal(err) +// } +// +// if rsaPubKey, ok := pub.(*rsa.PublicKey); ok { +// mysql.RegisterServerPubKey("mykey", rsaPubKey) +// } else { +// log.Fatal("not a RSA public key") +// } +func RegisterServerPubKey(name string, pubKey *rsa.PublicKey) { + serverPubKeyLock.Lock() + if serverPubKeyRegistry == nil { + serverPubKeyRegistry = make(map[string]*rsa.PublicKey) + } + + serverPubKeyRegistry[name] = pubKey + serverPubKeyLock.Unlock() +} + +// DeregisterServerPubKey removes the public key registered with the given name. +func DeregisterServerPubKey(name string) { + serverPubKeyLock.Lock() + if serverPubKeyRegistry != nil { + delete(serverPubKeyRegistry, name) + } + serverPubKeyLock.Unlock() +} + +func getServerPubKey(name string) (pubKey *rsa.PublicKey) { + serverPubKeyLock.RLock() + if v, ok := serverPubKeyRegistry[name]; ok { + pubKey = v + } + serverPubKeyLock.RUnlock() + return +} + +// Hash password using pre 4.1 (old password) method +// https://github.com/atcurtis/mariadb/blob/master/mysys/my_rnd.c +type myRnd struct { + seed1, seed2 uint32 +} + +const myRndMaxVal = 0x3FFFFFFF + +// Pseudo random number generator +func newMyRnd(seed1, seed2 uint32) *myRnd { + return &myRnd{ + seed1: seed1 % myRndMaxVal, + seed2: seed2 % myRndMaxVal, + } +} + +// Tested to be equivalent to MariaDB's floating point variant +// http://play.golang.org/p/QHvhd4qved +// http://play.golang.org/p/RG0q4ElWDx +func (r *myRnd) NextByte() byte { + r.seed1 = (r.seed1*3 + r.seed2) % myRndMaxVal + r.seed2 = (r.seed1 + r.seed2 + 33) % myRndMaxVal + + return byte(uint64(r.seed1) * 31 / myRndMaxVal) +} + +// Generate binary hash from byte string using insecure pre 4.1 method +func pwHash(password []byte) (result [2]uint32) { + var add uint32 = 7 + var tmp uint32 + + result[0] = 1345345333 + result[1] = 0x12345671 + + for _, c := range password { + // skip spaces and tabs in password + if c == ' ' || c == '\t' { + continue + } + + tmp = uint32(c) + result[0] ^= (((result[0] & 63) + add) * tmp) + (result[0] << 8) + result[1] += (result[1] << 8) ^ result[0] + add += tmp + } + + // Remove sign bit (1<<31)-1) + result[0] &= 0x7FFFFFFF + result[1] &= 0x7FFFFFFF + + return +} + +// Hash password using insecure pre 4.1 method +func scrambleOldPassword(scramble []byte, password string) []byte { + scramble = scramble[:8] + + hashPw := pwHash([]byte(password)) + hashSc := pwHash(scramble) + + r := newMyRnd(hashPw[0]^hashSc[0], hashPw[1]^hashSc[1]) + + var out [8]byte + for i := range out { + out[i] = r.NextByte() + 64 + } + + mask := r.NextByte() + for i := range out { + out[i] ^= mask + } + + return out[:] +} + +// Hash password using 4.1+ method (SHA1) +func scramblePassword(scramble []byte, password string) []byte { + if len(password) == 0 { + return nil + } + + // stage1Hash = SHA1(password) + crypt := sha1.New() + crypt.Write([]byte(password)) + stage1 := crypt.Sum(nil) + + // scrambleHash = SHA1(scramble + SHA1(stage1Hash)) + // inner Hash + crypt.Reset() + crypt.Write(stage1) + hash := crypt.Sum(nil) + + // outer Hash + crypt.Reset() + crypt.Write(scramble) + crypt.Write(hash) + scramble = crypt.Sum(nil) + + // token = scrambleHash XOR stage1Hash + for i := range scramble { + scramble[i] ^= stage1[i] + } + return scramble +} + +// Hash password using MySQL 8+ method (SHA256) +func scrambleSHA256Password(scramble []byte, password string) []byte { + if len(password) == 0 { + return nil + } + + // XOR(SHA256(password), SHA256(SHA256(SHA256(password)), scramble)) + + crypt := sha256.New() + crypt.Write([]byte(password)) + message1 := crypt.Sum(nil) + + crypt.Reset() + crypt.Write(message1) + message1Hash := crypt.Sum(nil) + + crypt.Reset() + crypt.Write(message1Hash) + crypt.Write(scramble) + message2 := crypt.Sum(nil) + + for i := range message1 { + message1[i] ^= message2[i] + } + + return message1 +} + +func encryptPassword(password string, seed []byte, pub *rsa.PublicKey) ([]byte, error) { + plain := make([]byte, len(password)+1) + copy(plain, password) + for i := range plain { + j := i % len(seed) + plain[i] ^= seed[j] + } + sha1 := sha1.New() + return rsa.EncryptOAEP(sha1, rand.Reader, pub, plain, nil) +} + +// authEd25519 does ed25519 authentication used by MariaDB. +func authEd25519(scramble []byte, password string) ([]byte, error) { + // Derived from https://github.com/MariaDB/server/blob/d8e6bb00888b1f82c031938f4c8ac5d97f6874c3/plugin/auth_ed25519/ref10/sign.c + // Code style is from https://cs.opensource.google/go/go/+/refs/tags/go1.21.5:src/crypto/ed25519/ed25519.go;l=207 + h := sha512.Sum512([]byte(password)) + + s, err := edwards25519.NewScalar().SetBytesWithClamping(h[:32]) + if err != nil { + return nil, err + } + A := (&edwards25519.Point{}).ScalarBaseMult(s) + + mh := sha512.New() + mh.Write(h[32:]) + mh.Write(scramble) + messageDigest := mh.Sum(nil) + r, err := edwards25519.NewScalar().SetUniformBytes(messageDigest) + if err != nil { + return nil, err + } + + R := (&edwards25519.Point{}).ScalarBaseMult(r) + + kh := sha512.New() + kh.Write(R.Bytes()) + kh.Write(A.Bytes()) + kh.Write(scramble) + hramDigest := kh.Sum(nil) + k, err := edwards25519.NewScalar().SetUniformBytes(hramDigest) + if err != nil { + return nil, err + } + + S := k.MultiplyAdd(k, s, r) + + return append(R.Bytes(), S.Bytes()...), nil +} + +func (mc *mysqlConn) sendEncryptedPassword(seed []byte, pub *rsa.PublicKey) error { + enc, err := encryptPassword(mc.cfg.Passwd, seed, pub) + if err != nil { + return err + } + return mc.writeAuthSwitchPacket(enc) +} + +func (mc *mysqlConn) auth(authData []byte, plugin string) ([]byte, error) { + switch plugin { + case "caching_sha2_password": + authResp := scrambleSHA256Password(authData, mc.cfg.Passwd) + return authResp, nil + + case "mysql_old_password": + if !mc.cfg.AllowOldPasswords { + return nil, ErrOldPassword + } + if len(mc.cfg.Passwd) == 0 { + return nil, nil + } + // Note: there are edge cases where this should work but doesn't; + // this is currently "wontfix": + // https://github.com/go-sql-driver/mysql/issues/184 + authResp := append(scrambleOldPassword(authData[:8], mc.cfg.Passwd), 0) + return authResp, nil + + case "mysql_clear_password": + if !mc.cfg.AllowCleartextPasswords { + return nil, ErrCleartextPassword + } + // http://dev.mysql.com/doc/refman/5.7/en/cleartext-authentication-plugin.html + // http://dev.mysql.com/doc/refman/5.7/en/pam-authentication-plugin.html + return append([]byte(mc.cfg.Passwd), 0), nil + + case "mysql_native_password": + if !mc.cfg.AllowNativePasswords { + return nil, ErrNativePassword + } + // https://dev.mysql.com/doc/dev/mysql-server/8.4.5/page_protocol_connection_phase_authentication_methods_native_password_authentication.html + // Native password authentication only need and will need 20-byte challenge. + authResp := scramblePassword(authData[:20], mc.cfg.Passwd) + return authResp, nil + + case "sha256_password": + if len(mc.cfg.Passwd) == 0 { + return []byte{0}, nil + } + // unlike caching_sha2_password, sha256_password does not accept + // cleartext password on unix transport. + if mc.cfg.TLS != nil { + // write cleartext auth packet + return append([]byte(mc.cfg.Passwd), 0), nil + } + + pubKey := mc.cfg.pubKey + if pubKey == nil { + // request public key from server + return []byte{1}, nil + } + + // encrypted password + enc, err := encryptPassword(mc.cfg.Passwd, authData, pubKey) + return enc, err + + case "client_ed25519": + if len(authData) != 32 { + return nil, ErrMalformPkt + } + return authEd25519(authData, mc.cfg.Passwd) + + default: + mc.log("unknown auth plugin:", plugin) + return nil, ErrUnknownPlugin + } +} + +func (mc *mysqlConn) handleAuthResult(oldAuthData []byte, plugin string) error { + // Read Result Packet + authData, newPlugin, err := mc.readAuthResult() + if err != nil { + return err + } + + // handle auth plugin switch, if requested + if newPlugin != "" { + // If CLIENT_PLUGIN_AUTH capability is not supported, no new cipher is + // sent and we have to keep using the cipher sent in the init packet. + if authData == nil { + authData = oldAuthData + } else { + // copy data from read buffer to owned slice + copy(oldAuthData, authData) + } + + plugin = newPlugin + + authResp, err := mc.auth(authData, plugin) + if err != nil { + return err + } + if err = mc.writeAuthSwitchPacket(authResp); err != nil { + return err + } + + // Read Result Packet + authData, newPlugin, err = mc.readAuthResult() + if err != nil { + return err + } + + // Do not allow to change the auth plugin more than once + if newPlugin != "" { + return ErrMalformPkt + } + } + + switch plugin { + + // https://dev.mysql.com/blog-archive/preparing-your-community-connector-for-mysql-8-part-2-sha256/ + case "caching_sha2_password": + switch len(authData) { + case 0: + return nil // auth successful + case 1: + switch authData[0] { + case cachingSha2PasswordFastAuthSuccess: + if err = mc.resultUnchanged().readResultOK(); err == nil { + return nil // auth successful + } + + case cachingSha2PasswordPerformFullAuthentication: + if mc.cfg.TLS != nil || mc.cfg.Net == "unix" { + // write cleartext auth packet + err = mc.writeAuthSwitchPacket(append([]byte(mc.cfg.Passwd), 0)) + if err != nil { + return err + } + } else { + pubKey := mc.cfg.pubKey + if pubKey == nil { + // request public key from server + data, err := mc.buf.takeSmallBuffer(4 + 1) + if err != nil { + return err + } + data[4] = cachingSha2PasswordRequestPublicKey + err = mc.writePacket(data) + if err != nil { + return err + } + + if data, err = mc.readPacket(); err != nil { + return err + } + + if data[0] != iAuthMoreData { + return fmt.Errorf("unexpected resp from server for caching_sha2_password, perform full authentication") + } + + // parse public key + block, rest := pem.Decode(data[1:]) + if block == nil { + return fmt.Errorf("no pem data found, data: %s", rest) + } + pkix, err := x509.ParsePKIXPublicKey(block.Bytes) + if err != nil { + return err + } + pubKey = pkix.(*rsa.PublicKey) + } + + // send encrypted password + err = mc.sendEncryptedPassword(oldAuthData, pubKey) + if err != nil { + return err + } + } + return mc.resultUnchanged().readResultOK() + + default: + return ErrMalformPkt + } + default: + return ErrMalformPkt + } + + case "sha256_password": + switch len(authData) { + case 0: + return nil // auth successful + default: + block, _ := pem.Decode(authData) + if block == nil { + return fmt.Errorf("no Pem data found, data: %s", authData) + } + + pub, err := x509.ParsePKIXPublicKey(block.Bytes) + if err != nil { + return err + } + + // send encrypted password + err = mc.sendEncryptedPassword(oldAuthData, pub.(*rsa.PublicKey)) + if err != nil { + return err + } + return mc.resultUnchanged().readResultOK() + } + + default: + return nil // auth successful + } + + return err +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/auth_test.go b/_LICENSES/github.com/go-sql-driver/mysql/auth_test.go new file mode 100644 index 0000000000..46e1e3b4ed --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/auth_test.go @@ -0,0 +1,1381 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2018 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "crypto/rsa" + "crypto/tls" + "crypto/x509" + "encoding/pem" + "fmt" + "testing" +) + +var testPubKey = []byte("-----BEGIN PUBLIC KEY-----\n" + + "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAol0Z8G8U+25Btxk/g/fm\n" + + "UAW/wEKjQCTjkibDE4B+qkuWeiumg6miIRhtilU6m9BFmLQSy1ltYQuu4k17A4tQ\n" + + "rIPpOQYZges/qsDFkZh3wyK5jL5WEFVdOasf6wsfszExnPmcZS4axxoYJfiuilrN\n" + + "hnwinBAqfi3S0sw5MpSI4Zl1AbOrHG4zDI62Gti2PKiMGyYDZTS9xPrBLbN95Kby\n" + + "FFclQLEzA9RJcS1nHFsWtRgHjGPhhjCQxEm9NQ1nePFhCfBfApyfH1VM2VCOQum6\n" + + "Ci9bMuHWjTjckC84mzF99kOxOWVU7mwS6gnJqBzpuz8t3zq8/iQ2y7QrmZV+jTJP\n" + + "WQIDAQAB\n" + + "-----END PUBLIC KEY-----\n") + +var testPubKeyRSA *rsa.PublicKey + +func init() { + block, _ := pem.Decode(testPubKey) + pub, err := x509.ParsePKIXPublicKey(block.Bytes) + if err != nil { + panic(err) + } + testPubKeyRSA = pub.(*rsa.PublicKey) +} + +func TestScrambleOldPass(t *testing.T) { + scramble := []byte{9, 8, 7, 6, 5, 4, 3, 2} + vectors := []struct { + pass string + out string + }{ + {" pass", "47575c5a435b4251"}, + {"pass ", "47575c5a435b4251"}, + {"123\t456", "575c47505b5b5559"}, + {"C0mpl!ca ted#PASS123", "5d5d554849584a45"}, + } + for _, tuple := range vectors { + ours := scrambleOldPassword(scramble, tuple.pass) + if tuple.out != fmt.Sprintf("%x", ours) { + t.Errorf("Failed old password %q", tuple.pass) + } + } +} + +func TestScrambleSHA256Pass(t *testing.T) { + scramble := []byte{10, 47, 74, 111, 75, 73, 34, 48, 88, 76, 114, 74, 37, 13, 3, 80, 82, 2, 23, 21} + vectors := []struct { + pass string + out string + }{ + {"secret", "f490e76f66d9d86665ce54d98c78d0acfe2fb0b08b423da807144873d30b312c"}, + {"secret2", "abc3934a012cf342e876071c8ee202de51785b430258a7a0138bc79c4d800bc6"}, + } + for _, tuple := range vectors { + ours := scrambleSHA256Password(scramble, tuple.pass) + if tuple.out != fmt.Sprintf("%x", ours) { + t.Errorf("Failed SHA256 password %q", tuple.pass) + } + } +} + +func TestAuthFastCachingSHA256PasswordCached(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + + authData := []byte{90, 105, 74, 126, 30, 48, 37, 56, 3, 23, 115, 127, 69, + 22, 41, 84, 32, 123, 43, 118} + plugin := "caching_sha2_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{102, 32, 5, 35, 143, 161, 140, 241, 171, 232, 56, + 139, 43, 14, 107, 196, 249, 170, 147, 60, 220, 204, 120, 178, 214, 15, + 184, 150, 26, 61, 57, 235} + if writtenAuthRespLen != 32 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 2, 0, 0, 2, 1, 3, // Fast Auth Success + 7, 0, 0, 3, 0, 0, 0, 2, 0, 0, 0, // OK + } + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } +} + +func TestAuthFastCachingSHA256PasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "" + + authData := []byte{90, 105, 74, 126, 30, 48, 37, 56, 3, 23, 115, 127, 69, + 22, 41, 84, 32, 123, 43, 118} + plugin := "caching_sha2_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + if writtenAuthRespLen != 0 { + t.Fatalf("unexpected written auth response (%d bytes): %v", + writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 7, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, // OK + } + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } +} + +func TestAuthFastCachingSHA256PasswordFullRSA(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + + authData := []byte{6, 81, 96, 114, 14, 42, 50, 30, 76, 47, 1, 95, 126, 81, + 62, 94, 83, 80, 52, 85} + plugin := "caching_sha2_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{171, 201, 138, 146, 89, 159, 11, 170, 0, 67, 165, + 49, 175, 94, 218, 68, 177, 109, 110, 86, 34, 33, 44, 190, 67, 240, 70, + 110, 40, 139, 124, 41} + if writtenAuthRespLen != 32 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 2, 0, 0, 2, 1, 4, // Perform Full Authentication + } + conn.queuedReplies = [][]byte{ + // pub key response + append([]byte{byte(1 + len(testPubKey)), 1, 0, 4, 1}, testPubKey...), + + // OK + {7, 0, 0, 6, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 3 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + if !bytes.HasPrefix(conn.written, []byte{1, 0, 0, 3, 2, 0, 1, 0, 5}) { + t.Errorf("unexpected written data: %v", conn.written) + } +} + +func TestAuthFastCachingSHA256PasswordFullRSAWithKey(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + mc.cfg.pubKey = testPubKeyRSA + + authData := []byte{6, 81, 96, 114, 14, 42, 50, 30, 76, 47, 1, 95, 126, 81, + 62, 94, 83, 80, 52, 85} + plugin := "caching_sha2_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{171, 201, 138, 146, 89, 159, 11, 170, 0, 67, 165, + 49, 175, 94, 218, 68, 177, 109, 110, 86, 34, 33, 44, 190, 67, 240, 70, + 110, 40, 139, 124, 41} + if writtenAuthRespLen != 32 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 2, 0, 0, 2, 1, 4, // Perform Full Authentication + } + conn.queuedReplies = [][]byte{ + // OK + {7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 2 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + if !bytes.HasPrefix(conn.written, []byte{0, 1, 0, 3}) { + t.Errorf("unexpected written data: %v", conn.written) + } +} + +func TestAuthFastCachingSHA256PasswordFullSecure(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + + authData := []byte{6, 81, 96, 114, 14, 42, 50, 30, 76, 47, 1, 95, 126, 81, + 62, 94, 83, 80, 52, 85} + plugin := "caching_sha2_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // Hack to make the caching_sha2_password plugin believe that the connection + // is secure + mc.cfg.TLS = &tls.Config{InsecureSkipVerify: true} + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{171, 201, 138, 146, 89, 159, 11, 170, 0, 67, 165, + 49, 175, 94, 218, 68, 177, 109, 110, 86, 34, 33, 44, 190, 67, 240, 70, + 110, 40, 139, 124, 41} + if writtenAuthRespLen != 32 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 2, 0, 0, 2, 1, 4, // Perform Full Authentication + } + conn.queuedReplies = [][]byte{ + // OK + {7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 3 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + if !bytes.Equal(conn.written, []byte{7, 0, 0, 3, 115, 101, 99, 114, 101, 116, 0}) { + t.Errorf("unexpected written data: %v", conn.written) + } +} + +func TestAuthFastCleartextPasswordNotAllowed(t *testing.T) { + _, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + + authData := []byte{70, 114, 92, 94, 1, 38, 11, 116, 63, 114, 23, 101, 126, + 103, 26, 95, 81, 17, 24, 21} + plugin := "mysql_clear_password" + + // Send Client Authentication Packet + _, err := mc.auth(authData, plugin) + if err != ErrCleartextPassword { + t.Errorf("expected ErrCleartextPassword, got %v", err) + } +} + +func TestAuthFastCleartextPassword(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + mc.cfg.AllowCleartextPasswords = true + + authData := []byte{70, 114, 92, 94, 1, 38, 11, 116, 63, 114, 23, 101, 126, + 103, 26, 95, 81, 17, 24, 21} + plugin := "mysql_clear_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{115, 101, 99, 114, 101, 116, 0} + if writtenAuthRespLen != 7 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 7, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, // OK + } + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } +} + +func TestAuthFastCleartextPasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "" + mc.cfg.AllowCleartextPasswords = true + + authData := []byte{70, 114, 92, 94, 1, 38, 11, 116, 63, 114, 23, 101, 126, + 103, 26, 95, 81, 17, 24, 21} + plugin := "mysql_clear_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{0} + if writtenAuthRespLen != 1 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 7, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, // OK + } + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } +} + +func TestAuthFastNativePasswordNotAllowed(t *testing.T) { + _, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + mc.cfg.AllowNativePasswords = false + + authData := []byte{70, 114, 92, 94, 1, 38, 11, 116, 63, 114, 23, 101, 126, + 103, 26, 95, 81, 17, 24, 21} + plugin := "mysql_native_password" + + // Send Client Authentication Packet + _, err := mc.auth(authData, plugin) + if err != ErrNativePassword { + t.Errorf("expected ErrNativePassword, got %v", err) + } +} + +func TestAuthFastNativePassword(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + + authData := []byte{70, 114, 92, 94, 1, 38, 11, 116, 63, 114, 23, 101, 126, + 103, 26, 95, 81, 17, 24, 21} + plugin := "mysql_native_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{53, 177, 140, 159, 251, 189, 127, 53, 109, 252, + 172, 50, 211, 192, 240, 164, 26, 48, 207, 45} + if writtenAuthRespLen != 20 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 7, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, // OK + } + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } +} + +func TestAuthFastNativePasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "" + + authData := []byte{70, 114, 92, 94, 1, 38, 11, 116, 63, 114, 23, 101, 126, + 103, 26, 95, 81, 17, 24, 21} + plugin := "mysql_native_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + if writtenAuthRespLen != 0 { + t.Fatalf("unexpected written auth response (%d bytes): %v", + writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 7, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, // OK + } + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } +} + +func TestAuthFastSHA256PasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "" + + authData := []byte{6, 81, 96, 114, 14, 42, 50, 30, 76, 47, 1, 95, 126, 81, + 62, 94, 83, 80, 52, 85} + plugin := "sha256_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{0} + if writtenAuthRespLen != 1 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response (pub key response) + conn.data = append([]byte{byte(1 + len(testPubKey)), 1, 0, 2, 1}, testPubKey...) + conn.queuedReplies = [][]byte{ + // OK + {7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 2 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + if !bytes.HasPrefix(conn.written, []byte{0, 1, 0, 3}) { + t.Errorf("unexpected written data: %v", conn.written) + } +} + +func TestAuthFastSHA256PasswordRSA(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + + authData := []byte{6, 81, 96, 114, 14, 42, 50, 30, 76, 47, 1, 95, 126, 81, + 62, 94, 83, 80, 52, 85} + plugin := "sha256_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{1} + if writtenAuthRespLen != 1 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response (pub key response) + conn.data = append([]byte{byte(1 + len(testPubKey)), 1, 0, 2, 1}, testPubKey...) + conn.queuedReplies = [][]byte{ + // OK + {7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 2 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + if !bytes.HasPrefix(conn.written, []byte{0, 1, 0, 3}) { + t.Errorf("unexpected written data: %v", conn.written) + } +} + +func TestAuthFastSHA256PasswordRSAWithKey(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + mc.cfg.pubKey = testPubKeyRSA + + authData := []byte{6, 81, 96, 114, 14, 42, 50, 30, 76, 47, 1, 95, 126, 81, + 62, 94, 83, 80, 52, 85} + plugin := "sha256_password" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // auth response (OK) + conn.data = []byte{7, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0} + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } +} + +func TestAuthFastSHA256PasswordSecure(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "secret" + + // hack to make the caching_sha2_password plugin believe that the connection + // is secure + mc.cfg.TLS = &tls.Config{InsecureSkipVerify: true} + + authData := []byte{6, 81, 96, 114, 14, 42, 50, 30, 76, 47, 1, 95, 126, 81, + 62, 94, 83, 80, 52, 85} + plugin := "sha256_password" + + // send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + + // unset TLS config to prevent the actual establishment of a TLS wrapper + mc.cfg.TLS = nil + + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{115, 101, 99, 114, 101, 116, 0} + if writtenAuthRespLen != 7 || !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("unexpected written auth response (%d bytes): %v", writtenAuthRespLen, writtenAuthResp) + } + conn.written = nil + + // auth response (OK) + conn.data = []byte{7, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0} + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + if !bytes.Equal(conn.written, []byte{}) { + t.Errorf("unexpected written data: %v", conn.written) + } +} + +func TestAuthSwitchCachingSHA256PasswordCached(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "secret" + + // auth switch request + conn.data = []byte{44, 0, 0, 2, 254, 99, 97, 99, 104, 105, 110, 103, 95, + 115, 104, 97, 50, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 101, + 11, 26, 18, 94, 97, 22, 72, 2, 46, 70, 106, 29, 55, 45, 94, 76, 90, 84, + 50, 0} + + // auth response + conn.queuedReplies = [][]byte{ + {7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}, // OK + } + conn.maxReads = 3 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{ + // 1. Packet: Hash + 32, 0, 0, 3, 219, 72, 64, 97, 56, 197, 167, 203, 64, 236, 168, 80, 223, + 56, 103, 217, 196, 176, 124, 60, 253, 41, 195, 10, 205, 190, 177, 206, 63, + 118, 211, 69, + } + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchCachingSHA256PasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "" + + // auth switch request + conn.data = []byte{44, 0, 0, 2, 254, 99, 97, 99, 104, 105, 110, 103, 95, + 115, 104, 97, 50, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 101, + 11, 26, 18, 94, 97, 22, 72, 2, 46, 70, 106, 29, 55, 45, 94, 76, 90, 84, + 50, 0} + + // auth response + conn.queuedReplies = [][]byte{{7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{0, 0, 0, 3} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchCachingSHA256PasswordFullRSA(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "secret" + + // auth switch request + conn.data = []byte{44, 0, 0, 2, 254, 99, 97, 99, 104, 105, 110, 103, 95, + 115, 104, 97, 50, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 101, + 11, 26, 18, 94, 97, 22, 72, 2, 46, 70, 106, 29, 55, 45, 94, 76, 90, 84, + 50, 0} + + conn.queuedReplies = [][]byte{ + // Perform Full Authentication + {2, 0, 0, 4, 1, 4}, + + // Pub Key Response + append([]byte{byte(1 + len(testPubKey)), 1, 0, 6, 1}, testPubKey...), + + // OK + {7, 0, 0, 8, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 4 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReplyPrefix := []byte{ + // 1. Packet: Hash + 32, 0, 0, 3, 219, 72, 64, 97, 56, 197, 167, 203, 64, 236, 168, 80, 223, + 56, 103, 217, 196, 176, 124, 60, 253, 41, 195, 10, 205, 190, 177, 206, 63, + 118, 211, 69, + + // 2. Packet: Pub Key Request + 1, 0, 0, 5, 2, + + // 3. Packet: Encrypted Password + 0, 1, 0, 7, // [changing bytes] + } + if !bytes.HasPrefix(conn.written, expectedReplyPrefix) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchCachingSHA256PasswordFullRSAWithKey(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "secret" + mc.cfg.pubKey = testPubKeyRSA + + // auth switch request + conn.data = []byte{44, 0, 0, 2, 254, 99, 97, 99, 104, 105, 110, 103, 95, + 115, 104, 97, 50, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 101, + 11, 26, 18, 94, 97, 22, 72, 2, 46, 70, 106, 29, 55, 45, 94, 76, 90, 84, + 50, 0} + + conn.queuedReplies = [][]byte{ + // Perform Full Authentication + {2, 0, 0, 4, 1, 4}, + + // OK + {7, 0, 0, 6, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 3 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReplyPrefix := []byte{ + // 1. Packet: Hash + 32, 0, 0, 3, 219, 72, 64, 97, 56, 197, 167, 203, 64, 236, 168, 80, 223, + 56, 103, 217, 196, 176, 124, 60, 253, 41, 195, 10, 205, 190, 177, 206, 63, + 118, 211, 69, + + // 2. Packet: Encrypted Password + 0, 1, 0, 5, // [changing bytes] + } + if !bytes.HasPrefix(conn.written, expectedReplyPrefix) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchCachingSHA256PasswordFullSecure(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "secret" + + // Hack to make the caching_sha2_password plugin believe that the connection + // is secure + mc.cfg.TLS = &tls.Config{InsecureSkipVerify: true} + + // auth switch request + conn.data = []byte{44, 0, 0, 2, 254, 99, 97, 99, 104, 105, 110, 103, 95, + 115, 104, 97, 50, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 101, + 11, 26, 18, 94, 97, 22, 72, 2, 46, 70, 106, 29, 55, 45, 94, 76, 90, 84, + 50, 0} + + // auth response + conn.queuedReplies = [][]byte{ + {2, 0, 0, 4, 1, 4}, // Perform Full Authentication + {7, 0, 0, 6, 0, 0, 0, 2, 0, 0, 0}, // OK + } + conn.maxReads = 3 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{ + // 1. Packet: Hash + 32, 0, 0, 3, 219, 72, 64, 97, 56, 197, 167, 203, 64, 236, 168, 80, 223, + 56, 103, 217, 196, 176, 124, 60, 253, 41, 195, 10, 205, 190, 177, 206, 63, + 118, 211, 69, + + // 2. Packet: Cleartext password + 7, 0, 0, 5, 115, 101, 99, 114, 101, 116, 0, + } + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchCleartextPasswordNotAllowed(t *testing.T) { + conn, mc := newRWMockConn(2) + + conn.data = []byte{22, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 99, 108, + 101, 97, 114, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0} + conn.maxReads = 1 + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + err := mc.handleAuthResult(authData, plugin) + if err != ErrCleartextPassword { + t.Errorf("expected ErrCleartextPassword, got %v", err) + } +} + +func TestAuthSwitchCleartextPassword(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowCleartextPasswords = true + mc.cfg.Passwd = "secret" + + // auth switch request + conn.data = []byte{22, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 99, 108, + 101, 97, 114, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0} + + // auth response + conn.queuedReplies = [][]byte{{7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{7, 0, 0, 3, 115, 101, 99, 114, 101, 116, 0} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchCleartextPasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowCleartextPasswords = true + mc.cfg.Passwd = "" + + // auth switch request + conn.data = []byte{22, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 99, 108, + 101, 97, 114, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0} + + // auth response + conn.queuedReplies = [][]byte{{7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{1, 0, 0, 3, 0} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchNativePasswordNotAllowed(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowNativePasswords = false + + conn.data = []byte{44, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 110, 97, + 116, 105, 118, 101, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 96, + 71, 63, 8, 1, 58, 75, 12, 69, 95, 66, 60, 117, 31, 48, 31, 89, 39, 55, + 31, 0} + conn.maxReads = 1 + authData := []byte{96, 71, 63, 8, 1, 58, 75, 12, 69, 95, 66, 60, 117, 31, + 48, 31, 89, 39, 55, 31} + plugin := "caching_sha2_password" + err := mc.handleAuthResult(authData, plugin) + if err != ErrNativePassword { + t.Errorf("expected ErrNativePassword, got %v", err) + } +} + +func TestAuthSwitchNativePassword(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowNativePasswords = true + mc.cfg.Passwd = "secret" + + // auth switch request + conn.data = []byte{44, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 110, 97, + 116, 105, 118, 101, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 96, + 71, 63, 8, 1, 58, 75, 12, 69, 95, 66, 60, 117, 31, 48, 31, 89, 39, 55, + 31, 0} + + // auth response + conn.queuedReplies = [][]byte{{7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{96, 71, 63, 8, 1, 58, 75, 12, 69, 95, 66, 60, 117, 31, + 48, 31, 89, 39, 55, 31} + plugin := "caching_sha2_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{20, 0, 0, 3, 202, 41, 195, 164, 34, 226, 49, 103, + 21, 211, 167, 199, 227, 116, 8, 48, 57, 71, 149, 146} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchNativePasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowNativePasswords = true + mc.cfg.Passwd = "" + + // auth switch request + conn.data = []byte{44, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 110, 97, + 116, 105, 118, 101, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 96, + 71, 63, 8, 1, 58, 75, 12, 69, 95, 66, 60, 117, 31, 48, 31, 89, 39, 55, + 31, 0} + + // auth response + conn.queuedReplies = [][]byte{{7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{96, 71, 63, 8, 1, 58, 75, 12, 69, 95, 66, 60, 117, 31, + 48, 31, 89, 39, 55, 31} + plugin := "caching_sha2_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{0, 0, 0, 3} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchOldPasswordNotAllowed(t *testing.T) { + conn, mc := newRWMockConn(2) + + conn.data = []byte{41, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 111, 108, + 100, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 95, 84, 103, 43, 61, + 49, 123, 61, 91, 50, 40, 113, 35, 84, 96, 101, 92, 123, 121, 107, 0} + conn.maxReads = 1 + authData := []byte{95, 84, 103, 43, 61, 49, 123, 61, 91, 50, 40, 113, 35, + 84, 96, 101, 92, 123, 121, 107} + plugin := "mysql_native_password" + err := mc.handleAuthResult(authData, plugin) + if err != ErrOldPassword { + t.Errorf("expected ErrOldPassword, got %v", err) + } +} + +// Same to TestAuthSwitchOldPasswordNotAllowed, but use OldAuthSwitch request. +func TestOldAuthSwitchNotAllowed(t *testing.T) { + conn, mc := newRWMockConn(2) + + // OldAuthSwitch request + conn.data = []byte{1, 0, 0, 2, 0xfe} + conn.maxReads = 1 + authData := []byte{95, 84, 103, 43, 61, 49, 123, 61, 91, 50, 40, 113, 35, + 84, 96, 101, 92, 123, 121, 107} + plugin := "mysql_native_password" + err := mc.handleAuthResult(authData, plugin) + if err != ErrOldPassword { + t.Errorf("expected ErrOldPassword, got %v", err) + } +} + +func TestAuthSwitchOldPassword(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowOldPasswords = true + mc.cfg.Passwd = "secret" + + // auth switch request + conn.data = []byte{41, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 111, 108, + 100, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 95, 84, 103, 43, 61, + 49, 123, 61, 91, 50, 40, 113, 35, 84, 96, 101, 92, 123, 121, 107, 0} + + // auth response + conn.queuedReplies = [][]byte{{8, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{95, 84, 103, 43, 61, 49, 123, 61, 91, 50, 40, 113, 35, + 84, 96, 101, 92, 123, 121, 107} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{9, 0, 0, 3, 86, 83, 83, 79, 74, 78, 65, 66, 0} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +// Same to TestAuthSwitchOldPassword, but use OldAuthSwitch request. +func TestOldAuthSwitch(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowOldPasswords = true + mc.cfg.Passwd = "secret" + + // OldAuthSwitch request + conn.data = []byte{1, 0, 0, 2, 0xfe} + + // auth response + conn.queuedReplies = [][]byte{{8, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{95, 84, 103, 43, 61, 49, 123, 61, 91, 50, 40, 113, 35, + 84, 96, 101, 92, 123, 121, 107} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{9, 0, 0, 3, 86, 83, 83, 79, 74, 78, 65, 66, 0} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} +func TestAuthSwitchOldPasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowOldPasswords = true + mc.cfg.Passwd = "" + + // auth switch request + conn.data = []byte{41, 0, 0, 2, 254, 109, 121, 115, 113, 108, 95, 111, 108, + 100, 95, 112, 97, 115, 115, 119, 111, 114, 100, 0, 95, 84, 103, 43, 61, + 49, 123, 61, 91, 50, 40, 113, 35, 84, 96, 101, 92, 123, 121, 107, 0} + + // auth response + conn.queuedReplies = [][]byte{{8, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{95, 84, 103, 43, 61, 49, 123, 61, 91, 50, 40, 113, 35, + 84, 96, 101, 92, 123, 121, 107} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{0, 0, 0, 3} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +// Same to TestAuthSwitchOldPasswordEmpty, but use OldAuthSwitch request. +func TestOldAuthSwitchPasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.AllowOldPasswords = true + mc.cfg.Passwd = "" + + // OldAuthSwitch request. + conn.data = []byte{1, 0, 0, 2, 0xfe} + + // auth response + conn.queuedReplies = [][]byte{{8, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0, 0}} + conn.maxReads = 2 + + authData := []byte{95, 84, 103, 43, 61, 49, 123, 61, 91, 50, 40, 113, 35, + 84, 96, 101, 92, 123, 121, 107} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReply := []byte{0, 0, 0, 3} + if !bytes.Equal(conn.written, expectedReply) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchSHA256PasswordEmpty(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "" + + // auth switch request + conn.data = []byte{38, 0, 0, 2, 254, 115, 104, 97, 50, 53, 54, 95, 112, 97, + 115, 115, 119, 111, 114, 100, 0, 78, 82, 62, 40, 100, 1, 59, 31, 44, 69, + 33, 112, 8, 81, 51, 96, 65, 82, 16, 114, 0} + + conn.queuedReplies = [][]byte{ + // OK + {7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 3 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReplyPrefix := []byte{ + // 1. Packet: Empty Password + 1, 0, 0, 3, 0, + } + if !bytes.HasPrefix(conn.written, expectedReplyPrefix) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchSHA256PasswordRSA(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "secret" + + // auth switch request + conn.data = []byte{38, 0, 0, 2, 254, 115, 104, 97, 50, 53, 54, 95, 112, 97, + 115, 115, 119, 111, 114, 100, 0, 78, 82, 62, 40, 100, 1, 59, 31, 44, 69, + 33, 112, 8, 81, 51, 96, 65, 82, 16, 114, 0} + + conn.queuedReplies = [][]byte{ + // Pub Key Response + append([]byte{byte(1 + len(testPubKey)), 1, 0, 4, 1}, testPubKey...), + + // OK + {7, 0, 0, 6, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 3 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReplyPrefix := []byte{ + // 1. Packet: Pub Key Request + 1, 0, 0, 3, 1, + + // 2. Packet: Encrypted Password + 0, 1, 0, 5, // [changing bytes] + } + if !bytes.HasPrefix(conn.written, expectedReplyPrefix) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchSHA256PasswordRSAWithKey(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "secret" + mc.cfg.pubKey = testPubKeyRSA + + // auth switch request + conn.data = []byte{38, 0, 0, 2, 254, 115, 104, 97, 50, 53, 54, 95, 112, 97, + 115, 115, 119, 111, 114, 100, 0, 78, 82, 62, 40, 100, 1, 59, 31, 44, 69, + 33, 112, 8, 81, 51, 96, 65, 82, 16, 114, 0} + + conn.queuedReplies = [][]byte{ + // OK + {7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 2 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReplyPrefix := []byte{ + // 1. Packet: Encrypted Password + 0, 1, 0, 3, // [changing bytes] + } + if !bytes.HasPrefix(conn.written, expectedReplyPrefix) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +func TestAuthSwitchSHA256PasswordSecure(t *testing.T) { + conn, mc := newRWMockConn(2) + mc.cfg.Passwd = "secret" + + // Hack to make the caching_sha2_password plugin believe that the connection + // is secure + mc.cfg.TLS = &tls.Config{InsecureSkipVerify: true} + + // auth switch request + conn.data = []byte{38, 0, 0, 2, 254, 115, 104, 97, 50, 53, 54, 95, 112, 97, + 115, 115, 119, 111, 114, 100, 0, 78, 82, 62, 40, 100, 1, 59, 31, 44, 69, + 33, 112, 8, 81, 51, 96, 65, 82, 16, 114, 0} + + conn.queuedReplies = [][]byte{ + // OK + {7, 0, 0, 4, 0, 0, 0, 2, 0, 0, 0}, + } + conn.maxReads = 2 + + authData := []byte{123, 87, 15, 84, 20, 58, 37, 121, 91, 117, 51, 24, 19, + 47, 43, 9, 41, 112, 67, 110} + plugin := "mysql_native_password" + + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } + + expectedReplyPrefix := []byte{ + // 1. Packet: Cleartext Password + 7, 0, 0, 3, 115, 101, 99, 114, 101, 116, 0, + } + if !bytes.Equal(conn.written, expectedReplyPrefix) { + t.Errorf("got unexpected data: %v", conn.written) + } +} + +// Derived from https://github.com/MariaDB/server/blob/6b2287fff23fbdc362499501c562f01d0d2db52e/plugin/auth_ed25519/ed25519-t.c +func TestEd25519Auth(t *testing.T) { + conn, mc := newRWMockConn(1) + mc.cfg.User = "root" + mc.cfg.Passwd = "foobar" + + authData := []byte("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") + plugin := "client_ed25519" + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + t.Fatal(err) + } + err = mc.writeHandshakeResponsePacket(authResp, plugin) + if err != nil { + t.Fatal(err) + } + + // check written auth response + authRespStart := 4 + 4 + 4 + 1 + 23 + len(mc.cfg.User) + 1 + authRespEnd := authRespStart + 1 + len(authResp) + writtenAuthRespLen := conn.written[authRespStart] + writtenAuthResp := conn.written[authRespStart+1 : authRespEnd] + expectedAuthResp := []byte{ + 232, 61, 201, 63, 67, 63, 51, 53, 86, 73, 238, 35, 170, 117, 146, + 214, 26, 17, 35, 9, 8, 132, 245, 141, 48, 99, 66, 58, 36, 228, 48, + 84, 115, 254, 187, 168, 88, 162, 249, 57, 35, 85, 79, 238, 167, 106, + 68, 117, 56, 135, 171, 47, 20, 14, 133, 79, 15, 229, 124, 160, 176, + 100, 138, 14, + } + if writtenAuthRespLen != 64 { + t.Fatalf("expected 64 bytes from client, got %d", writtenAuthRespLen) + } + if !bytes.Equal(writtenAuthResp, expectedAuthResp) { + t.Fatalf("auth response did not match expected value:\n%v\n%v", writtenAuthResp, expectedAuthResp) + } + conn.written = nil + + // auth response + conn.data = []byte{ + 7, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, // OK + } + conn.maxReads = 1 + + // Handle response to auth packet + if err := mc.handleAuthResult(authData, plugin); err != nil { + t.Errorf("got error: %v", err) + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/benchmark_test.go b/_LICENSES/github.com/go-sql-driver/mysql/benchmark_test.go new file mode 100644 index 0000000000..87844a9b2a --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/benchmark_test.go @@ -0,0 +1,511 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "context" + "database/sql" + "database/sql/driver" + "fmt" + "math" + "runtime" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +type TB testing.B + +func (tb *TB) check(err error) { + if err != nil { + tb.Fatal(err) + } +} + +func (tb *TB) checkDB(db *sql.DB, err error) *sql.DB { + tb.check(err) + return db +} + +func (tb *TB) checkRows(rows *sql.Rows, err error) *sql.Rows { + tb.check(err) + return rows +} + +func (tb *TB) checkStmt(stmt *sql.Stmt, err error) *sql.Stmt { + tb.check(err) + return stmt +} + +func initDB(b *testing.B, compress bool, queries ...string) *sql.DB { + tb := (*TB)(b) + comprStr := "" + if compress { + comprStr = "&compress=1" + } + db := tb.checkDB(sql.Open(driverNameTest, dsn+comprStr)) + for _, query := range queries { + if _, err := db.Exec(query); err != nil { + b.Fatalf("error on %q: %v", query, err) + } + } + return db +} + +const concurrencyLevel = 10 + +func BenchmarkQuery(b *testing.B) { + benchmarkQuery(b, false) +} + +func BenchmarkQueryCompressed(b *testing.B) { + benchmarkQuery(b, true) +} + +func benchmarkQuery(b *testing.B, compr bool) { + tb := (*TB)(b) + b.ReportAllocs() + db := initDB(b, compr, + "DROP TABLE IF EXISTS foo", + "CREATE TABLE foo (id INT PRIMARY KEY, val CHAR(50))", + `INSERT INTO foo VALUES (1, "one")`, + `INSERT INTO foo VALUES (2, "two")`, + ) + db.SetMaxIdleConns(concurrencyLevel) + defer db.Close() + + stmt := tb.checkStmt(db.Prepare("SELECT val FROM foo WHERE id=?")) + defer stmt.Close() + + remain := int64(b.N) + var wg sync.WaitGroup + wg.Add(concurrencyLevel) + defer wg.Wait() + b.StartTimer() + + for range concurrencyLevel { + go func() { + for { + if atomic.AddInt64(&remain, -1) < 0 { + wg.Done() + return + } + + var got string + tb.check(stmt.QueryRow(1).Scan(&got)) + if got != "one" { + b.Errorf("query = %q; want one", got) + wg.Done() + return + } + } + }() + } +} + +func BenchmarkExec(b *testing.B) { + tb := (*TB)(b) + db := tb.checkDB(sql.Open(driverNameTest, dsn)) + db.SetMaxIdleConns(concurrencyLevel) + defer db.Close() + + stmt := tb.checkStmt(db.Prepare("DO 1")) + defer stmt.Close() + + remain := int64(b.N) + var wg sync.WaitGroup + wg.Add(concurrencyLevel) + defer wg.Wait() + + b.ReportAllocs() + b.ResetTimer() + + for range concurrencyLevel { + go func() { + for { + if atomic.AddInt64(&remain, -1) < 0 { + wg.Done() + return + } + + if _, err := stmt.Exec(); err != nil { + b.Logf("stmt.Exec failed: %v", err) + b.Fail() + } + } + }() + } +} + +// data, but no db writes +var roundtripSample []byte + +func initRoundtripBenchmarks() ([]byte, int, int) { + if roundtripSample == nil { + roundtripSample = []byte(strings.Repeat("0123456789abcdef", 1024*1024)) + } + return roundtripSample, 16, len(roundtripSample) +} + +func BenchmarkRoundtripTxt(b *testing.B) { + sample, min, max := initRoundtripBenchmarks() + sampleString := string(sample) + tb := (*TB)(b) + db := tb.checkDB(sql.Open(driverNameTest, dsn)) + defer db.Close() + + b.ReportAllocs() + + var result string + for i := 0; b.Loop(); i++ { + length := min + i + if length > max { + length = max + } + test := sampleString[0:length] + rows := tb.checkRows(db.Query(`SELECT "` + test + `"`)) + if !rows.Next() { + rows.Close() + b.Fatalf("crashed") + } + err := rows.Scan(&result) + if err != nil { + rows.Close() + b.Fatalf("crashed") + } + if result != test { + rows.Close() + b.Errorf("mismatch") + } + rows.Close() + } +} + +func BenchmarkRoundtripBin(b *testing.B) { + sample, min, max := initRoundtripBenchmarks() + tb := (*TB)(b) + db := tb.checkDB(sql.Open(driverNameTest, dsn)) + defer db.Close() + stmt := tb.checkStmt(db.Prepare("SELECT ?")) + defer stmt.Close() + + b.ReportAllocs() + + var result sql.RawBytes + for i := 0; b.Loop(); i++ { + length := min + i + if length > max { + length = max + } + test := sample[0:length] + rows := tb.checkRows(stmt.Query(test)) + if !rows.Next() { + rows.Close() + b.Fatalf("crashed") + } + err := rows.Scan(&result) + if err != nil { + rows.Close() + b.Fatalf("crashed") + } + if !bytes.Equal(result, test) { + rows.Close() + b.Errorf("mismatch") + } + rows.Close() + } +} + +func BenchmarkInterpolation(b *testing.B) { + mc := &mysqlConn{ + cfg: &Config{ + InterpolateParams: true, + Loc: time.UTC, + }, + maxAllowedPacket: maxPacketSize, + maxWriteSize: maxPacketSize - 1, + buf: newBuffer(), + } + + args := []driver.Value{ + int64(42424242), + float64(math.Pi), + false, + time.Unix(1423411542, 807015000), + []byte("bytes containing special chars ' \" \a \x00"), + "string containing special chars ' \" \a \x00", + } + q := "SELECT ?, ?, ?, ?, ?, ?" + + b.ReportAllocs() + + for b.Loop() { + _, err := mc.interpolateParams(q, args) + if err != nil { + b.Fatal(err) + } + } +} + +func benchmarkQueryContext(b *testing.B, db *sql.DB, p int) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db.SetMaxIdleConns(p * runtime.GOMAXPROCS(0)) + + tb := (*TB)(b) + stmt := tb.checkStmt(db.PrepareContext(ctx, "SELECT val FROM foo WHERE id=?")) + defer stmt.Close() + + b.SetParallelism(p) + b.ReportAllocs() + b.ResetTimer() + b.RunParallel(func(pb *testing.PB) { + var got string + for pb.Next() { + tb.check(stmt.QueryRow(1).Scan(&got)) + if got != "one" { + b.Fatalf("query = %q; want one", got) + } + } + }) +} + +func BenchmarkQueryContext(b *testing.B) { + db := initDB(b, false, + "DROP TABLE IF EXISTS foo", + "CREATE TABLE foo (id INT PRIMARY KEY, val CHAR(50))", + `INSERT INTO foo VALUES (1, "one")`, + `INSERT INTO foo VALUES (2, "two")`, + ) + defer db.Close() + for _, p := range []int{1, 2, 3, 4} { + b.Run(fmt.Sprintf("%d", p), func(b *testing.B) { + benchmarkQueryContext(b, db, p) + }) + } +} + +func benchmarkExecContext(b *testing.B, db *sql.DB, p int) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db.SetMaxIdleConns(p * runtime.GOMAXPROCS(0)) + + tb := (*TB)(b) + stmt := tb.checkStmt(db.PrepareContext(ctx, "DO 1")) + defer stmt.Close() + + b.SetParallelism(p) + b.ReportAllocs() + b.ResetTimer() + b.RunParallel(func(pb *testing.PB) { + for pb.Next() { + if _, err := stmt.ExecContext(ctx); err != nil { + b.Fatal(err) + } + } + }) +} + +func BenchmarkExecContext(b *testing.B) { + db := initDB(b, false, + "DROP TABLE IF EXISTS foo", + "CREATE TABLE foo (id INT PRIMARY KEY, val CHAR(50))", + `INSERT INTO foo VALUES (1, "one")`, + `INSERT INTO foo VALUES (2, "two")`, + ) + defer db.Close() + for _, p := range []int{1, 2, 3, 4} { + b.Run(fmt.Sprintf("%d", p), func(b *testing.B) { + benchmarkExecContext(b, db, p) + }) + } +} + +// BenchmarkQueryRawBytes benchmarks fetching 100 blobs using sql.RawBytes. +// "size=" means size of each blobs. +func BenchmarkQueryRawBytes(b *testing.B) { + var sizes []int = []int{100, 1000, 2000, 4000, 8000, 12000, 16000, 32000, 64000, 256000} + db := initDB(b, false, + "DROP TABLE IF EXISTS bench_rawbytes", + "CREATE TABLE bench_rawbytes (id INT PRIMARY KEY, val LONGBLOB)", + ) + defer db.Close() + + blob := make([]byte, sizes[len(sizes)-1]) + for i := range blob { + blob[i] = 42 + } + for i := range 100 { + _, err := db.Exec("INSERT INTO bench_rawbytes VALUES (?, ?)", i, blob) + if err != nil { + b.Fatal(err) + } + } + + for _, s := range sizes { + b.Run(fmt.Sprintf("size=%v", s), func(b *testing.B) { + db.SetMaxIdleConns(0) + db.SetMaxIdleConns(1) + b.ReportAllocs() + b.ResetTimer() + + for j := 0; j < b.N; j++ { + rows, err := db.Query("SELECT LEFT(val, ?) as v FROM bench_rawbytes", s) + if err != nil { + b.Fatal(err) + } + nrows := 0 + for rows.Next() { + var buf sql.RawBytes + err := rows.Scan(&buf) + if err != nil { + b.Fatal(err) + } + if len(buf) != s { + b.Fatalf("size mismatch: expected %v, got %v", s, len(buf)) + } + nrows++ + } + rows.Close() + if nrows != 100 { + b.Fatalf("numbers of rows mismatch: expected %v, got %v", 100, nrows) + } + } + }) + } +} + +func benchmark10kRows(b *testing.B, compress bool) { + // Setup -- prepare 10000 rows. + db := initDB(b, compress, + "DROP TABLE IF EXISTS foo", + "CREATE TABLE foo (id INT PRIMARY KEY, val TEXT)") + defer db.Close() + + sval := strings.Repeat("x", 50) + stmt, err := db.Prepare(`INSERT INTO foo (id, val) VALUES (?, ?)` + strings.Repeat(",(?,?)", 99)) + if err != nil { + b.Errorf("failed to prepare query: %v", err) + return + } + + args := make([]any, 200) + for i := 1; i < 200; i += 2 { + args[i] = sval + } + for i := 0; i < 10000; i += 100 { + for j := range 100 { + args[j*2] = i + j + } + _, err := stmt.Exec(args...) + if err != nil { + b.Error(err) + return + } + } + stmt.Close() + + // benchmark function called several times with different b.N. + // it means heavy setup is called multiple times. + // Use b.Run() to run expensive setup only once. + // Go 1.24 introduced b.Loop() for this purpose. But we keep this + // benchmark compatible with Go 1.20. + b.Run("query", func(b *testing.B) { + b.ReportAllocs() + for i := 0; i < b.N; i++ { + rows, err := db.Query(`SELECT id, val FROM foo`) + if err != nil { + b.Errorf("failed to select: %v", err) + return + } + // rows.Scan() escapes arguments. So these variables must be defined + // before loop. + var i int + var s sql.RawBytes + for rows.Next() { + if err := rows.Scan(&i, &s); err != nil { + b.Errorf("failed to scan: %v", err) + rows.Close() + return + } + } + if err = rows.Err(); err != nil { + b.Errorf("failed to read rows: %v", err) + } + rows.Close() + } + }) +} + +// BenchmarkReceive10kRows measures performance of receiving large number of rows. +func BenchmarkReceive10kRows(b *testing.B) { + benchmark10kRows(b, false) +} + +func BenchmarkReceive10kRowsCompressed(b *testing.B) { + benchmark10kRows(b, true) +} + +// BenchmarkReceiveMetadata measures performance of receiving lots of metadata compare to data in rows +func BenchmarkReceiveMetadata(b *testing.B) { + tb := (*TB)(b) + + // Create a table with 1000 integer fields + var createTableQuery strings.Builder + createTableQuery.WriteString("CREATE TABLE large_integer_table (") + for i := range 1000 { + createTableQuery.WriteString(fmt.Sprintf("col_%d INT", i)) + if i < 999 { + createTableQuery.WriteString(", ") + } + } + createTableQuery.WriteString(")") + + // Initialize database + db := initDB(b, false, + "DROP TABLE IF EXISTS large_integer_table", + createTableQuery.String(), + "INSERT INTO large_integer_table VALUES ("+ + strings.Repeat("0,", 999)+"0)", // Insert a row of zeros + ) + defer db.Close() + + b.Run("query", func(b *testing.B) { + db.SetMaxIdleConns(0) + db.SetMaxIdleConns(1) + + // Create a slice to scan all columns + values := make([]any, 1000) + valuePtrs := make([]any, 1000) + for j := range values { + valuePtrs[j] = &values[j] + } + + // Prepare a SELECT query to retrieve metadata + stmt := tb.checkStmt(db.Prepare("SELECT * FROM large_integer_table LIMIT 1")) + defer stmt.Close() + + // Benchmark metadata retrieval + b.ReportAllocs() + b.ResetTimer() + for range b.N { + rows := tb.checkRows(stmt.Query()) + + rows.Next() + // Scan the row + err := rows.Scan(valuePtrs...) + tb.check(err) + + rows.Close() + } + }) +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/buffer.go b/_LICENSES/github.com/go-sql-driver/mysql/buffer.go new file mode 100644 index 0000000000..f895e87b3c --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/buffer.go @@ -0,0 +1,149 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "io" +) + +const defaultBufSize = 4096 +const maxCachedBufSize = 256 * 1024 + +// readerFunc is a function that compatible with io.Reader. +// We use this function type instead of io.Reader because we want to +// just pass mc.readWithTimeout. +type readerFunc func([]byte) (int, error) + +// A buffer which is used for both reading and writing. +// This is possible since communication on each connection is synchronous. +// In other words, we can't write and read simultaneously on the same connection. +// The buffer is similar to bufio.Reader / Writer but zero-copy-ish +// Also highly optimized for this particular use case. +type buffer struct { + buf []byte // read buffer. + cachedBuf []byte // buffer that will be reused. len(cachedBuf) <= maxCachedBufSize. +} + +// newBuffer allocates and returns a new buffer. +func newBuffer() buffer { + return buffer{ + cachedBuf: make([]byte, defaultBufSize), + } +} + +// busy returns true if the read buffer is not empty. +func (b *buffer) busy() bool { + return len(b.buf) > 0 +} + +// len returns how many bytes in the read buffer. +func (b *buffer) len() int { + return len(b.buf) +} + +// fill reads into the read buffer until at least _need_ bytes are in it. +func (b *buffer) fill(need int, r readerFunc) error { + // we'll move the contents of the current buffer to dest before filling it. + dest := b.cachedBuf + + // grow buffer if necessary to fit the whole packet. + if need > len(dest) { + // Round up to the next multiple of the default size + dest = make([]byte, ((need/defaultBufSize)+1)*defaultBufSize) + + // if the allocated buffer is not too large, move it to backing storage + // to prevent extra allocations on applications that perform large reads + if len(dest) <= maxCachedBufSize { + b.cachedBuf = dest + } + } + + // move the existing data to the start of the buffer. + n := len(b.buf) + copy(dest[:n], b.buf) + + for { + nn, err := r(dest[n:]) + n += nn + + if err == nil && n < need { + continue + } + + b.buf = dest[:n] + + if err == io.EOF { + if n < need { + err = io.ErrUnexpectedEOF + } else { + err = nil + } + } + return err + } +} + +// returns next N bytes from buffer. +// The returned slice is only guaranteed to be valid until the next read +func (b *buffer) readNext(need int) []byte { + data := b.buf[:need:need] + b.buf = b.buf[need:] + return data +} + +// takeBuffer returns a buffer with the requested size. +// If possible, a slice from the existing buffer is returned. +// Otherwise a bigger buffer is made. +// Only one buffer (total) can be used at a time. +func (b *buffer) takeBuffer(length int) ([]byte, error) { + if b.busy() { + return nil, ErrBusyBuffer + } + + // test (cheap) general case first + if length <= len(b.cachedBuf) { + return b.cachedBuf[:length], nil + } + + if length < maxCachedBufSize { + b.cachedBuf = make([]byte, length) + return b.cachedBuf, nil + } + + // buffer is larger than we want to store. + return make([]byte, length), nil +} + +// takeSmallBuffer is shortcut which can be used if length is +// known to be smaller than defaultBufSize. +// Only one buffer (total) can be used at a time. +func (b *buffer) takeSmallBuffer(length int) ([]byte, error) { + if b.busy() { + return nil, ErrBusyBuffer + } + return b.cachedBuf[:length], nil +} + +// takeCompleteBuffer returns the complete existing buffer. +// This can be used if the necessary buffer size is unknown. +// cap and len of the returned buffer will be equal. +// Only one buffer (total) can be used at a time. +func (b *buffer) takeCompleteBuffer() ([]byte, error) { + if b.busy() { + return nil, ErrBusyBuffer + } + return b.cachedBuf, nil +} + +// store stores buf, an updated buffer, if its suitable to do so. +func (b *buffer) store(buf []byte) { + if cap(buf) <= maxCachedBufSize && cap(buf) > cap(b.cachedBuf) { + b.cachedBuf = buf[:cap(buf)] + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/collations.go b/_LICENSES/github.com/go-sql-driver/mysql/collations.go new file mode 100644 index 0000000000..29b1aa43f2 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/collations.go @@ -0,0 +1,266 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2014 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +const defaultCollationID = 45 // utf8mb4_general_ci +const binaryCollationID = 63 + +// A list of available collations mapped to the internal ID. +// To update this map use the following MySQL query: +// +// SELECT COLLATION_NAME, ID FROM information_schema.COLLATIONS WHERE ID<256 ORDER BY ID +// +// Handshake packet have only 1 byte for collation_id. So we can't use collations with ID > 255. +// +// ucs2, utf16, and utf32 can't be used for connection charset. +// https://dev.mysql.com/doc/refman/5.7/en/charset-connection.html#charset-connection-impermissible-client-charset +// They are commented out to reduce this map. +var collations = map[string]byte{ + "big5_chinese_ci": 1, + "latin2_czech_cs": 2, + "dec8_swedish_ci": 3, + "cp850_general_ci": 4, + "latin1_german1_ci": 5, + "hp8_english_ci": 6, + "koi8r_general_ci": 7, + "latin1_swedish_ci": 8, + "latin2_general_ci": 9, + "swe7_swedish_ci": 10, + "ascii_general_ci": 11, + "ujis_japanese_ci": 12, + "sjis_japanese_ci": 13, + "cp1251_bulgarian_ci": 14, + "latin1_danish_ci": 15, + "hebrew_general_ci": 16, + "tis620_thai_ci": 18, + "euckr_korean_ci": 19, + "latin7_estonian_cs": 20, + "latin2_hungarian_ci": 21, + "koi8u_general_ci": 22, + "cp1251_ukrainian_ci": 23, + "gb2312_chinese_ci": 24, + "greek_general_ci": 25, + "cp1250_general_ci": 26, + "latin2_croatian_ci": 27, + "gbk_chinese_ci": 28, + "cp1257_lithuanian_ci": 29, + "latin5_turkish_ci": 30, + "latin1_german2_ci": 31, + "armscii8_general_ci": 32, + "utf8_general_ci": 33, + "cp1250_czech_cs": 34, + //"ucs2_general_ci": 35, + "cp866_general_ci": 36, + "keybcs2_general_ci": 37, + "macce_general_ci": 38, + "macroman_general_ci": 39, + "cp852_general_ci": 40, + "latin7_general_ci": 41, + "latin7_general_cs": 42, + "macce_bin": 43, + "cp1250_croatian_ci": 44, + "utf8mb4_general_ci": 45, + "utf8mb4_bin": 46, + "latin1_bin": 47, + "latin1_general_ci": 48, + "latin1_general_cs": 49, + "cp1251_bin": 50, + "cp1251_general_ci": 51, + "cp1251_general_cs": 52, + "macroman_bin": 53, + //"utf16_general_ci": 54, + //"utf16_bin": 55, + //"utf16le_general_ci": 56, + "cp1256_general_ci": 57, + "cp1257_bin": 58, + "cp1257_general_ci": 59, + //"utf32_general_ci": 60, + //"utf32_bin": 61, + //"utf16le_bin": 62, + "binary": 63, + "armscii8_bin": 64, + "ascii_bin": 65, + "cp1250_bin": 66, + "cp1256_bin": 67, + "cp866_bin": 68, + "dec8_bin": 69, + "greek_bin": 70, + "hebrew_bin": 71, + "hp8_bin": 72, + "keybcs2_bin": 73, + "koi8r_bin": 74, + "koi8u_bin": 75, + "utf8_tolower_ci": 76, + "latin2_bin": 77, + "latin5_bin": 78, + "latin7_bin": 79, + "cp850_bin": 80, + "cp852_bin": 81, + "swe7_bin": 82, + "utf8_bin": 83, + "big5_bin": 84, + "euckr_bin": 85, + "gb2312_bin": 86, + "gbk_bin": 87, + "sjis_bin": 88, + "tis620_bin": 89, + //"ucs2_bin": 90, + "ujis_bin": 91, + "geostd8_general_ci": 92, + "geostd8_bin": 93, + "latin1_spanish_ci": 94, + "cp932_japanese_ci": 95, + "cp932_bin": 96, + "eucjpms_japanese_ci": 97, + "eucjpms_bin": 98, + "cp1250_polish_ci": 99, + //"utf16_unicode_ci": 101, + //"utf16_icelandic_ci": 102, + //"utf16_latvian_ci": 103, + //"utf16_romanian_ci": 104, + //"utf16_slovenian_ci": 105, + //"utf16_polish_ci": 106, + //"utf16_estonian_ci": 107, + //"utf16_spanish_ci": 108, + //"utf16_swedish_ci": 109, + //"utf16_turkish_ci": 110, + //"utf16_czech_ci": 111, + //"utf16_danish_ci": 112, + //"utf16_lithuanian_ci": 113, + //"utf16_slovak_ci": 114, + //"utf16_spanish2_ci": 115, + //"utf16_roman_ci": 116, + //"utf16_persian_ci": 117, + //"utf16_esperanto_ci": 118, + //"utf16_hungarian_ci": 119, + //"utf16_sinhala_ci": 120, + //"utf16_german2_ci": 121, + //"utf16_croatian_ci": 122, + //"utf16_unicode_520_ci": 123, + //"utf16_vietnamese_ci": 124, + //"ucs2_unicode_ci": 128, + //"ucs2_icelandic_ci": 129, + //"ucs2_latvian_ci": 130, + //"ucs2_romanian_ci": 131, + //"ucs2_slovenian_ci": 132, + //"ucs2_polish_ci": 133, + //"ucs2_estonian_ci": 134, + //"ucs2_spanish_ci": 135, + //"ucs2_swedish_ci": 136, + //"ucs2_turkish_ci": 137, + //"ucs2_czech_ci": 138, + //"ucs2_danish_ci": 139, + //"ucs2_lithuanian_ci": 140, + //"ucs2_slovak_ci": 141, + //"ucs2_spanish2_ci": 142, + //"ucs2_roman_ci": 143, + //"ucs2_persian_ci": 144, + //"ucs2_esperanto_ci": 145, + //"ucs2_hungarian_ci": 146, + //"ucs2_sinhala_ci": 147, + //"ucs2_german2_ci": 148, + //"ucs2_croatian_ci": 149, + //"ucs2_unicode_520_ci": 150, + //"ucs2_vietnamese_ci": 151, + //"ucs2_general_mysql500_ci": 159, + //"utf32_unicode_ci": 160, + //"utf32_icelandic_ci": 161, + //"utf32_latvian_ci": 162, + //"utf32_romanian_ci": 163, + //"utf32_slovenian_ci": 164, + //"utf32_polish_ci": 165, + //"utf32_estonian_ci": 166, + //"utf32_spanish_ci": 167, + //"utf32_swedish_ci": 168, + //"utf32_turkish_ci": 169, + //"utf32_czech_ci": 170, + //"utf32_danish_ci": 171, + //"utf32_lithuanian_ci": 172, + //"utf32_slovak_ci": 173, + //"utf32_spanish2_ci": 174, + //"utf32_roman_ci": 175, + //"utf32_persian_ci": 176, + //"utf32_esperanto_ci": 177, + //"utf32_hungarian_ci": 178, + //"utf32_sinhala_ci": 179, + //"utf32_german2_ci": 180, + //"utf32_croatian_ci": 181, + //"utf32_unicode_520_ci": 182, + //"utf32_vietnamese_ci": 183, + "utf8_unicode_ci": 192, + "utf8_icelandic_ci": 193, + "utf8_latvian_ci": 194, + "utf8_romanian_ci": 195, + "utf8_slovenian_ci": 196, + "utf8_polish_ci": 197, + "utf8_estonian_ci": 198, + "utf8_spanish_ci": 199, + "utf8_swedish_ci": 200, + "utf8_turkish_ci": 201, + "utf8_czech_ci": 202, + "utf8_danish_ci": 203, + "utf8_lithuanian_ci": 204, + "utf8_slovak_ci": 205, + "utf8_spanish2_ci": 206, + "utf8_roman_ci": 207, + "utf8_persian_ci": 208, + "utf8_esperanto_ci": 209, + "utf8_hungarian_ci": 210, + "utf8_sinhala_ci": 211, + "utf8_german2_ci": 212, + "utf8_croatian_ci": 213, + "utf8_unicode_520_ci": 214, + "utf8_vietnamese_ci": 215, + "utf8_general_mysql500_ci": 223, + "utf8mb4_unicode_ci": 224, + "utf8mb4_icelandic_ci": 225, + "utf8mb4_latvian_ci": 226, + "utf8mb4_romanian_ci": 227, + "utf8mb4_slovenian_ci": 228, + "utf8mb4_polish_ci": 229, + "utf8mb4_estonian_ci": 230, + "utf8mb4_spanish_ci": 231, + "utf8mb4_swedish_ci": 232, + "utf8mb4_turkish_ci": 233, + "utf8mb4_czech_ci": 234, + "utf8mb4_danish_ci": 235, + "utf8mb4_lithuanian_ci": 236, + "utf8mb4_slovak_ci": 237, + "utf8mb4_spanish2_ci": 238, + "utf8mb4_roman_ci": 239, + "utf8mb4_persian_ci": 240, + "utf8mb4_esperanto_ci": 241, + "utf8mb4_hungarian_ci": 242, + "utf8mb4_sinhala_ci": 243, + "utf8mb4_german2_ci": 244, + "utf8mb4_croatian_ci": 245, + "utf8mb4_unicode_520_ci": 246, + "utf8mb4_vietnamese_ci": 247, + "gb18030_chinese_ci": 248, + "gb18030_bin": 249, + "gb18030_unicode_520_ci": 250, + "utf8mb4_0900_ai_ci": 255, +} + +// A denylist of collations which is unsafe to interpolate parameters. +// These multibyte encodings may contains 0x5c (`\`) in their trailing bytes. +var unsafeCollations = map[string]bool{ + "big5_chinese_ci": true, + "sjis_japanese_ci": true, + "gbk_chinese_ci": true, + "big5_bin": true, + "gb2312_bin": true, + "gbk_bin": true, + "sjis_bin": true, + "cp932_japanese_ci": true, + "cp932_bin": true, + "gb18030_chinese_ci": true, + "gb18030_bin": true, + "gb18030_unicode_520_ci": true, +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/compress.go b/_LICENSES/github.com/go-sql-driver/mysql/compress.go new file mode 100644 index 0000000000..38bfa000e8 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/compress.go @@ -0,0 +1,213 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2024 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "compress/zlib" + "fmt" + "io" + "sync" +) + +var ( + zrPool *sync.Pool // Do not use directly. Use zDecompress() instead. + zwPool *sync.Pool // Do not use directly. Use zCompress() instead. +) + +func init() { + zrPool = &sync.Pool{ + New: func() any { return nil }, + } + zwPool = &sync.Pool{ + New: func() any { + zw, err := zlib.NewWriterLevel(new(bytes.Buffer), 2) + if err != nil { + panic(err) // compress/zlib return non-nil error only if level is invalid + } + return zw + }, + } +} + +func zDecompress(src []byte, dst *bytes.Buffer) (int, error) { + br := bytes.NewReader(src) + var zr io.ReadCloser + var err error + + if a := zrPool.Get(); a == nil { + if zr, err = zlib.NewReader(br); err != nil { + return 0, err + } + } else { + zr = a.(io.ReadCloser) + if err := zr.(zlib.Resetter).Reset(br, nil); err != nil { + return 0, err + } + } + + n, _ := dst.ReadFrom(zr) // ignore err because zr.Close() will return it again. + err = zr.Close() // zr.Close() may return chuecksum error. + zrPool.Put(zr) + return int(n), err +} + +func zCompress(src []byte, dst io.Writer) error { + zw := zwPool.Get().(*zlib.Writer) + zw.Reset(dst) + if _, err := zw.Write(src); err != nil { + return err + } + err := zw.Close() + zwPool.Put(zw) + return err +} + +type compIO struct { + mc *mysqlConn + buff bytes.Buffer +} + +func newCompIO(mc *mysqlConn) *compIO { + return &compIO{ + mc: mc, + } +} + +func (c *compIO) reset() { + c.buff.Reset() +} + +func (c *compIO) readNext(need int) ([]byte, error) { + for c.buff.Len() < need { + if err := c.readCompressedPacket(); err != nil { + return nil, err + } + } + data := c.buff.Next(need) + return data[:need:need], nil // prevent caller writes into c.buff +} + +func (c *compIO) readCompressedPacket() error { + header, err := c.mc.readNext(7) + if err != nil { + return err + } + _ = header[6] // bounds check hint to compiler; guaranteed by readNext + + // compressed header structure + comprLength := getUint24(header[0:3]) + compressionSequence := header[3] + uncompressedLength := getUint24(header[4:7]) + if debug { + fmt.Printf("uncompress cmplen=%v uncomplen=%v pkt_cmp_seq=%v expected_cmp_seq=%v\n", + comprLength, uncompressedLength, compressionSequence, c.mc.sequence) + } + // Do not return ErrPktSync here. + // Server may return error packet (e.g. 1153 Got a packet bigger than 'max_allowed_packet' bytes) + // before receiving all packets from client. In this case, seqnr is younger than expected. + // NOTE: Both of mariadbclient and mysqlclient do not check seqnr. Only server checks it. + if debug && compressionSequence != c.mc.compressSequence { + fmt.Printf("WARN: unexpected cmpress seq nr: expected %v, got %v", + c.mc.compressSequence, compressionSequence) + } + c.mc.compressSequence = compressionSequence + 1 + + comprData, err := c.mc.readNext(comprLength) + if err != nil { + return err + } + + // if payload is uncompressed, its length will be specified as zero, and its + // true length is contained in comprLength + if uncompressedLength == 0 { + c.buff.Write(comprData) + return nil + } + + // use existing capacity in bytesBuf if possible + c.buff.Grow(uncompressedLength) + nread, err := zDecompress(comprData, &c.buff) + if err != nil { + return err + } + if nread != uncompressedLength { + return fmt.Errorf("invalid compressed packet: uncompressed length in header is %d, actual %d", + uncompressedLength, nread) + } + return nil +} + +const minCompressLength = 150 +const maxPayloadLen = maxPacketSize - 4 + +// writePackets sends one or some packets with compression. +// Use this instead of mc.netConn.Write() when mc.compress is true. +func (c *compIO) writePackets(packets []byte) (int, error) { + totalBytes := len(packets) + blankHeader := make([]byte, 7) + buf := &c.buff + + for len(packets) > 0 { + payloadLen := min(maxPayloadLen, len(packets)) + payload := packets[:payloadLen] + uncompressedLen := payloadLen + + buf.Reset() + buf.Write(blankHeader) // Buffer.Write() never returns error + + // If payload is less than minCompressLength, don't compress. + if uncompressedLen < minCompressLength { + buf.Write(payload) + uncompressedLen = 0 + } else { + err := zCompress(payload, buf) + if debug && err != nil { + fmt.Printf("zCompress error: %v", err) + } + // do not compress if compressed data is larger than uncompressed data + // I intentionally miss 7 byte header in the buf; zCompress must compress more than 7 bytes. + if err != nil || buf.Len() >= uncompressedLen { + buf.Reset() + buf.Write(blankHeader) + buf.Write(payload) + uncompressedLen = 0 + } + } + + if n, err := c.writeCompressedPacket(buf.Bytes(), uncompressedLen); err != nil { + // To allow returning ErrBadConn when sending really 0 bytes, we sum + // up compressed bytes that is returned by underlying Write(). + return totalBytes - len(packets) + n, err + } + packets = packets[payloadLen:] + } + + return totalBytes, nil +} + +// writeCompressedPacket writes a compressed packet with header. +// data should start with 7 size space for header followed by payload. +func (c *compIO) writeCompressedPacket(data []byte, uncompressedLen int) (int, error) { + mc := c.mc + comprLength := len(data) - 7 + if debug { + fmt.Printf( + "writeCompressedPacket: comprLength=%v, uncompressedLen=%v, seq=%v\n", + comprLength, uncompressedLen, mc.compressSequence) + } + + // compression header + putUint24(data[0:3], comprLength) + data[3] = mc.compressSequence + putUint24(data[4:7], uncompressedLen) + + mc.compressSequence++ + return mc.writeWithTimeout(data) +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/compress_test.go b/_LICENSES/github.com/go-sql-driver/mysql/compress_test.go new file mode 100644 index 0000000000..030deaefa0 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/compress_test.go @@ -0,0 +1,119 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2024 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "crypto/rand" + "io" + "testing" +) + +func makeRandByteSlice(size int) []byte { + randBytes := make([]byte, size) + rand.Read(randBytes) + return randBytes +} + +// compressHelper compresses uncompressedPacket and checks state variables +func compressHelper(t *testing.T, mc *mysqlConn, uncompressedPacket []byte) []byte { + conn := new(mockConn) + mc.netConn = conn + + err := mc.writePacket(append(make([]byte, 4), uncompressedPacket...)) + if err != nil { + t.Fatal(err) + } + + return conn.written +} + +// uncompressHelper uncompresses compressedPacket and checks state variables +func uncompressHelper(t *testing.T, mc *mysqlConn, compressedPacket []byte) []byte { + // mocking out buf variable + conn := new(mockConn) + conn.data = compressedPacket + mc.netConn = conn + + uncompressedPacket, err := mc.readPacket() + if err != nil { + if err != io.EOF { + t.Fatalf("non-nil/non-EOF error when reading contents: %s", err.Error()) + } + } + return uncompressedPacket +} + +// roundtripHelper compresses then uncompresses uncompressedPacket and checks state variables +func roundtripHelper(t *testing.T, cSend *mysqlConn, cReceive *mysqlConn, uncompressedPacket []byte) []byte { + compressed := compressHelper(t, cSend, uncompressedPacket) + return uncompressHelper(t, cReceive, compressed) +} + +// TestRoundtrip tests two connections, where one is reading and the other is writing +func TestRoundtrip(t *testing.T) { + tests := []struct { + uncompressed []byte + desc string + }{ + {uncompressed: []byte("a"), + desc: "a"}, + {uncompressed: []byte("hello world"), + desc: "hello world"}, + {uncompressed: make([]byte, 100), + desc: "100 bytes"}, + {uncompressed: make([]byte, 32768), + desc: "32768 bytes"}, + {uncompressed: make([]byte, 330000), + desc: "33000 bytes"}, + {uncompressed: makeRandByteSlice(10), + desc: "10 rand bytes", + }, + {uncompressed: makeRandByteSlice(100), + desc: "100 rand bytes", + }, + {uncompressed: makeRandByteSlice(32768), + desc: "32768 rand bytes", + }, + {uncompressed: bytes.Repeat(makeRandByteSlice(100), 10000), + desc: "100 rand * 10000 repeat bytes", + }, + } + + _, cSend := newRWMockConn(0) + cSend.compress = true + cSend.compIO = newCompIO(cSend) + _, cReceive := newRWMockConn(0) + cReceive.compress = true + cReceive.compIO = newCompIO(cReceive) + + for _, test := range tests { + t.Run(test.desc, func(t *testing.T) { + cSend.resetSequence() + cReceive.resetSequence() + + uncompressed := roundtripHelper(t, cSend, cReceive, test.uncompressed) + if len(uncompressed) != len(test.uncompressed) { + t.Errorf("uncompressed size is unexpected. expected %d but got %d", + len(test.uncompressed), len(uncompressed)) + } + if !bytes.Equal(uncompressed, test.uncompressed) { + t.Errorf("roundtrip failed") + } + if cSend.sequence != cReceive.sequence { + t.Errorf("inconsistent sequence number: send=%v recv=%v", + cSend.sequence, cReceive.sequence) + } + if cSend.compressSequence != cReceive.compressSequence { + t.Errorf("inconsistent compress sequence number: send=%v recv=%v", + cSend.compressSequence, cReceive.compressSequence) + } + }) + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/conncheck.go b/_LICENSES/github.com/go-sql-driver/mysql/conncheck.go new file mode 100644 index 0000000000..f9c5cb65cb --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/conncheck.go @@ -0,0 +1,54 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2019 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +//go:build linux || darwin || dragonfly || freebsd || netbsd || openbsd || solaris || illumos + +package mysql + +import ( + "errors" + "io" + "net" + "syscall" +) + +var errUnexpectedRead = errors.New("unexpected read from socket") + +func connCheck(conn net.Conn) error { + var sysErr error + + sysConn, ok := conn.(syscall.Conn) + if !ok { + return nil + } + rawConn, err := sysConn.SyscallConn() + if err != nil { + return err + } + + err = rawConn.Read(func(fd uintptr) bool { + var buf [1]byte + n, err := syscall.Read(int(fd), buf[:]) + switch { + case n == 0 && err == nil: + sysErr = io.EOF + case n > 0: + sysErr = errUnexpectedRead + case err == syscall.EAGAIN || err == syscall.EWOULDBLOCK: + sysErr = nil + default: + sysErr = err + } + return true + }) + if err != nil { + return err + } + + return sysErr +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/conncheck_dummy.go b/_LICENSES/github.com/go-sql-driver/mysql/conncheck_dummy.go new file mode 100644 index 0000000000..0ebf05c213 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/conncheck_dummy.go @@ -0,0 +1,17 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2019 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +//go:build !linux && !darwin && !dragonfly && !freebsd && !netbsd && !openbsd && !solaris && !illumos + +package mysql + +import "net" + +func connCheck(conn net.Conn) error { + return nil +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/conncheck_test.go b/_LICENSES/github.com/go-sql-driver/mysql/conncheck_test.go new file mode 100644 index 0000000000..1c38fad041 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/conncheck_test.go @@ -0,0 +1,38 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +//go:build linux || darwin || dragonfly || freebsd || netbsd || openbsd || solaris || illumos + +package mysql + +import ( + "testing" + "time" +) + +func TestStaleConnectionChecks(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + dbt.mustExec("SET @@SESSION.wait_timeout = 2") + + if err := dbt.db.Ping(); err != nil { + dbt.Fatal(err) + } + + // wait for MySQL to close our connection + time.Sleep(3 * time.Second) + + tx, err := dbt.db.Begin() + if err != nil { + dbt.Fatal(err) + } + + if err := tx.Rollback(); err != nil { + dbt.Fatal(err) + } + }) +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/connection.go b/_LICENSES/github.com/go-sql-driver/mysql/connection.go new file mode 100644 index 0000000000..5a25c878ac --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/connection.go @@ -0,0 +1,818 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "context" + "database/sql" + "database/sql/driver" + "encoding/json" + "fmt" + "io" + "net" + "runtime" + "strconv" + "strings" + "sync/atomic" + "time" +) + +type mysqlConn struct { + buf buffer + netConn net.Conn + rawConn net.Conn // underlying connection when netConn is TLS connection. + result mysqlResult // managed by clearResult() and handleOkPacket(). + compIO *compIO + cfg *Config + connector *connector + maxAllowedPacket int + maxWriteSize int + capabilities capabilityFlag + extCapabilities extendedCapabilityFlag + status statusFlag + sequence uint8 + compressSequence uint8 + parseTime bool + compress bool + + // for context support (Go 1.8+) + watching bool + watcher chan<- context.Context + closech chan struct{} + finished chan<- struct{} + canceled atomicError // set non-nil if conn is canceled + closed atomic.Bool // set when conn is closed, before closech is closed +} + +// Helper function to call per-connection logger. +func (mc *mysqlConn) log(v ...any) { + _, filename, lineno, ok := runtime.Caller(1) + if ok { + pos := strings.LastIndexByte(filename, '/') + if pos != -1 { + filename = filename[pos+1:] + } + prefix := fmt.Sprintf("%s:%d ", filename, lineno) + v = append([]any{prefix}, v...) + } + + mc.cfg.Logger.Print(v...) +} + +func (mc *mysqlConn) readWithTimeout(b []byte) (int, error) { + to := mc.cfg.ReadTimeout + if to > 0 { + if err := mc.netConn.SetReadDeadline(time.Now().Add(to)); err != nil { + return 0, err + } + } + return mc.netConn.Read(b) +} + +func (mc *mysqlConn) writeWithTimeout(b []byte) (int, error) { + to := mc.cfg.WriteTimeout + if to > 0 { + if err := mc.netConn.SetWriteDeadline(time.Now().Add(to)); err != nil { + return 0, err + } + } + return mc.netConn.Write(b) +} + +func (mc *mysqlConn) resetSequence() { + mc.sequence = 0 + mc.compressSequence = 0 +} + +// syncSequence must be called when finished writing some packet and before start reading. +func (mc *mysqlConn) syncSequence() { + // Syncs compressionSequence to sequence. + // This is not documented but done in `net_flush()` in MySQL and MariaDB. + // https://github.com/mariadb-corporation/mariadb-connector-c/blob/8228164f850b12353da24df1b93a1e53cc5e85e9/libmariadb/ma_net.c#L170-L171 + // https://github.com/mysql/mysql-server/blob/824e2b4064053f7daf17d7f3f84b7a3ed92e5fb4/sql-common/net_serv.cc#L293 + if mc.compress { + mc.sequence = mc.compressSequence + mc.compIO.reset() + } +} + +// Handles parameters set in DSN after the connection is established +func (mc *mysqlConn) handleParams() (err error) { + var cmdSet strings.Builder + + for param, val := range mc.cfg.Params { + if cmdSet.Len() == 0 { + // Heuristic: 29 chars for each other key=value to reduce reallocations + cmdSet.Grow(4 + len(param) + 3 + len(val) + 30*(len(mc.cfg.Params)-1)) + cmdSet.WriteString("SET ") + } else { + cmdSet.WriteString(", ") + } + cmdSet.WriteString(param) + cmdSet.WriteString(" = ") + cmdSet.WriteString(val) + } + + if cmdSet.Len() > 0 { + err = mc.exec(cmdSet.String()) + } + + return +} + +// markBadConn replaces errBadConnNoWrite with driver.ErrBadConn. +// This function is used to return driver.ErrBadConn only when safe to retry. +func (mc *mysqlConn) markBadConn(err error) error { + if err == errBadConnNoWrite { + return driver.ErrBadConn + } + return err +} + +func (mc *mysqlConn) Begin() (driver.Tx, error) { + return mc.begin(false) +} + +func (mc *mysqlConn) begin(readOnly bool) (driver.Tx, error) { + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + var q string + if readOnly { + q = "START TRANSACTION READ ONLY" + } else { + q = "START TRANSACTION" + } + err := mc.exec(q) + if err == nil { + return &mysqlTx{mc}, err + } + return nil, mc.markBadConn(err) +} + +func (mc *mysqlConn) Close() (err error) { + // Makes Close idempotent + if !mc.closed.Load() { + err = mc.writeCommandPacket(comQuit) + } + mc.close() + return +} + +// close closes the network connection and clear results without sending COM_QUIT. +func (mc *mysqlConn) close() { + mc.cleanup() + mc.clearResult() +} + +// Closes the network connection and unsets internal variables. Do not call this +// function after successful authentication, call Close instead. This function +// is called before auth or on auth failure because MySQL will have already +// closed the network connection. +func (mc *mysqlConn) cleanup() { + if mc.closed.Swap(true) { + return + } + + // Makes cleanup idempotent + close(mc.closech) + conn := mc.rawConn + if conn == nil { + return + } + if err := conn.Close(); err != nil { + mc.log("closing connection:", err) + } + // This function can be called from multiple goroutines. + // So we can not mc.clearResult() here. + // Caller should do it if they are in safe goroutine. +} + +func (mc *mysqlConn) error() error { + if mc.closed.Load() { + if err := mc.canceled.Value(); err != nil { + return err + } + return ErrInvalidConn + } + return nil +} + +func (mc *mysqlConn) Prepare(query string) (driver.Stmt, error) { + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + // Send command + err := mc.writeCommandPacketStr(comStmtPrepare, query) + if err != nil { + // STMT_PREPARE is safe to retry. So we can return ErrBadConn here. + mc.log(err) + return nil, driver.ErrBadConn + } + + stmt := &mysqlStmt{ + mc: mc, + } + + // Read Result + columnCount, err := stmt.readPrepareResultPacket() + if err == nil { + if stmt.paramCount > 0 { + if err = mc.skipColumns(stmt.paramCount); err != nil { + return nil, err + } + } + + if columnCount > 0 { + if mc.extCapabilities&clientCacheMetadata != 0 { + if stmt.columns, err = mc.readColumns(int(columnCount), nil); err != nil { + return nil, err + } + } else { + if err = mc.skipColumns(int(columnCount)); err != nil { + return nil, err + } + } + } + } + + return stmt, err +} + +func (mc *mysqlConn) interpolateParams(query string, args []driver.Value) (string, error) { + noBackslashEscapes := (mc.status & statusNoBackslashEscapes) != 0 + const ( + stateNormal = iota + stateString + stateEscape + stateEOLComment + stateSlashStarComment + stateBacktick + ) + + const ( + QUOTE_BYTE = byte('\'') + DBL_QUOTE_BYTE = byte('"') + BACKSLASH_BYTE = byte('\\') + QUESTION_MARK_BYTE = byte('?') + SLASH_BYTE = byte('/') + STAR_BYTE = byte('*') + HASH_BYTE = byte('#') + MINUS_BYTE = byte('-') + LINE_FEED_BYTE = byte('\n') + BACKTICK_BYTE = byte('`') + ) + + buf, err := mc.buf.takeCompleteBuffer() + if err != nil { + mc.cleanup() + return "", driver.ErrBadConn + } + buf = buf[:0] + state := stateNormal + singleQuotes := false + lastChar := byte(0) + argPos := 0 + lenQuery := len(query) + lastIdx := 0 + + for i := range lenQuery { + currentChar := query[i] + if state == stateEscape && !((currentChar == QUOTE_BYTE && singleQuotes) || (currentChar == DBL_QUOTE_BYTE && !singleQuotes)) { + state = stateString + lastChar = currentChar + continue + } + switch currentChar { + case STAR_BYTE: + if state == stateNormal && lastChar == SLASH_BYTE { + state = stateSlashStarComment + } + case SLASH_BYTE: + if state == stateSlashStarComment && lastChar == STAR_BYTE { + state = stateNormal + // Clear lastChar so the '/' that closed the comment isn't + // reused to start a new comment with a following '*'. + lastChar = 0 + continue + } + case HASH_BYTE: + if state == stateNormal { + state = stateEOLComment + } + case MINUS_BYTE: + if state == stateNormal && lastChar == MINUS_BYTE { + // -- only starts a comment if followed by whitespace or control char + if i+1 < lenQuery { + nextChar := query[i+1] + if nextChar == ' ' || nextChar == '\t' || nextChar == '\n' || nextChar == '\r' { + state = stateEOLComment + } + } else { + state = stateEOLComment + } + } + case LINE_FEED_BYTE: + if state == stateEOLComment { + state = stateNormal + } + case DBL_QUOTE_BYTE: + if state == stateNormal { + state = stateString + singleQuotes = false + } else if state == stateString && !singleQuotes { + state = stateNormal + } else if state == stateEscape { + state = stateString + } + case QUOTE_BYTE: + if state == stateNormal { + state = stateString + singleQuotes = true + } else if state == stateString && singleQuotes { + state = stateNormal + } else if state == stateEscape { + state = stateString + } + case BACKSLASH_BYTE: + if state == stateString && !noBackslashEscapes { + state = stateEscape + } + case QUESTION_MARK_BYTE: + if state == stateNormal { + if argPos >= len(args) { + return "", driver.ErrSkip + } + buf = append(buf, query[lastIdx:i]...) + arg := args[argPos] + argPos++ + + if arg == nil { + buf = append(buf, "NULL"...) + lastIdx = i + 1 + break + } + + switch v := arg.(type) { + case int64: + buf = strconv.AppendInt(buf, v, 10) + case uint64: + buf = strconv.AppendUint(buf, v, 10) + case float64: + buf = strconv.AppendFloat(buf, v, 'g', -1, 64) + case bool: + if v { + buf = append(buf, '1') + } else { + buf = append(buf, '0') + } + case time.Time: + if v.IsZero() { + buf = append(buf, "'0000-00-00'"...) + } else { + buf = append(buf, '\'') + buf, err = appendDateTime(buf, v.In(mc.cfg.Loc), mc.cfg.timeTruncate) + if err != nil { + return "", err + } + buf = append(buf, '\'') + } + case json.RawMessage: + if v == nil { + buf = append(buf, "NULL"...) + } else if noBackslashEscapes { + buf = escapeBytesQuotes(buf, v, false) + } else { + buf = escapeBytesBackslash(buf, v, false) + } + case []byte: + if v == nil { + buf = append(buf, "NULL"...) + } else { + if noBackslashEscapes { + buf = escapeBytesQuotes(buf, v, true) + } else { + buf = escapeBytesBackslash(buf, v, true) + } + } + case string: + if noBackslashEscapes { + buf = escapeStringQuotes(buf, v) + } else { + buf = escapeStringBackslash(buf, v) + } + default: + return "", driver.ErrSkip + } + + if len(buf)+4 > mc.maxAllowedPacket { + return "", driver.ErrSkip + } + lastIdx = i + 1 + } + case BACKTICK_BYTE: + if state == stateBacktick { + state = stateNormal + } else if state == stateNormal { + state = stateBacktick + } + } + lastChar = currentChar + } + buf = append(buf, query[lastIdx:]...) + if argPos != len(args) { + return "", driver.ErrSkip + } + return string(buf), nil +} + +func (mc *mysqlConn) Exec(query string, args []driver.Value) (driver.Result, error) { + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + if len(args) != 0 { + if !mc.cfg.InterpolateParams { + return nil, driver.ErrSkip + } + // try to interpolate the parameters to save extra roundtrips for preparing and closing a statement + prepared, err := mc.interpolateParams(query, args) + if err != nil { + return nil, err + } + query = prepared + } + + err := mc.exec(query) + if err == nil { + copied := mc.result + return &copied, err + } + return nil, mc.markBadConn(err) +} + +// Internal function to execute commands +func (mc *mysqlConn) exec(query string) error { + handleOk := mc.clearResult() + // Send command + if err := mc.writeCommandPacketStr(comQuery, query); err != nil { + return mc.markBadConn(err) + } + + // Read Result + resLen, _, err := handleOk.readResultSetHeaderPacket() + if err != nil { + return err + } + + if resLen > 0 { + // columns + if err := mc.skipColumns(resLen); err != nil { + return err + } + + // rows + if err := mc.skipRows(); err != nil { + return err + } + } + + return handleOk.discardResults() +} + +func (mc *mysqlConn) Query(query string, args []driver.Value) (driver.Rows, error) { + return mc.query(query, args) +} + +func (mc *mysqlConn) query(query string, args []driver.Value) (*textRows, error) { + handleOk := mc.clearResult() + + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + if len(args) != 0 { + if !mc.cfg.InterpolateParams { + return nil, driver.ErrSkip + } + // try client-side prepare to reduce roundtrip + prepared, err := mc.interpolateParams(query, args) + if err != nil { + return nil, err + } + query = prepared + } + // Send command + err := mc.writeCommandPacketStr(comQuery, query) + if err != nil { + return nil, mc.markBadConn(err) + } + + // Read Result + var resLen int + resLen, _, err = handleOk.readResultSetHeaderPacket() + if err != nil { + return nil, err + } + + rows := new(textRows) + rows.mc = mc + + if resLen == 0 { + rows.rs.done = true + + switch err := rows.NextResultSet(); err { + case nil, io.EOF: + return rows, nil + default: + return nil, err + } + } + + // Columns + rows.rs.columns, err = mc.readColumns(resLen, nil) + return rows, err +} + +// Gets the value of the given MySQL System Variable +func (mc *mysqlConn) getSystemVar(name string) (string, error) { + // Send command + handleOk := mc.clearResult() + if err := mc.writeCommandPacketStr(comQuery, "SELECT @@"+name); err != nil { + return "", err + } + + // Read Result + resLen, _, err := handleOk.readResultSetHeaderPacket() + if err == nil { + rows := new(textRows) + rows.mc = mc + rows.rs.columns = []mysqlField{{fieldType: fieldTypeVarChar}} + + if resLen > 0 { + // Columns + if err := mc.skipColumns(resLen); err != nil { + return "", err + } + } + + dest := make([]driver.Value, resLen) + if err = rows.readRow(dest); err == nil { + // Convert to string before skipRows, which may + // overwrite the read buffer that dest[0] points into. + val := string(dest[0].([]byte)) + return val, mc.skipRows() + } + } + return "", err +} + +// cancel is called when the query has canceled. +func (mc *mysqlConn) cancel(err error) { + mc.canceled.Set(err) + mc.cleanup() +} + +// finish is called when the query has succeeded. +func (mc *mysqlConn) finish() { + if !mc.watching || mc.finished == nil { + return + } + select { + case mc.finished <- struct{}{}: + mc.watching = false + case <-mc.closech: + } +} + +// Ping implements driver.Pinger interface +func (mc *mysqlConn) Ping(ctx context.Context) (err error) { + if mc.closed.Load() { + return driver.ErrBadConn + } + + if err = mc.watchCancel(ctx); err != nil { + return + } + defer mc.finish() + + handleOk := mc.clearResult() + if err = mc.writeCommandPacket(comPing); err != nil { + return mc.markBadConn(err) + } + + return handleOk.readResultOK() +} + +// BeginTx implements driver.ConnBeginTx interface +func (mc *mysqlConn) BeginTx(ctx context.Context, opts driver.TxOptions) (driver.Tx, error) { + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + + if err := mc.watchCancel(ctx); err != nil { + return nil, err + } + defer mc.finish() + + if sql.IsolationLevel(opts.Isolation) != sql.LevelDefault { + level, err := mapIsolationLevel(opts.Isolation) + if err != nil { + return nil, err + } + err = mc.exec("SET TRANSACTION ISOLATION LEVEL " + level) + if err != nil { + return nil, err + } + } + + return mc.begin(opts.ReadOnly) +} + +func (mc *mysqlConn) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error) { + dargs, err := namedValueToValue(args) + if err != nil { + return nil, err + } + + if err := mc.watchCancel(ctx); err != nil { + return nil, err + } + + rows, err := mc.query(query, dargs) + if err != nil { + mc.finish() + return nil, err + } + rows.finish = mc.finish + return rows, err +} + +func (mc *mysqlConn) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error) { + dargs, err := namedValueToValue(args) + if err != nil { + return nil, err + } + + if err := mc.watchCancel(ctx); err != nil { + return nil, err + } + defer mc.finish() + + return mc.Exec(query, dargs) +} + +func (mc *mysqlConn) PrepareContext(ctx context.Context, query string) (driver.Stmt, error) { + if err := mc.watchCancel(ctx); err != nil { + return nil, err + } + + stmt, err := mc.Prepare(query) + mc.finish() + if err != nil { + return nil, err + } + + select { + default: + case <-ctx.Done(): + stmt.Close() + return nil, ctx.Err() + } + return stmt, nil +} + +func (stmt *mysqlStmt) QueryContext(ctx context.Context, args []driver.NamedValue) (driver.Rows, error) { + dargs, err := namedValueToValue(args) + if err != nil { + return nil, err + } + + if err := stmt.mc.watchCancel(ctx); err != nil { + return nil, err + } + + rows, err := stmt.query(dargs) + if err != nil { + stmt.mc.finish() + return nil, err + } + rows.finish = stmt.mc.finish + return rows, err +} + +func (stmt *mysqlStmt) ExecContext(ctx context.Context, args []driver.NamedValue) (driver.Result, error) { + dargs, err := namedValueToValue(args) + if err != nil { + return nil, err + } + + if err := stmt.mc.watchCancel(ctx); err != nil { + return nil, err + } + defer stmt.mc.finish() + + return stmt.Exec(dargs) +} + +func (mc *mysqlConn) watchCancel(ctx context.Context) error { + if mc.watching { + // Reach here if canceled, + // so the connection is already invalid + mc.cleanup() + return nil + } + // When ctx is already cancelled, don't watch it. + if err := ctx.Err(); err != nil { + return err + } + // When ctx is not cancellable, don't watch it. + if ctx.Done() == nil { + return nil + } + // When watcher is not alive, can't watch it. + if mc.watcher == nil { + return nil + } + + mc.watching = true + mc.watcher <- ctx + return nil +} + +func (mc *mysqlConn) startWatcher() { + watcher := make(chan context.Context, 1) + mc.watcher = watcher + finished := make(chan struct{}) + mc.finished = finished + go func() { + for { + var ctx context.Context + select { + case ctx = <-watcher: + case <-mc.closech: + return + } + + select { + case <-ctx.Done(): + mc.cancel(ctx.Err()) + case <-finished: + case <-mc.closech: + return + } + } + }() +} + +func (mc *mysqlConn) CheckNamedValue(nv *driver.NamedValue) (err error) { + nv.Value, err = converter{}.ConvertValue(nv.Value) + return +} + +// ResetSession implements driver.SessionResetter. +// (From Go 1.10) +func (mc *mysqlConn) ResetSession(ctx context.Context) error { + if mc.closed.Load() || mc.buf.busy() { + return driver.ErrBadConn + } + + // Perform a stale connection check. We only perform this check for + // the first query on a connection that has been checked out of the + // connection pool: a fresh connection from the pool is more likely + // to be stale, and it has not performed any previous writes that + // could cause data corruption, so it's safe to return ErrBadConn + // if the check fails. + if mc.cfg.CheckConnLiveness { + conn := mc.netConn + if mc.rawConn != nil { + conn = mc.rawConn + } + var err error + if mc.cfg.ReadTimeout != 0 { + err = conn.SetReadDeadline(time.Now().Add(mc.cfg.ReadTimeout)) + } + if err == nil { + err = connCheck(conn) + } + if err != nil { + mc.log("closing bad idle connection: ", err) + return driver.ErrBadConn + } + } + + return nil +} + +// IsValid implements driver.Validator interface +// (From Go 1.15) +func (mc *mysqlConn) IsValid() bool { + return !mc.closed.Load() && !mc.buf.busy() +} + +var _ driver.SessionResetter = &mysqlConn{} +var _ driver.Validator = &mysqlConn{} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/connection_test.go b/_LICENSES/github.com/go-sql-driver/mysql/connection_test.go new file mode 100644 index 0000000000..1f848641ea --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/connection_test.go @@ -0,0 +1,384 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2016 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "context" + "database/sql/driver" + "encoding/json" + "errors" + "net" + "testing" + "time" +) + +func TestInterpolateParams(t *testing.T) { + mc := &mysqlConn{ + buf: newBuffer(), + maxAllowedPacket: maxPacketSize, + cfg: &Config{ + InterpolateParams: true, + }, + } + + q, err := mc.interpolateParams("SELECT ?+?", []driver.Value{int64(42), "gopher"}) + if err != nil { + t.Errorf("Expected err=nil, got %#v", err) + return + } + expected := `SELECT 42+'gopher'` + if q != expected { + t.Errorf("Expected: %q\nGot: %q", expected, q) + } +} + +func TestInterpolateParamsJSONRawMessage(t *testing.T) { + mc := &mysqlConn{ + buf: newBuffer(), + maxAllowedPacket: maxPacketSize, + cfg: &Config{ + InterpolateParams: true, + }, + } + + buf, err := json.Marshal(struct { + Value int `json:"value"` + }{Value: 42}) + if err != nil { + t.Errorf("Expected err=nil, got %#v", err) + return + } + q, err := mc.interpolateParams("SELECT ?", []driver.Value{json.RawMessage(buf)}) + if err != nil { + t.Errorf("Expected err=nil, got %#v", err) + return + } + expected := `SELECT '{\"value\":42}'` + if q != expected { + t.Errorf("Expected: %q\nGot: %q", expected, q) + } +} + +func TestInterpolateParamsJSONRawMessageNil(t *testing.T) { + mc := &mysqlConn{ + buf: newBuffer(), + maxAllowedPacket: maxPacketSize, + cfg: &Config{ + InterpolateParams: true, + }, + } + + // A typed nil json.RawMessage must be interpolated as NULL, not as an + // empty quoted string. See https://github.com/go-sql-driver/mysql/issues/1781 + q, err := mc.interpolateParams("SELECT ?", []driver.Value{json.RawMessage(nil)}) + if err != nil { + t.Errorf("Expected err=nil, got %#v", err) + return + } + expected := `SELECT NULL` + if q != expected { + t.Errorf("Expected: %q\nGot: %q", expected, q) + } +} + +func TestInterpolateParamsTooManyPlaceholders(t *testing.T) { + mc := &mysqlConn{ + buf: newBuffer(), + maxAllowedPacket: maxPacketSize, + cfg: &Config{ + InterpolateParams: true, + }, + } + + q, err := mc.interpolateParams("SELECT ?+?", []driver.Value{int64(42)}) + if err != driver.ErrSkip { + t.Errorf("Expected err=driver.ErrSkip, got err=%#v, q=%#v", err, q) + } +} + +func TestInterpolateParamsUint64(t *testing.T) { + mc := &mysqlConn{ + buf: newBuffer(), + maxAllowedPacket: maxPacketSize, + cfg: &Config{ + InterpolateParams: true, + }, + } + + q, err := mc.interpolateParams("SELECT ?", []driver.Value{uint64(42)}) + if err != nil { + t.Errorf("Expected err=nil, got err=%#v, q=%#v", err, q) + } + if q != "SELECT 42" { + t.Errorf("Expected uint64 interpolation to work, got q=%#v", q) + } +} + +func TestCheckNamedValue(t *testing.T) { + value := driver.NamedValue{Value: ^uint64(0)} + mc := &mysqlConn{} + err := mc.CheckNamedValue(&value) + + if err != nil { + t.Fatal("uint64 high-bit not convertible", err) + } + + if value.Value != ^uint64(0) { + t.Fatalf("uint64 high-bit converted, got %#v %T", value.Value, value.Value) + } +} + +// TestCleanCancel tests passed context is cancelled at start. +// No packet should be sent. Connection should keep current status. +func TestCleanCancel(t *testing.T) { + mc := &mysqlConn{ + closech: make(chan struct{}), + } + mc.startWatcher() + defer mc.cleanup() + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + for range 3 { // Repeat same behavior + err := mc.Ping(ctx) + if err != context.Canceled { + t.Errorf("expected context.Canceled, got %#v", err) + } + + if mc.closed.Load() { + t.Error("expected mc is not closed, closed actually") + } + + if mc.watching { + t.Error("expected watching is false, but true") + } + } +} + +func TestPingMarkBadConnection(t *testing.T) { + nc := badConnection{err: errors.New("boom")} + mc := &mysqlConn{ + netConn: nc, + buf: newBuffer(), + maxAllowedPacket: defaultMaxAllowedPacket, + closech: make(chan struct{}), + cfg: NewConfig(), + } + + err := mc.Ping(context.Background()) + + if err != driver.ErrBadConn { + t.Errorf("expected driver.ErrBadConn, got %#v", err) + } +} + +func TestPingErrInvalidConn(t *testing.T) { + nc := badConnection{err: errors.New("failed to write"), n: 10} + mc := &mysqlConn{ + netConn: nc, + buf: newBuffer(), + maxAllowedPacket: defaultMaxAllowedPacket, + closech: make(chan struct{}), + cfg: NewConfig(), + } + + err := mc.Ping(context.Background()) + + if err != nc.err { + t.Errorf("expected %#v, got %#v", nc.err, err) + } +} + +type badConnection struct { + n int + err error + net.Conn +} + +func (bc badConnection) Write(b []byte) (n int, err error) { + return bc.n, bc.err +} + +func (bc badConnection) Close() error { + return nil +} + +func TestInterpolateParamsWithComments(t *testing.T) { + mc := &mysqlConn{ + buf: newBuffer(), + maxAllowedPacket: maxPacketSize, + cfg: &Config{ + InterpolateParams: true, + }, + } + + tests := []struct { + query string + args []driver.Value + expected string + shouldSkip bool + }{ + // ? in single-line comment (--) should not be replaced + {"SELECT 1 -- ?\n, ?", []driver.Value{int64(42)}, "SELECT 1 -- ?\n, 42", false}, + // ? in single-line comment (#) should not be replaced + {"SELECT 1 # ?\n, ?", []driver.Value{int64(42)}, "SELECT 1 # ?\n, 42", false}, + // ? in multi-line comment should not be replaced + {"SELECT /* ? */ ?", []driver.Value{int64(42)}, "SELECT /* ? */ 42", false}, + // ? in string literal should not be replaced + {"SELECT '?', ?", []driver.Value{int64(42)}, "SELECT '?', 42", false}, + // ? in backtick identifier should not be replaced + {"SELECT `?`, ?", []driver.Value{int64(42)}, "SELECT `?`, 42", false}, + // ? in backslash-escaped string literal should not be replaced + {"SELECT 'C:\\path\\?x.txt', ?", []driver.Value{int64(42)}, "SELECT 'C:\\path\\?x.txt', 42", false}, + // ? in backslash-escaped string literal should not be replaced + {"SELECT '\\'?', col FROM tbl WHERE id = ? AND desc = 'foo\\'bar?'", []driver.Value{int64(42)}, "SELECT '\\'?', col FROM tbl WHERE id = 42 AND desc = 'foo\\'bar?'", false}, + // Multiple comments and real placeholders + {"SELECT ? -- comment ?\n, ? /* ? */ , ? # ?\n, ?", []driver.Value{int64(1), int64(2), int64(3)}, "SELECT 1 -- comment ?\n, 2 /* ? */ , 3 # ?\n, ?", true}, + // 2--1: -- followed by digit is NOT a comment (it's the number 2 minus minus 1) + {"SELECT ?--1", []driver.Value{int64(2)}, "SELECT 2--1", false}, + // /* */*: After closing block comment, */* should NOT start a new comment + {"SELECT /* comment */* ?, ?", []driver.Value{int64(1), int64(2)}, "SELECT /* comment */* 1, 2", false}, + // /* */*: More complex case with actual comment after + {"SELECT /* c1 */*/* c2 */ ?, ?", []driver.Value{int64(1), int64(2)}, "SELECT /* c1 */*/* c2 */ 1, 2", false}, + } + + for i, test := range tests { + + q, err := mc.interpolateParams(test.query, test.args) + if test.shouldSkip { + if err != driver.ErrSkip { + t.Errorf("Test %d: Expected driver.ErrSkip, got err=%#v, q=%#v", i, err, q) + } + continue + } + if err != nil { + t.Errorf("Test %d: Expected err=nil, got %#v", i, err) + continue + } + if q != test.expected { + t.Errorf("Test %d: Expected: %q\nGot: %q", i, test.expected, q) + } + } +} + +// chunkedConn is a net.Conn that serves pre-built data chunks, one per Read +// call. This simulates the behavior seen with TLS connections, where the +// server's TLS library typically produces a separate TLS record per write +// and Go's crypto/tls.Read returns one record at a time. +type chunkedConn struct { + chunks [][]byte + idx int // current chunk index + off int // offset within current chunk +} + +func (c *chunkedConn) Read(b []byte) (int, error) { + if c.idx >= len(c.chunks) { + return 0, errors.New("no more data") + } + n := copy(b, c.chunks[c.idx][c.off:]) + c.off += n + if c.off >= len(c.chunks[c.idx]) { + c.idx++ + c.off = 0 + } + return n, nil +} + +func (c *chunkedConn) Write(b []byte) (int, error) { return len(b), nil } // swallow writes (e.g. COM_QUERY) +func (c *chunkedConn) Close() error { return nil } +func (c *chunkedConn) LocalAddr() net.Addr { return nil } +func (c *chunkedConn) RemoteAddr() net.Addr { return nil } +func (c *chunkedConn) SetDeadline(_ time.Time) error { return nil } +func (c *chunkedConn) SetReadDeadline(_ time.Time) error { return nil } +func (c *chunkedConn) SetWriteDeadline(_ time.Time) error { return nil } + +var _ net.Conn = (*chunkedConn)(nil) + +// makePacket wraps a payload in a MySQL protocol packet header. +func makePacket(seq byte, payload []byte) []byte { + pkt := make([]byte, 4+len(payload)) + pkt[0] = byte(len(payload)) + pkt[1] = byte(len(payload) >> 8) + pkt[2] = byte(len(payload) >> 16) + pkt[3] = seq + copy(pkt[4:], payload) + return pkt +} + +// TestGetSystemVarBufferReuse verifies that getSystemVar returns a value that +// is not corrupted by the subsequent skipRows call. +// +// The row value returned by readRow points into the read buffer. skipRows may +// call fill(), which overwrites that memory. The test feeds each protocol +// packet as a separate Read call via chunkedConn (mimicking TLS record +// boundaries), guaranteeing that fill() is called for the trailing EOF. +func TestGetSystemVarBufferReuse(t *testing.T) { + // Protocol response for: SELECT @@max_allowed_packet → "67108864" + // + // Sequence numbers start at 1 (client sent COM_QUERY as seq 0). + // + // seq 1: column count = 1 + // seq 2: column definition (minimal valid) + // seq 3: EOF (end of column defs) + // seq 4: row data — length-encoded string "67108864" + // seq 5: EOF (end of rows) + + colCountPkt := makePacket(1, []byte{0x01}) + + colDef := []byte{ + 0x03, 'd', 'e', 'f', // catalog = "def" + 0x00, // schema = "" + 0x00, // table = "" + 0x00, // org_table = "" + 0x14, // name length = 20 + '@', '@', 'm', 'a', 'x', '_', 'a', 'l', 'l', 'o', + 'w', 'e', 'd', '_', 'p', 'a', 'c', 'k', 'e', 't', + 0x00, // org_name = "" + 0x0c, // length of fixed fields + 0x3f, 0x00, // charset = 63 (binary) + 0x14, 0x00, 0x00, 0x00, // column_length = 20 + 0x0f, // type = FIELD_TYPE_VARCHAR + 0x00, 0x00, // flags + 0x00, // decimals + 0x00, 0x00, // filler + } + colDefPkt := makePacket(2, colDef) + + eof1 := makePacket(3, []byte{0xfe, 0x00, 0x00, 0x02, 0x00}) + + // Row: length-encoded string "67108864" (8 bytes → length prefix 0x08) + rowPkt := makePacket(4, []byte{0x08, '6', '7', '1', '0', '8', '8', '6', '4'}) + + eof2 := makePacket(5, []byte{0xfe, 0x00, 0x00, 0x02, 0x00}) + + // Each packet arrives in its own Read call, simulating TLS record + // boundaries where each server Write becomes a separate TLS record + // and each client Read returns exactly one record. + conn := &chunkedConn{chunks: [][]byte{colCountPkt, colDefPkt, eof1, rowPkt, eof2}} + + mc := &mysqlConn{ + netConn: conn, + buf: newBuffer(), + cfg: NewConfig(), + closech: make(chan struct{}), + maxAllowedPacket: defaultMaxAllowedPacket, + sequence: 1, // after COM_QUERY (seq 0) + } + + val, err := mc.getSystemVar("max_allowed_packet") + if err != nil { + t.Fatalf("getSystemVar failed: %v", err) + } + + const expected = "67108864" + if val != expected { + t.Fatalf("getSystemVar(max_allowed_packet) = %q, want %q", val, expected) + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/connector.go b/_LICENSES/github.com/go-sql-driver/mysql/connector.go new file mode 100644 index 0000000000..3d37604775 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/connector.go @@ -0,0 +1,229 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2018 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "context" + "database/sql/driver" + "fmt" + "net" + "os" + "strconv" + "strings" +) + +type connector struct { + cfg *Config // immutable private copy. + encodedAttributes string // Encoded connection attributes. +} + +func encodeConnectionAttributes(cfg *Config) string { + connAttrsBuf := make([]byte, 0) + + // default connection attributes + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrClientName) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrClientNameValue) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrOS) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrOSValue) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrPlatform) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrPlatformValue) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrPid) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, strconv.Itoa(os.Getpid())) + serverHost, _, _ := net.SplitHostPort(cfg.Addr) + if serverHost != "" { + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrServerHost) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, serverHost) + } + + // user-defined connection attributes + for connAttr := range strings.SplitSeq(cfg.ConnectionAttributes, ",") { + k, v, found := strings.Cut(connAttr, ":") + if !found { + continue + } + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, k) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, v) + } + + return string(connAttrsBuf) +} + +func newConnector(cfg *Config) *connector { + encodedAttributes := encodeConnectionAttributes(cfg) + return &connector{ + cfg: cfg, + encodedAttributes: encodedAttributes, + } +} + +// Connect implements driver.Connector interface. +// Connect returns a connection to the database. +func (c *connector) Connect(ctx context.Context) (driver.Conn, error) { + var err error + + // Invoke beforeConnect if present, with a copy of the configuration + cfg := c.cfg + if c.cfg.beforeConnect != nil { + cfg = c.cfg.Clone() + err = c.cfg.beforeConnect(ctx, cfg) + if err != nil { + return nil, err + } + } + + // New mysqlConn + mc := &mysqlConn{ + maxAllowedPacket: maxPacketSize, + maxWriteSize: maxPacketSize - 1, + closech: make(chan struct{}), + cfg: cfg, + connector: c, + } + mc.parseTime = mc.cfg.ParseTime + + // Connect to Server + dctx := ctx + if mc.cfg.Timeout > 0 { + var cancel context.CancelFunc + dctx, cancel = context.WithTimeout(ctx, c.cfg.Timeout) + defer cancel() + } + + if c.cfg.DialFunc != nil { + mc.netConn, err = c.cfg.DialFunc(dctx, mc.cfg.Net, mc.cfg.Addr) + } else { + dialsLock.RLock() + dial, ok := dials[mc.cfg.Net] + dialsLock.RUnlock() + if ok { + mc.netConn, err = dial(dctx, mc.cfg.Addr) + } else { + nd := net.Dialer{} + mc.netConn, err = nd.DialContext(dctx, mc.cfg.Net, mc.cfg.Addr) + } + } + if err != nil { + return nil, err + } + mc.rawConn = mc.netConn + + // Enable TCP Keepalives on TCP connections + if tc, ok := mc.netConn.(*net.TCPConn); ok { + if err := tc.SetKeepAlive(true); err != nil { + c.cfg.Logger.Print(err) + } + } + + // Call startWatcher for context support (From Go 1.8) + mc.startWatcher() + if err := mc.watchCancel(ctx); err != nil { + mc.cleanup() + return nil, err + } + defer mc.finish() + + mc.buf = newBuffer() + + // Reading Handshake Initialization Packet + authData, serverCapabilities, serverExtCapabilities, plugin, err := mc.readHandshakePacket() + if err != nil { + mc.cleanup() + return nil, err + } + + if plugin == "" { + plugin = defaultAuthPlugin + } + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + // try the default auth plugin, if using the requested plugin failed + c.cfg.Logger.Print("could not use requested auth plugin '"+plugin+"': ", err.Error()) + plugin = defaultAuthPlugin + authResp, err = mc.auth(authData, plugin) + if err != nil { + mc.cleanup() + return nil, err + } + } + mc.initCapabilities(serverCapabilities, serverExtCapabilities, mc.cfg) + if err = mc.writeHandshakeResponsePacket(authResp, plugin); err != nil { + mc.cleanup() + return nil, err + } + + // Handle response to auth packet, switch methods if possible + if err = mc.handleAuthResult(authData, plugin); err != nil { + // Authentication failed and MySQL has already closed the connection + // (https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase.html#sect_protocol_connection_phase_fast_path_fails). + // Do not send COM_QUIT, just cleanup and return the error. + mc.cleanup() + return nil, err + } + + // compression is enabled after auth, not right after sending handshake response. + if mc.capabilities&clientCompress > 0 { + mc.compress = true + mc.compIO = newCompIO(mc) + } + if mc.cfg.MaxAllowedPacket > 0 { + mc.maxAllowedPacket = mc.cfg.MaxAllowedPacket + } else { + // Get max allowed packet size + maxap, err := mc.getSystemVar("max_allowed_packet") + if err != nil { + mc.Close() + return nil, err + } + n, err := strconv.Atoi(maxap) + if err != nil { + mc.Close() + return nil, fmt.Errorf("invalid max_allowed_packet value (%q): %w", maxap, err) + } + mc.maxAllowedPacket = n - 1 + } + if mc.maxAllowedPacket < maxPacketSize { + mc.maxWriteSize = mc.maxAllowedPacket + } + + // Charset: character_set_connection, character_set_client, character_set_results + if len(mc.cfg.charsets) > 0 { + for _, cs := range mc.cfg.charsets { + // ignore errors here - a charset may not exist + if mc.cfg.Collation != "" { + err = mc.exec("SET NAMES " + cs + " COLLATE " + mc.cfg.Collation) + } else { + err = mc.exec("SET NAMES " + cs) + } + if err == nil { + break + } + } + if err != nil { + mc.Close() + return nil, err + } + } + + // Handle DSN Params + err = mc.handleParams() + if err != nil { + mc.Close() + return nil, err + } + + return mc, nil +} + +// Driver implements driver.Connector interface. +// Driver returns &MySQLDriver{}. +func (c *connector) Driver() driver.Driver { + return &MySQLDriver{} +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/connector_test.go b/_LICENSES/github.com/go-sql-driver/mysql/connector_test.go new file mode 100644 index 0000000000..82d8c5989d --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/connector_test.go @@ -0,0 +1,30 @@ +package mysql + +import ( + "context" + "net" + "testing" + "time" +) + +func TestConnectorReturnsTimeout(t *testing.T) { + connector := newConnector(&Config{ + Net: "tcp", + Addr: "1.1.1.1:1234", + Timeout: 10 * time.Millisecond, + }) + + _, err := connector.Connect(context.Background()) + if err == nil { + t.Fatal("error expected") + } + + if nerr, ok := err.(*net.OpError); ok { + expected := "dial tcp 1.1.1.1:1234: i/o timeout" + if nerr.Error() != expected { + t.Fatalf("expected %q, got %q", expected, nerr.Error()) + } + } else { + t.Fatalf("expected %T, got %T", nerr, err) + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/const.go b/_LICENSES/github.com/go-sql-driver/mysql/const.go new file mode 100644 index 0000000000..6f0cdf3032 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/const.go @@ -0,0 +1,205 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import "runtime" + +const ( + debug = false // for debugging. Set true only in development. + + defaultAuthPlugin = "mysql_native_password" + defaultMaxAllowedPacket = 64 << 20 // 64 MiB. See https://github.com/go-sql-driver/mysql/issues/1355 + minProtocolVersion = 10 + maxPacketSize = 1<<24 - 1 + timeFormat = "2006-01-02 15:04:05.999999" + + // Connection attributes + // See https://dev.mysql.com/doc/refman/8.0/en/performance-schema-connection-attribute-tables.html#performance-schema-connection-attributes-available + connAttrClientName = "_client_name" + connAttrClientNameValue = "Go-MySQL-Driver" + connAttrOS = "_os" + connAttrOSValue = runtime.GOOS + connAttrPlatform = "_platform" + connAttrPlatformValue = runtime.GOARCH + connAttrPid = "_pid" + connAttrServerHost = "_server_host" +) + +// MySQL constants documentation: +// https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_PROTOCOL.html + +const ( + iOK byte = 0x00 + iAuthMoreData byte = 0x01 + iLocalInFile byte = 0xfb + iEOF byte = 0xfe + iERR byte = 0xff +) + +// https://dev.mysql.com/doc/dev/mysql-server/latest/group__group__cs__capabilities__flags.html +// https://mariadb.com/kb/en/connection/#capabilities +type capabilityFlag uint32 + +const ( + clientMySQL capabilityFlag = 1 << iota + clientFoundRows + clientLongFlag + clientConnectWithDB + clientNoSchema + clientCompress + clientODBC + clientLocalFiles + clientIgnoreSpace + clientProtocol41 + clientInteractive + clientSSL + clientIgnoreSIGPIPE + clientTransactions + clientReserved + clientSecureConn + clientMultiStatements + clientMultiResults + clientPSMultiResults + clientPluginAuth + clientConnectAttrs + clientPluginAuthLenEncClientData + clientCanHandleExpiredPasswords + clientSessionTrack + clientDeprecateEOF +) + +// https://mariadb.com/kb/en/connection/#capabilities +type extendedCapabilityFlag uint32 + +const ( + progressIndicator extendedCapabilityFlag = 1 << iota + clientComMulti + clientStmtBulkOperations + clientExtendedMetadata + clientCacheMetadata + clientUnitBulkResult +) + +const ( + comQuit byte = iota + 1 + comInitDB + comQuery + comFieldList + comCreateDB + comDropDB + comRefresh + comShutdown + comStatistics + comProcessInfo + comConnect + comProcessKill + comDebug + comPing + comTime + comDelayedInsert + comChangeUser + comBinlogDump + comTableDump + comConnectOut + comRegisterSlave + comStmtPrepare + comStmtExecute + comStmtSendLongData + comStmtClose + comStmtReset + comSetOption + comStmtFetch +) + +// https://dev.mysql.com/doc/internals/en/com-query-response.html#packet-Protocol::ColumnType +type fieldType byte + +const ( + fieldTypeDecimal fieldType = iota + fieldTypeTiny + fieldTypeShort + fieldTypeLong + fieldTypeFloat + fieldTypeDouble + fieldTypeNULL + fieldTypeTimestamp + fieldTypeLongLong + fieldTypeInt24 + fieldTypeDate + fieldTypeTime + fieldTypeDateTime + fieldTypeYear + fieldTypeNewDate + fieldTypeVarChar + fieldTypeBit +) +const ( + fieldTypeVector fieldType = iota + 0xf2 + fieldTypeInvalid + fieldTypeBool + fieldTypeJSON + fieldTypeNewDecimal + fieldTypeEnum + fieldTypeSet + fieldTypeTinyBLOB + fieldTypeMediumBLOB + fieldTypeLongBLOB + fieldTypeBLOB + fieldTypeVarString + fieldTypeString + fieldTypeGeometry +) + +type fieldFlag uint16 + +const ( + flagNotNULL fieldFlag = 1 << iota + flagPriKey + flagUniqueKey + flagMultipleKey + flagBLOB + flagUnsigned + flagZeroFill + flagBinary + flagEnum + flagAutoIncrement + flagTimestamp + flagSet + flagUnknown1 + flagUnknown2 + flagUnknown3 + flagUnknown4 +) + +// http://dev.mysql.com/doc/internals/en/status-flags.html +type statusFlag uint16 + +const ( + statusInTrans statusFlag = 1 << iota + statusInAutocommit + statusReserved // Not in documentation + statusMoreResultsExists + statusNoGoodIndexUsed + statusNoIndexUsed + statusCursorExists + statusLastRowSent + statusDbDropped + statusNoBackslashEscapes + statusMetadataChanged + statusQueryWasSlow + statusPsOutParams + statusInTransReadonly + statusSessionStateChanged +) + +const ( + cachingSha2PasswordRequestPublicKey = 2 + cachingSha2PasswordFastAuthSuccess = 3 + cachingSha2PasswordPerformFullAuthentication = 4 +) diff --git a/_LICENSES/github.com/go-sql-driver/mysql/driver.go b/_LICENSES/github.com/go-sql-driver/mysql/driver.go new file mode 100644 index 0000000000..105316b816 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/driver.go @@ -0,0 +1,118 @@ +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +// Package mysql provides a MySQL driver for Go's database/sql package. +// +// The driver should be used via the database/sql package: +// +// import "database/sql" +// import _ "github.com/go-sql-driver/mysql" +// +// db, err := sql.Open("mysql", "user:password@/dbname") +// +// See https://github.com/go-sql-driver/mysql#usage for details +package mysql + +import ( + "context" + "database/sql" + "database/sql/driver" + "net" + "sync" +) + +// MySQLDriver is exported to make the driver directly accessible. +// In general the driver is used via the database/sql package. +type MySQLDriver struct{} + +// DialFunc is a function which can be used to establish the network connection. +// Custom dial functions must be registered with RegisterDial +// +// Deprecated: users should register a DialContextFunc instead +type DialFunc func(addr string) (net.Conn, error) + +// DialContextFunc is a function which can be used to establish the network connection. +// Custom dial functions must be registered with RegisterDialContext +type DialContextFunc func(ctx context.Context, addr string) (net.Conn, error) + +var ( + dialsLock sync.RWMutex + dials map[string]DialContextFunc +) + +// RegisterDialContext registers a custom dial function. It can then be used by the +// network address mynet(addr), where mynet is the registered new network. +// The current context for the connection and its address is passed to the dial function. +func RegisterDialContext(net string, dial DialContextFunc) { + dialsLock.Lock() + defer dialsLock.Unlock() + if dials == nil { + dials = make(map[string]DialContextFunc) + } + dials[net] = dial +} + +// DeregisterDialContext removes the custom dial function registered with the given net. +func DeregisterDialContext(net string) { + dialsLock.Lock() + defer dialsLock.Unlock() + if dials != nil { + delete(dials, net) + } +} + +// RegisterDial registers a custom dial function. It can then be used by the +// network address mynet(addr), where mynet is the registered new network. +// addr is passed as a parameter to the dial function. +// +// Deprecated: users should call RegisterDialContext instead +func RegisterDial(network string, dial DialFunc) { + RegisterDialContext(network, func(_ context.Context, addr string) (net.Conn, error) { + return dial(addr) + }) +} + +// Open new Connection. +// See https://github.com/go-sql-driver/mysql#dsn-data-source-name for how +// the DSN string is formatted +func (d MySQLDriver) Open(dsn string) (driver.Conn, error) { + cfg, err := ParseDSN(dsn) + if err != nil { + return nil, err + } + c := newConnector(cfg) + return c.Connect(context.Background()) +} + +// This variable can be replaced with -ldflags like below: +// go build "-ldflags=-X github.com/go-sql-driver/mysql.driverName=custom" +var driverName = "mysql" + +func init() { + if driverName != "" { + sql.Register(driverName, &MySQLDriver{}) + } +} + +// NewConnector returns new driver.Connector. +func NewConnector(cfg *Config) (driver.Connector, error) { + cfg = cfg.Clone() + // normalize the contents of cfg so calls to NewConnector have the same + // behavior as MySQLDriver.OpenConnector + if err := cfg.normalize(); err != nil { + return nil, err + } + return newConnector(cfg), nil +} + +// OpenConnector implements driver.DriverContext. +func (d MySQLDriver) OpenConnector(dsn string) (driver.Connector, error) { + cfg, err := ParseDSN(dsn) + if err != nil { + return nil, err + } + return newConnector(cfg), nil +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/driver_test.go b/_LICENSES/github.com/go-sql-driver/mysql/driver_test.go new file mode 100644 index 0000000000..761236f547 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/driver_test.go @@ -0,0 +1,3659 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "context" + "crypto/rand" + "crypto/tls" + "database/sql" + "database/sql/driver" + "encoding/json" + "fmt" + "io" + "log" + "math" + mrand "math/rand" + "net" + "net/url" + "os" + "reflect" + "runtime" + "slices" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +// This variable can be replaced with -ldflags like below: +// go test "-ldflags=-X github.com/go-sql-driver/mysql.driverNameTest=custom" +var driverNameTest string + +func init() { + if driverNameTest == "" { + driverNameTest = driverName + } +} + +// Ensure that all the driver interfaces are implemented +var ( + _ driver.Rows = &binaryRows{} + _ driver.Rows = &textRows{} +) + +var ( + user string + pass string + prot string + addr string + dbname string + dsn string + netAddr string + available bool +) + +var ( + tDate = time.Date(2012, 6, 14, 0, 0, 0, 0, time.UTC) + sDate = "2012-06-14" + tDateTime = time.Date(2011, 11, 20, 21, 27, 37, 0, time.UTC) + sDateTime = "2011-11-20 21:27:37" + tDate0 = time.Time{} + sDate0 = "0000-00-00" + sDateTime0 = "0000-00-00 00:00:00" +) + +// See https://github.com/go-sql-driver/mysql/wiki/Testing +func init() { + // get environment variables + env := func(key, defaultValue string) string { + if value := os.Getenv(key); value != "" { + return value + } + return defaultValue + } + user = env("MYSQL_TEST_USER", "root") + pass = env("MYSQL_TEST_PASS", "") + prot = env("MYSQL_TEST_PROT", "tcp") + addr = env("MYSQL_TEST_ADDR", "localhost:3306") + dbname = env("MYSQL_TEST_DBNAME", "gotest") + netAddr = fmt.Sprintf("%s(%s)", prot, addr) + dsn = fmt.Sprintf("%s:%s@%s/%s?timeout=30s", user, pass, netAddr, dbname) + c, err := net.Dial(prot, addr) + if err == nil { + available = true + c.Close() + } +} + +type DBTest struct { + testing.TB + db *sql.DB +} + +type netErrorMock struct { + temporary bool + timeout bool +} + +func (e netErrorMock) Temporary() bool { + return e.temporary +} + +func (e netErrorMock) Timeout() bool { + return e.timeout +} + +func (e netErrorMock) Error() string { + return fmt.Sprintf("mock net error. Temporary: %v, Timeout %v", e.temporary, e.timeout) +} + +func runTestsWithMultiStatement(t *testing.T, dsn string, tests ...func(dbt *DBTest)) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + dsn += "&multiStatements=true" + var db *sql.DB + if _, err := ParseDSN(dsn); err != errInvalidDSNUnsafeCollation { + db, err = sql.Open(driverNameTest, dsn) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + defer db.Close() + } + // Previous test may be skipped without dropping the test table + db.Exec("DROP TABLE IF EXISTS test") + + dbt := &DBTest{t, db} + for _, test := range tests { + test(dbt) + dbt.db.Exec("DROP TABLE IF EXISTS test") + } +} + +func runTests(t *testing.T, dsn string, tests ...func(dbt *DBTest)) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + db, err := sql.Open(driverNameTest, dsn) + if err != nil { + t.Fatalf("connecting %q: %s", dsn, err) + } + defer db.Close() + if err = db.Ping(); err != nil { + t.Fatalf("connecting %q: %s", dsn, err) + } + + dsn2 := dsn + "&interpolateParams=true" + var db2 *sql.DB + if _, err := ParseDSN(dsn2); err != errInvalidDSNUnsafeCollation { + db2, err = sql.Open(driverNameTest, dsn2) + if err != nil { + t.Fatalf("connecting %q: %s", dsn2, err) + } + defer db2.Close() + } + + dsn3 := dsn + "&compress=true" + var db3 *sql.DB + db3, err = sql.Open(driverNameTest, dsn3) + if err != nil { + t.Fatalf("connecting %q: %s", dsn3, err) + } + defer db3.Close() + + cleanupSql := "DROP TABLE IF EXISTS test" + + for _, test := range tests { + t.Run("default", func(t *testing.T) { + dbt := &DBTest{t, db} + t.Cleanup(func() { + db.Exec(cleanupSql) + }) + test(dbt) + }) + if db2 != nil { + t.Run("interpolateParams", func(t *testing.T) { + dbt2 := &DBTest{t, db2} + t.Cleanup(func() { + db2.Exec(cleanupSql) + }) + test(dbt2) + }) + } + t.Run("compress", func(t *testing.T) { + dbt3 := &DBTest{t, db3} + t.Cleanup(func() { + db3.Exec(cleanupSql) + }) + test(dbt3) + }) + } +} + +// runTestsParallel runs the tests in parallel with a separate database connection for each test. +func runTestsParallel(t *testing.T, dsn string, tests ...func(dbt *DBTest, tableName string)) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + newTableName := func(t *testing.T) string { + t.Helper() + var buf [8]byte + if _, err := rand.Read(buf[:]); err != nil { + t.Fatal(err) + } + return fmt.Sprintf("test_%x", buf[:]) + } + + t.Parallel() + for _, test := range tests { + + t.Run("default", func(t *testing.T) { + t.Parallel() + + tableName := newTableName(t) + db, err := sql.Open("mysql", dsn) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + t.Cleanup(func() { + db.Exec("DROP TABLE IF EXISTS " + tableName) + db.Close() + }) + + dbt := &DBTest{t, db} + test(dbt, tableName) + }) + + dsn2 := dsn + "&interpolateParams=true" + if _, err := ParseDSN(dsn2); err == errInvalidDSNUnsafeCollation { + t.Run("interpolateParams", func(t *testing.T) { + t.Parallel() + + tableName := newTableName(t) + db, err := sql.Open("mysql", dsn2) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + t.Cleanup(func() { + db.Exec("DROP TABLE IF EXISTS " + tableName) + db.Close() + }) + + dbt := &DBTest{t, db} + test(dbt, tableName) + }) + } + } +} + +func (dbt *DBTest) fail(method, query string, err error) { + dbt.Helper() + if len(query) > 300 { + query = "[query too large to print]" + } + dbt.Fatalf("error on %s %s: %s", method, query, err.Error()) +} + +func (dbt *DBTest) mustExec(query string, args ...any) (res sql.Result) { + dbt.Helper() + res, err := dbt.db.Exec(query, args...) + if err != nil { + dbt.fail("exec", query, err) + } + return res +} + +func (dbt *DBTest) mustQuery(query string, args ...any) (rows *sql.Rows) { + dbt.Helper() + rows, err := dbt.db.Query(query, args...) + if err != nil { + dbt.fail("query", query, err) + } + return rows +} + +func maybeSkip(t *testing.T, err error, skipErrno uint16) { + mySQLErr, ok := err.(*MySQLError) + if !ok { + return + } + + if mySQLErr.Number == skipErrno { + t.Skipf("skipping test for error: %v", err) + } +} + +func TestEmptyQuery(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + // just a comment, no query + rows := dbt.mustQuery("--") + defer rows.Close() + // will hang before #255 + if rows.Next() { + dbt.Errorf("next on rows must be false") + } + }) +} + +func TestCRUD(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + // Create Table + dbt.mustExec("CREATE TABLE " + tbl + " (value BOOL)") + + // Test for unexpected data + var out bool + rows := dbt.mustQuery("SELECT * FROM " + tbl) + if rows.Next() { + dbt.Error("unexpected data in empty table") + } + rows.Close() + + // Create Data + res := dbt.mustExec("INSERT INTO " + tbl + " VALUES (1)") + count, err := res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 1 { + dbt.Fatalf("expected 1 affected row, got %d", count) + } + + id, err := res.LastInsertId() + if err != nil { + dbt.Fatalf("res.LastInsertId() returned error: %s", err.Error()) + } + if id != 0 { + dbt.Fatalf("expected InsertId 0, got %d", id) + } + + // Read + rows = dbt.mustQuery("SELECT value FROM " + tbl) + if rows.Next() { + rows.Scan(&out) + if true != out { + dbt.Errorf("true != %t", out) + } + + if rows.Next() { + dbt.Error("unexpected data") + } + } else { + dbt.Error("no data") + } + rows.Close() + + // Update + res = dbt.mustExec("UPDATE "+tbl+" SET value = ? WHERE value = ?", false, true) + count, err = res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 1 { + dbt.Fatalf("expected 1 affected row, got %d", count) + } + + // Check Update + rows = dbt.mustQuery("SELECT value FROM " + tbl) + if rows.Next() { + rows.Scan(&out) + if false != out { + dbt.Errorf("false != %t", out) + } + + if rows.Next() { + dbt.Error("unexpected data") + } + } else { + dbt.Error("no data") + } + rows.Close() + + // Delete + res = dbt.mustExec("DELETE FROM "+tbl+" WHERE value = ?", false) + count, err = res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 1 { + dbt.Fatalf("expected 1 affected row, got %d", count) + } + + // Check for unexpected rows + res = dbt.mustExec("DELETE FROM " + tbl) + count, err = res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 0 { + dbt.Fatalf("expected 0 affected row, got %d", count) + } + }) +} + +// TestNumbers test that selecting numeric columns. +// Both of textRows and binaryRows should return same type and value. +func TestNumbersToAny(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (id INT PRIMARY KEY, b BOOL, i8 TINYINT, " + + "i16 SMALLINT, i32 INT, i64 BIGINT, f32 FLOAT, f64 DOUBLE, iu32 INT UNSIGNED)") + dbt.mustExec("INSERT INTO " + tbl + " VALUES (1, true, 127, 32767, 2147483647, 9223372036854775807, 1.25, 2.5, 4294967295)") + + // Use binaryRows for interpolateParams=false and textRows for interpolateParams=true. + rows := dbt.mustQuery("SELECT b, i8, i16, i32, i64, f32, f64, iu32 FROM "+tbl+" WHERE id=?", 1) + if !rows.Next() { + dbt.Fatal("no data") + } + var b, i8, i16, i32, i64, f32, f64, iu32 any + err := rows.Scan(&b, &i8, &i16, &i32, &i64, &f32, &f64, &iu32) + if err != nil { + dbt.Fatal(err) + } + if b.(int64) != 1 { + dbt.Errorf("b != 1") + } + if i8.(int64) != 127 { + dbt.Errorf("i8 != 127") + } + if i16.(int64) != 32767 { + dbt.Errorf("i16 != 32767") + } + if i32.(int64) != 2147483647 { + dbt.Errorf("i32 != 2147483647") + } + if i64.(int64) != 9223372036854775807 { + dbt.Errorf("i64 != 9223372036854775807") + } + if f32.(float32) != 1.25 { + dbt.Errorf("f32 != 1.25") + } + if f64.(float64) != 2.5 { + dbt.Errorf("f64 != 2.5") + } + if iu32.(int64) != 4294967295 { + dbt.Errorf("iu32 != 4294967295") + } + }) +} + +func TestMultiQuery(t *testing.T) { + runTestsWithMultiStatement(t, dsn, func(dbt *DBTest) { + // Create Table + dbt.mustExec("CREATE TABLE `test` (`id` int(11) NOT NULL, `value` int(11) NOT NULL) ") + + // Create Data + res := dbt.mustExec("INSERT INTO test VALUES (1, 1)") + count, err := res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 1 { + dbt.Fatalf("expected 1 affected row, got %d", count) + } + + // Update + res = dbt.mustExec("UPDATE test SET value = 3 WHERE id = 1; UPDATE test SET value = 4 WHERE id = 1; UPDATE test SET value = 5 WHERE id = 1;") + count, err = res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 1 { + dbt.Fatalf("expected 1 affected row, got %d", count) + } + + // Read + var out int + rows := dbt.mustQuery("SELECT value FROM test WHERE id=1;") + if rows.Next() { + rows.Scan(&out) + if out != 5 { + dbt.Errorf("expected 5, got %d", out) + } + + if rows.Next() { + dbt.Error("unexpected data") + } + } else { + dbt.Error("no data") + } + rows.Close() + + }) +} + +func TestInt(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + types := [5]string{"TINYINT", "SMALLINT", "MEDIUMINT", "INT", "BIGINT"} + in := int64(42) + var out int64 + var rows *sql.Rows + + // SIGNED + for _, v := range types { + dbt.mustExec("CREATE TABLE " + tbl + " (value " + v + ")") + + dbt.mustExec("INSERT INTO "+tbl+" VALUES (?)", in) + + rows = dbt.mustQuery("SELECT value FROM " + tbl) + if rows.Next() { + rows.Scan(&out) + if in != out { + dbt.Errorf("%s: %d != %d", v, in, out) + } + } else { + dbt.Errorf("%s: no data", v) + } + rows.Close() + + dbt.mustExec("DROP TABLE IF EXISTS " + tbl) + } + + // UNSIGNED ZEROFILL + for _, v := range types { + dbt.mustExec("CREATE TABLE " + tbl + " (value " + v + " ZEROFILL)") + + dbt.mustExec("INSERT INTO "+tbl+" VALUES (?)", in) + + rows = dbt.mustQuery("SELECT value FROM " + tbl) + if rows.Next() { + rows.Scan(&out) + if in != out { + dbt.Errorf("%s ZEROFILL: %d != %d", v, in, out) + } + } else { + dbt.Errorf("%s ZEROFILL: no data", v) + } + rows.Close() + + dbt.mustExec("DROP TABLE IF EXISTS " + tbl) + } + }) +} + +func TestFloat32(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + types := [2]string{"FLOAT", "DOUBLE"} + in := float32(42.23) + var out float32 + var rows *sql.Rows + for _, v := range types { + dbt.mustExec("CREATE TABLE " + tbl + " (value " + v + ")") + dbt.mustExec("INSERT INTO "+tbl+" VALUES (?)", in) + rows = dbt.mustQuery("SELECT value FROM " + tbl) + if rows.Next() { + rows.Scan(&out) + if in != out { + dbt.Errorf("%s: %g != %g", v, in, out) + } + } else { + dbt.Errorf("%s: no data", v) + } + rows.Close() + dbt.mustExec("DROP TABLE IF EXISTS " + tbl) + } + }) +} + +func TestFloat64(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + types := [2]string{"FLOAT", "DOUBLE"} + var expected float64 = 42.23 + var out float64 + var rows *sql.Rows + for _, v := range types { + dbt.mustExec("CREATE TABLE " + tbl + " (value " + v + ")") + dbt.mustExec("INSERT INTO " + tbl + " VALUES (42.23)") + rows = dbt.mustQuery("SELECT value FROM " + tbl) + if rows.Next() { + rows.Scan(&out) + if expected != out { + dbt.Errorf("%s: %g != %g", v, expected, out) + } + } else { + dbt.Errorf("%s: no data", v) + } + rows.Close() + dbt.mustExec("DROP TABLE IF EXISTS " + tbl) + } + }) +} + +func TestFloat64Placeholder(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + types := [2]string{"FLOAT", "DOUBLE"} + var expected float64 = 42.23 + var out float64 + var rows *sql.Rows + for _, v := range types { + dbt.mustExec("CREATE TABLE " + tbl + " (id int, value " + v + ")") + dbt.mustExec("INSERT INTO " + tbl + " VALUES (1, 42.23)") + rows = dbt.mustQuery("SELECT value FROM "+tbl+" WHERE id = ?", 1) + if rows.Next() { + rows.Scan(&out) + if expected != out { + dbt.Errorf("%s: %g != %g", v, expected, out) + } + } else { + dbt.Errorf("%s: no data", v) + } + rows.Close() + dbt.mustExec("DROP TABLE IF EXISTS " + tbl) + } + }) +} + +func TestString(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + types := [6]string{"CHAR(255)", "VARCHAR(255)", "TINYTEXT", "TEXT", "MEDIUMTEXT", "LONGTEXT"} + in := "κόσμε üöäßñóùéàâÿœ'îë Árvíztűrő いろはにほへとちりぬるを イロハニホヘト דג סקרן чащах น่าฟังเอย" + var out string + var rows *sql.Rows + + for _, v := range types { + dbt.mustExec("CREATE TABLE " + tbl + " (value " + v + ") CHARACTER SET utf8") + + dbt.mustExec("INSERT INTO "+tbl+" VALUES (?)", in) + + rows = dbt.mustQuery("SELECT value FROM " + tbl) + if rows.Next() { + rows.Scan(&out) + if in != out { + dbt.Errorf("%s: %s != %s", v, in, out) + } + } else { + dbt.Errorf("%s: no data", v) + } + rows.Close() + + dbt.mustExec("DROP TABLE IF EXISTS " + tbl) + } + + // BLOB + dbt.mustExec("CREATE TABLE " + tbl + " (id int, value BLOB) CHARACTER SET utf8") + + id := 2 + in = "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, " + + "sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, " + + "sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. " + + "Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. " + + "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, " + + "sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, " + + "sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. " + + "Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet." + dbt.mustExec("INSERT INTO "+tbl+" VALUES (?, ?)", id, in) + + err := dbt.db.QueryRow("SELECT value FROM "+tbl+" WHERE id = ?", id).Scan(&out) + if err != nil { + dbt.Fatalf("Error on BLOB-Query: %s", err.Error()) + } else if out != in { + dbt.Errorf("BLOB: %s != %s", in, out) + } + }) +} + +func TestRawBytes(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + v1 := []byte("aaa") + v2 := []byte("bbb") + rows := dbt.mustQuery("SELECT ?, ?", v1, v2) + defer rows.Close() + if rows.Next() { + var o1, o2 sql.RawBytes + if err := rows.Scan(&o1, &o2); err != nil { + dbt.Errorf("Got error: %v", err) + } + if !bytes.Equal(v1, o1) { + dbt.Errorf("expected %v, got %v", v1, o1) + } + if !bytes.Equal(v2, o2) { + dbt.Errorf("expected %v, got %v", v2, o2) + } + // https://github.com/go-sql-driver/mysql/issues/765 + // Appending to RawBytes shouldn't overwrite next RawBytes. + o1 = append(o1, "xyzzy"...) + if !bytes.Equal(v2, o2) { + dbt.Errorf("expected %v, got %v", v2, o2) + } + } else { + dbt.Errorf("no data") + } + }) +} + +func TestRawMessage(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + v1 := json.RawMessage("{}") + v2 := json.RawMessage("[]") + rows := dbt.mustQuery("SELECT ?, ?", v1, v2) + defer rows.Close() + if rows.Next() { + var o1, o2 json.RawMessage + if err := rows.Scan(&o1, &o2); err != nil { + dbt.Errorf("Got error: %v", err) + } + if !bytes.Equal(v1, o1) { + dbt.Errorf("expected %v, got %v", v1, o1) + } + if !bytes.Equal(v2, o2) { + dbt.Errorf("expected %v, got %v", v2, o2) + } + } else { + dbt.Errorf("no data") + } + }) +} + +type testValuer struct { + value string +} + +func (tv testValuer) Value() (driver.Value, error) { + return tv.value, nil +} + +func TestValuer(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + in := testValuer{"a_value"} + var out string + var rows *sql.Rows + + dbt.mustExec("CREATE TABLE " + tbl + " (value VARCHAR(255)) CHARACTER SET utf8") + dbt.mustExec("INSERT INTO "+tbl+" VALUES (?)", in) + rows = dbt.mustQuery("SELECT value FROM " + tbl) + if rows.Next() { + rows.Scan(&out) + if in.value != out { + dbt.Errorf("Valuer: %v != %s", in, out) + } + } else { + dbt.Errorf("Valuer: no data") + } + rows.Close() + }) +} + +type testValuerWithValidation struct { + value string +} + +func (tv testValuerWithValidation) Value() (driver.Value, error) { + if len(tv.value) == 0 { + return nil, fmt.Errorf("Invalid string valuer. Value must not be empty") + } + + return tv.value, nil +} + +func TestValuerWithValidation(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + in := testValuerWithValidation{"a_value"} + var out string + var rows *sql.Rows + + dbt.mustExec("CREATE TABLE " + tbl + " (value VARCHAR(255)) CHARACTER SET utf8") + dbt.mustExec("INSERT INTO "+tbl+" VALUES (?)", in) + + rows = dbt.mustQuery("SELECT value FROM " + tbl) + defer rows.Close() + + if rows.Next() { + rows.Scan(&out) + if in.value != out { + dbt.Errorf("Valuer: %v != %s", in, out) + } + } else { + dbt.Errorf("Valuer: no data") + } + + if _, err := dbt.db.Exec("INSERT INTO "+tbl+" VALUES (?)", testValuerWithValidation{""}); err == nil { + dbt.Errorf("Failed to check valuer error") + } + + if _, err := dbt.db.Exec("INSERT INTO "+tbl+" VALUES (?)", nil); err != nil { + dbt.Errorf("Failed to check nil") + } + + if _, err := dbt.db.Exec("INSERT INTO "+tbl+" VALUES (?)", map[string]bool{}); err == nil { + dbt.Errorf("Failed to check not valuer") + } + }) +} + +type timeTests struct { + dbtype string + tlayout string + tests []timeTest +} + +type timeTest struct { + s string // leading "!": do not use t as value in queries + t time.Time +} + +type timeMode byte + +func (t timeMode) String() string { + switch t { + case binaryString: + return "binary:string" + case binaryTime: + return "binary:time.Time" + case textString: + return "text:string" + } + panic("unsupported timeMode") +} + +func (t timeMode) Binary() bool { + switch t { + case binaryString, binaryTime: + return true + } + return false +} + +const ( + binaryString timeMode = iota + binaryTime + textString +) + +func (t timeTest) genQuery(dbtype string, mode timeMode) string { + var inner string + if mode.Binary() { + inner = "?" + } else { + inner = `"%s"` + } + return `SELECT cast(` + inner + ` as ` + dbtype + `)` +} + +func (t timeTest) run(dbt *DBTest, dbtype, tlayout string, mode timeMode) { + var rows *sql.Rows + query := t.genQuery(dbtype, mode) + switch mode { + case binaryString: + rows = dbt.mustQuery(query, t.s) + case binaryTime: + rows = dbt.mustQuery(query, t.t) + case textString: + query = fmt.Sprintf(query, t.s) + rows = dbt.mustQuery(query) + default: + panic("unsupported mode") + } + defer rows.Close() + var err error + if !rows.Next() { + err = rows.Err() + if err == nil { + err = fmt.Errorf("no data") + } + dbt.Errorf("%s [%s]: %s", dbtype, mode, err) + return + } + var dst any + err = rows.Scan(&dst) + if err != nil { + dbt.Errorf("%s [%s]: %s", dbtype, mode, err) + return + } + switch val := dst.(type) { + case []uint8: + str := string(val) + if str == t.s { + return + } + if mode.Binary() && dbtype == "DATETIME" && len(str) == 26 && str[:19] == t.s { + // a fix mainly for TravisCI: + // accept full microsecond resolution in result for DATETIME columns + // where the binary protocol was used + return + } + dbt.Errorf("%s [%s] to string: expected %q, got %q", + dbtype, mode, + t.s, str, + ) + case time.Time: + if val == t.t { + return + } + dbt.Errorf("%s [%s] to string: expected %q, got %q", + dbtype, mode, + t.s, val.Format(tlayout), + ) + default: + fmt.Printf("%#v\n", []any{dbtype, tlayout, mode, t.s, t.t}) + dbt.Errorf("%s [%s]: unhandled type %T (is '%v')", + dbtype, mode, + val, val, + ) + } +} + +func TestDateTime(t *testing.T) { + afterTime := func(t time.Time, d string) time.Time { + dur, err := time.ParseDuration(d) + if err != nil { + panic(err) + } + return t.Add(dur) + } + // NOTE: MySQL rounds DATETIME(x) up - but that's not included in the tests + format := "2006-01-02 15:04:05.999999" + t0 := time.Time{} + tstr0 := "0000-00-00 00:00:00.000000" + testcases := []timeTests{ + {"DATE", format[:10], []timeTest{ + {t: time.Date(2011, 11, 20, 0, 0, 0, 0, time.UTC)}, + {t: t0, s: tstr0[:10]}, + }}, + {"DATETIME", format[:19], []timeTest{ + {t: time.Date(2011, 11, 20, 21, 27, 37, 0, time.UTC)}, + {t: t0, s: tstr0[:19]}, + }}, + {"DATETIME(0)", format[:21], []timeTest{ + {t: time.Date(2011, 11, 20, 21, 27, 37, 0, time.UTC)}, + {t: t0, s: tstr0[:19]}, + }}, + {"DATETIME(1)", format[:21], []timeTest{ + {t: time.Date(2011, 11, 20, 21, 27, 37, 100000000, time.UTC)}, + {t: t0, s: tstr0[:21]}, + }}, + {"DATETIME(6)", format, []timeTest{ + {t: time.Date(2011, 11, 20, 21, 27, 37, 123456000, time.UTC)}, + {t: t0, s: tstr0}, + }}, + {"TIME", format[11:19], []timeTest{ + {t: afterTime(t0, "12345s")}, + {s: "!-12:34:56"}, + {s: "!-838:59:59"}, + {s: "!838:59:59"}, + {t: t0, s: tstr0[11:19]}, + }}, + {"TIME(0)", format[11:19], []timeTest{ + {t: afterTime(t0, "12345s")}, + {s: "!-12:34:56"}, + {s: "!-838:59:59"}, + {s: "!838:59:59"}, + {t: t0, s: tstr0[11:19]}, + }}, + {"TIME(1)", format[11:21], []timeTest{ + {t: afterTime(t0, "12345600ms")}, + {s: "!-12:34:56.7"}, + {s: "!-838:59:58.9"}, + {s: "!838:59:58.9"}, + {t: t0, s: tstr0[11:21]}, + }}, + {"TIME(6)", format[11:], []timeTest{ + {t: afterTime(t0, "1234567890123000ns")}, + {s: "!-12:34:56.789012"}, + {s: "!-838:59:58.999999"}, + {s: "!838:59:58.999999"}, + {t: t0, s: tstr0[11:]}, + }}, + } + dsns := []string{ + dsn + "&parseTime=true", + dsn + "&parseTime=false", + } + for _, testdsn := range dsns { + runTests(t, testdsn, func(dbt *DBTest) { + microsecsSupported := false + zeroDateSupported := false + var rows *sql.Rows + var err error + rows, err = dbt.db.Query(`SELECT cast("00:00:00.1" as TIME(1)) = "00:00:00.1"`) + if err == nil { + if rows.Next() { + rows.Scan(µsecsSupported) + } + rows.Close() + } + rows, err = dbt.db.Query(`SELECT cast("0000-00-00" as DATE) = "0000-00-00"`) + if err == nil { + if rows.Next() { + rows.Scan(&zeroDateSupported) + } + rows.Close() + } + for _, setups := range testcases { + if t := setups.dbtype; !microsecsSupported && t[len(t)-1:] == ")" { + // skip fractional second tests if unsupported by server + continue + } + for _, setup := range setups.tests { + allowBinTime := true + if setup.s == "" { + // fill time string wherever Go can reliable produce it + setup.s = setup.t.Format(setups.tlayout) + } else if setup.s[0] == '!' { + // skip tests using setup.t as source in queries + allowBinTime = false + // fix setup.s - remove the "!" + setup.s = setup.s[1:] + } + if !zeroDateSupported && setup.s == tstr0[:len(setup.s)] { + // skip disallowed 0000-00-00 date + continue + } + setup.run(dbt, setups.dbtype, setups.tlayout, textString) + setup.run(dbt, setups.dbtype, setups.tlayout, binaryString) + if allowBinTime { + setup.run(dbt, setups.dbtype, setups.tlayout, binaryTime) + } + } + } + }) + } +} + +func TestTimestampMicros(t *testing.T) { + format := "2006-01-02 15:04:05.999999" + f0 := format[:19] + f1 := format[:21] + f6 := format[:26] + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + // check if microseconds are supported. + // Do not use timestamp(x) for that check - before 5.5.6, x would mean display width + // and not precision. + // Se last paragraph at http://dev.mysql.com/doc/refman/5.6/en/fractional-seconds.html + microsecsSupported := false + if rows, err := dbt.db.Query(`SELECT cast("00:00:00.1" as TIME(1)) = "00:00:00.1"`); err == nil { + rows.Scan(µsecsSupported) + rows.Close() + } + if !microsecsSupported { + // skip test + return + } + _, err := dbt.db.Exec(` + CREATE TABLE ` + tbl + ` ( + value0 TIMESTAMP NOT NULL DEFAULT '` + f0 + `', + value1 TIMESTAMP(1) NOT NULL DEFAULT '` + f1 + `', + value6 TIMESTAMP(6) NOT NULL DEFAULT '` + f6 + `' + )`, + ) + if err != nil { + dbt.Error(err) + } + defer dbt.mustExec("DROP TABLE IF EXISTS " + tbl) + dbt.mustExec("INSERT INTO "+tbl+" SET value0=?, value1=?, value6=?", f0, f1, f6) + var res0, res1, res6 string + rows := dbt.mustQuery("SELECT * FROM " + tbl) + defer rows.Close() + if !rows.Next() { + dbt.Errorf("test contained no selectable values") + } + err = rows.Scan(&res0, &res1, &res6) + if err != nil { + dbt.Error(err) + } + if res0 != f0 { + dbt.Errorf("expected %q, got %q", f0, res0) + } + if res1 != f1 { + dbt.Errorf("expected %q, got %q", f1, res1) + } + if res6 != f6 { + dbt.Errorf("expected %q, got %q", f6, res6) + } + }) +} + +func TestNULL(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + nullStmt, err := dbt.db.Prepare("SELECT NULL") + if err != nil { + dbt.Fatal(err) + } + defer nullStmt.Close() + + nonNullStmt, err := dbt.db.Prepare("SELECT 1") + if err != nil { + dbt.Fatal(err) + } + defer nonNullStmt.Close() + + // NullBool + var nb sql.NullBool + // Invalid + if err = nullStmt.QueryRow().Scan(&nb); err != nil { + dbt.Fatal(err) + } + if nb.Valid { + dbt.Error("valid NullBool which should be invalid") + } + // Valid + if err = nonNullStmt.QueryRow().Scan(&nb); err != nil { + dbt.Fatal(err) + } + if !nb.Valid { + dbt.Error("invalid NullBool which should be valid") + } else if nb.Bool != true { + dbt.Errorf("Unexpected NullBool value: %t (should be true)", nb.Bool) + } + + // NullFloat64 + var nf sql.NullFloat64 + // Invalid + if err = nullStmt.QueryRow().Scan(&nf); err != nil { + dbt.Fatal(err) + } + if nf.Valid { + dbt.Error("valid NullFloat64 which should be invalid") + } + // Valid + if err = nonNullStmt.QueryRow().Scan(&nf); err != nil { + dbt.Fatal(err) + } + if !nf.Valid { + dbt.Error("invalid NullFloat64 which should be valid") + } else if nf.Float64 != float64(1) { + dbt.Errorf("unexpected NullFloat64 value: %f (should be 1.0)", nf.Float64) + } + + // NullInt64 + var ni sql.NullInt64 + // Invalid + if err = nullStmt.QueryRow().Scan(&ni); err != nil { + dbt.Fatal(err) + } + if ni.Valid { + dbt.Error("valid NullInt64 which should be invalid") + } + // Valid + if err = nonNullStmt.QueryRow().Scan(&ni); err != nil { + dbt.Fatal(err) + } + if !ni.Valid { + dbt.Error("invalid NullInt64 which should be valid") + } else if ni.Int64 != int64(1) { + dbt.Errorf("unexpected NullInt64 value: %d (should be 1)", ni.Int64) + } + + // NullString + var ns sql.NullString + // Invalid + if err = nullStmt.QueryRow().Scan(&ns); err != nil { + dbt.Fatal(err) + } + if ns.Valid { + dbt.Error("valid NullString which should be invalid") + } + // Valid + if err = nonNullStmt.QueryRow().Scan(&ns); err != nil { + dbt.Fatal(err) + } + if !ns.Valid { + dbt.Error("invalid NullString which should be valid") + } else if ns.String != `1` { + dbt.Error("unexpected NullString value:" + ns.String + " (should be `1`)") + } + + // nil-bytes + var b []byte + // Read nil + if err = nullStmt.QueryRow().Scan(&b); err != nil { + dbt.Fatal(err) + } + if b != nil { + dbt.Error("non-nil []byte which should be nil") + } + // Read non-nil + if err = nonNullStmt.QueryRow().Scan(&b); err != nil { + dbt.Fatal(err) + } + if b == nil { + dbt.Error("nil []byte which should be non-nil") + } + // Insert nil + b = nil + success := false + if err = dbt.db.QueryRow("SELECT ? IS NULL", b).Scan(&success); err != nil { + dbt.Fatal(err) + } + if !success { + dbt.Error("inserting []byte(nil) as NULL failed") + } + // Check input==output with input==nil + b = nil + if err = dbt.db.QueryRow("SELECT ?", b).Scan(&b); err != nil { + dbt.Fatal(err) + } + if b != nil { + dbt.Error("non-nil echo from nil input") + } + // Check input==output with input!=nil + b = []byte("") + if err = dbt.db.QueryRow("SELECT ?", b).Scan(&b); err != nil { + dbt.Fatal(err) + } + if b == nil { + dbt.Error("nil echo from non-nil input") + } + + // Insert NULL + dbt.mustExec("CREATE TABLE " + tbl + " (dummmy1 int, value int, dummy2 int)") + + dbt.mustExec("INSERT INTO "+tbl+" VALUES (?, ?, ?)", 1, nil, 2) + + var out any + rows := dbt.mustQuery("SELECT * FROM " + tbl) + defer rows.Close() + if rows.Next() { + rows.Scan(&out) + if out != nil { + dbt.Errorf("%v != nil", out) + } + } else { + dbt.Error("no data") + } + }) +} + +func TestUint64(t *testing.T) { + const ( + u0 = uint64(0) + uall = ^u0 + uhigh = uall >> 1 + utop = ^uhigh + s0 = int64(0) + sall = ^s0 + shigh = int64(uhigh) + stop = ^shigh + ) + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + stmt, err := dbt.db.Prepare(`SELECT ?, ?, ? ,?, ?, ?, ?, ?`) + if err != nil { + dbt.Fatal(err) + } + defer stmt.Close() + row := stmt.QueryRow( + u0, uhigh, utop, uall, + s0, shigh, stop, sall, + ) + + var ua, ub, uc, ud uint64 + var sa, sb, sc, sd int64 + + err = row.Scan(&ua, &ub, &uc, &ud, &sa, &sb, &sc, &sd) + if err != nil { + dbt.Fatal(err) + } + switch { + case ua != u0, + ub != uhigh, + uc != utop, + ud != uall, + sa != s0, + sb != shigh, + sc != stop, + sd != sall: + dbt.Fatal("unexpected result value") + } + }) +} + +func TestLongData(t *testing.T) { + runTests(t, dsn+"&maxAllowedPacket=0", func(dbt *DBTest) { + var maxAllowedPacketSize int + err := dbt.db.QueryRow("select @@max_allowed_packet").Scan(&maxAllowedPacketSize) + if err != nil { + dbt.Fatal(err) + } + maxAllowedPacketSize-- + + // don't get too ambitious + if maxAllowedPacketSize > 1<<25 { + maxAllowedPacketSize = 1 << 25 + } + + dbt.mustExec("CREATE TABLE test (value LONGBLOB)") + + in := strings.Repeat(`a`, maxAllowedPacketSize+1) + var out string + var rows *sql.Rows + + // Long text data + inS := in[:maxAllowedPacketSize-100] + dbt.mustExec("INSERT INTO test VALUES('" + inS + "')") + rows = dbt.mustQuery("SELECT value FROM test") + defer rows.Close() + if rows.Next() { + rows.Scan(&out) + if inS != out { + dbt.Fatalf("LONGBLOB: length in: %d, length out: %d", len(inS), len(out)) + } + if rows.Next() { + dbt.Error("LONGBLOB: unexpected row") + } + } else { + dbt.Fatalf("LONGBLOB: no data") + } + + // Empty table + dbt.mustExec("TRUNCATE TABLE test") + + // Long binary data + dbt.mustExec("INSERT INTO test VALUES(?)", in) + rows = dbt.mustQuery("SELECT value FROM test WHERE 1=?", 1) + defer rows.Close() + if rows.Next() { + rows.Scan(&out) + if in != out { + dbt.Fatalf("LONGBLOB: length in: %d, length out: %d", len(in), len(out)) + } + if rows.Next() { + dbt.Error("LONGBLOB: unexpected row") + } + } else { + if err = rows.Err(); err != nil { + dbt.Fatalf("LONGBLOB: no data (err: %s)", err.Error()) + } else { + dbt.Fatal("LONGBLOB: no data (err: )") + } + } + }) +} + +func TestLoadData(t *testing.T) { + runTests(t, dsn, func(dbt *DBTest) { + verifyLoadDataResult := func() { + rows, err := dbt.db.Query("SELECT * FROM test") + if err != nil { + dbt.Fatal(err.Error()) + } + + i := 0 + values := [4]string{ + "a string", + "a string containing a \t", + "a string containing a \n", + "a string containing both \t\n", + } + + var id int + var value string + + for rows.Next() { + i++ + err = rows.Scan(&id, &value) + if err != nil { + dbt.Fatal(err.Error()) + } + if i != id { + dbt.Fatalf("%d != %d", i, id) + } + if values[i-1] != value { + dbt.Fatalf("%q != %q", values[i-1], value) + } + } + err = rows.Err() + if err != nil { + dbt.Fatal(err.Error()) + } + + if i != 4 { + dbt.Fatalf("rows count mismatch. Got %d, want 4", i) + } + } + + dbt.db.Exec("DROP TABLE IF EXISTS test") + dbt.mustExec("CREATE TABLE test (id INT NOT NULL PRIMARY KEY, value TEXT NOT NULL) CHARACTER SET utf8") + + // Local File + file, err := os.CreateTemp("", "gotest") + defer os.Remove(file.Name()) + if err != nil { + dbt.Fatal(err) + } + RegisterLocalFile(file.Name()) + + // Try first with empty file + dbt.mustExec(fmt.Sprintf("LOAD DATA LOCAL INFILE %q INTO TABLE test", file.Name())) + var count int + err = dbt.db.QueryRow("SELECT COUNT(*) FROM test").Scan(&count) + if err != nil { + dbt.Fatal(err.Error()) + } + if count != 0 { + dbt.Fatalf("unexpected row count: got %d, want 0", count) + } + + // Then fill File with data and try to load it + file.WriteString("1\ta string\n2\ta string containing a \\t\n3\ta string containing a \\n\n4\ta string containing both \\t\\n\n") + file.Close() + dbt.mustExec(fmt.Sprintf("LOAD DATA LOCAL INFILE %q INTO TABLE test", file.Name())) + verifyLoadDataResult() + + // Try with non-existing file + _, err = dbt.db.Exec("LOAD DATA LOCAL INFILE 'doesnotexist' INTO TABLE test") + if err == nil { + dbt.Fatal("load non-existent file didn't fail") + } else if err.Error() != "local file 'doesnotexist' is not registered" { + dbt.Fatal(err.Error()) + } + + // Empty table + dbt.mustExec("TRUNCATE TABLE test") + + // Reader + RegisterReaderHandler("test", func() io.Reader { + file, err = os.Open(file.Name()) + if err != nil { + dbt.Fatal(err) + } + return file + }) + dbt.mustExec("LOAD DATA LOCAL INFILE 'Reader::test' INTO TABLE test") + verifyLoadDataResult() + // negative test + _, err = dbt.db.Exec("LOAD DATA LOCAL INFILE 'Reader::doesnotexist' INTO TABLE test") + if err == nil { + dbt.Fatal("load non-existent Reader didn't fail") + } else if err.Error() != "reader 'doesnotexist' is not registered" { + dbt.Fatal(err.Error()) + } + }) +} + +func TestFoundRows1(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (id INT NOT NULL ,data INT NOT NULL)") + dbt.mustExec("INSERT INTO " + tbl + " (id, data) VALUES (0, 0),(0, 0),(1, 0),(1, 0),(1, 1)") + + res := dbt.mustExec("UPDATE " + tbl + " SET data = 1 WHERE id = 0") + count, err := res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 2 { + dbt.Fatalf("Expected 2 affected rows, got %d", count) + } + res = dbt.mustExec("UPDATE " + tbl + " SET data = 1 WHERE id = 1") + count, err = res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 2 { + dbt.Fatalf("Expected 2 affected rows, got %d", count) + } + }) +} + +func TestFoundRows2(t *testing.T) { + runTestsParallel(t, dsn+"&clientFoundRows=true", func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (id INT NOT NULL ,data INT NOT NULL)") + dbt.mustExec("INSERT INTO " + tbl + " (id, data) VALUES (0, 0),(0, 0),(1, 0),(1, 0),(1, 1)") + + res := dbt.mustExec("UPDATE " + tbl + " SET data = 1 WHERE id = 0") + count, err := res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 2 { + dbt.Fatalf("Expected 2 matched rows, got %d", count) + } + res = dbt.mustExec("UPDATE " + tbl + " SET data = 1 WHERE id = 1") + count, err = res.RowsAffected() + if err != nil { + dbt.Fatalf("res.RowsAffected() returned error: %s", err.Error()) + } + if count != 3 { + dbt.Fatalf("Expected 3 matched rows, got %d", count) + } + }) +} + +func TestTLS(t *testing.T) { + tlsTestReq := func(dbt *DBTest) { + if err := dbt.db.Ping(); err != nil { + if err == ErrNoTLS { + dbt.Skip("server does not support TLS") + } else { + dbt.Fatalf("error on Ping: %s", err.Error()) + } + } + + rows := dbt.mustQuery("SHOW STATUS LIKE 'Ssl_cipher'") + defer rows.Close() + + var variable, value *sql.RawBytes + for rows.Next() { + if err := rows.Scan(&variable, &value); err != nil { + dbt.Fatal(err.Error()) + } + + if len(*value) == 0 { + dbt.Fatalf("no Cipher") + } else { + dbt.Logf("Cipher: %s", *value) + } + } + } + tlsTestOpt := func(dbt *DBTest) { + if err := dbt.db.Ping(); err != nil { + dbt.Fatalf("error on Ping: %s", err.Error()) + } + } + + runTests(t, dsn+"&tls=preferred", tlsTestOpt) + runTests(t, dsn+"&tls=skip-verify", tlsTestReq) + + // Verify that registering / using a custom cfg works + RegisterTLSConfig("custom-skip-verify", &tls.Config{ + InsecureSkipVerify: true, + }) + runTests(t, dsn+"&tls=custom-skip-verify", tlsTestReq) +} + +func TestReuseClosedConnection(t *testing.T) { + // this test does not use sql.database, it uses the driver directly + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + md := &MySQLDriver{} + conn, err := md.Open(dsn) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + stmt, err := conn.Prepare("DO 1") + if err != nil { + t.Fatalf("error preparing statement: %s", err.Error()) + } + //lint:ignore SA1019 this is a test + _, err = stmt.Exec(nil) + if err != nil { + t.Fatalf("error executing statement: %s", err.Error()) + } + err = conn.Close() + if err != nil { + t.Fatalf("error closing connection: %s", err.Error()) + } + + defer func() { + if err := recover(); err != nil { + t.Errorf("panic after reusing a closed connection: %v", err) + } + }() + //lint:ignore SA1019 this is a test + _, err = stmt.Exec(nil) + if err != nil && err != driver.ErrBadConn { + t.Errorf("unexpected error '%s', expected '%s'", + err.Error(), driver.ErrBadConn.Error()) + } +} + +func TestCharset(t *testing.T) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + mustSetCharset := func(charsetParam, expected string) { + runTests(t, dsn+"&"+charsetParam, func(dbt *DBTest) { + rows := dbt.mustQuery("SELECT @@character_set_connection") + defer rows.Close() + + if !rows.Next() { + dbt.Fatalf("error getting connection charset: %s", rows.Err()) + } + + var got string + rows.Scan(&got) + + if got != expected { + dbt.Fatalf("expected connection charset %s but got %s", expected, got) + } + }) + } + + // non utf8 test + mustSetCharset("charset=ascii", "ascii") + + // when the first charset is invalid, use the second + mustSetCharset("charset=none,utf8mb4", "utf8mb4") + + // when the first charset is valid, use it + mustSetCharset("charset=ascii,utf8mb4", "ascii") + mustSetCharset("charset=utf8mb4,ascii", "utf8mb4") +} + +func TestFailingCharset(t *testing.T) { + runTestsParallel(t, dsn+"&charset=none", func(dbt *DBTest, _ string) { + // run query to really establish connection... + _, err := dbt.db.Exec("SELECT 1") + if err == nil { + dbt.db.Close() + t.Fatalf("connection must not succeed without a valid charset") + } + }) +} + +func TestCollation(t *testing.T) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + // MariaDB may override collation specified by handshake with `character_set_collations` variable. + // https://mariadb.com/kb/en/setting-character-sets-and-collations/#changing-default-collation + // https://mariadb.com/kb/en/server-system-variables/#character_set_collations + // utf8mb4_general_ci, utf8mb3_general_ci will be overridden by default MariaDB. + // Collations other than charasets default are not overridden. So utf8mb4_unicode_ci is safe. + testCollations := []string{ + "latin1_general_ci", + "binary", + "utf8mb4_unicode_ci", + "cp1257_bin", + } + + for _, collation := range testCollations { + t.Run(collation, func(t *testing.T) { + tdsn := dsn + "&collation=" + collation + expected := collation + + runTests(t, tdsn, func(dbt *DBTest) { + var got string + if err := dbt.db.QueryRow("SELECT @@collation_connection").Scan(&got); err != nil { + dbt.Fatal(err) + } + if got != expected { + dbt.Fatalf("expected connection collation %s but got %s", expected, got) + } + }) + }) + } +} + +func TestColumnsWithAlias(t *testing.T) { + runTestsParallel(t, dsn+"&columnsWithAlias=true", func(dbt *DBTest, _ string) { + rows := dbt.mustQuery("SELECT 1 AS A") + defer rows.Close() + cols, _ := rows.Columns() + if len(cols) != 1 { + t.Fatalf("expected 1 column, got %d", len(cols)) + } + if cols[0] != "A" { + t.Fatalf("expected column name \"A\", got \"%s\"", cols[0]) + } + + rows = dbt.mustQuery("SELECT * FROM (SELECT 1 AS one) AS A") + defer rows.Close() + cols, _ = rows.Columns() + if len(cols) != 1 { + t.Fatalf("expected 1 column, got %d", len(cols)) + } + if cols[0] != "A.one" { + t.Fatalf("expected column name \"A.one\", got \"%s\"", cols[0]) + } + }) +} + +func TestRawBytesResultExceedsBuffer(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + // defaultBufSize from buffer.go + expected := strings.Repeat("abc", defaultBufSize) + + rows := dbt.mustQuery("SELECT '" + expected + "'") + defer rows.Close() + if !rows.Next() { + dbt.Error("expected result, got none") + } + var result sql.RawBytes + rows.Scan(&result) + if expected != string(result) { + dbt.Error("result did not match expected value") + } + }) +} + +func TestTimezoneConversion(t *testing.T) { + zones := []string{"UTC", "America/New_York", "Asia/Hong_Kong", "Local"} + + // Regression test for timezone handling + tzTest := func(dbt *DBTest) { + // Create table + dbt.mustExec("CREATE TABLE test (ts TIMESTAMP)") + + // Insert local time into database (should be converted) + newYorkTz, _ := time.LoadLocation("America/New_York") + reftime := time.Date(2014, 05, 30, 18, 03, 17, 0, time.UTC).In(newYorkTz) + dbt.mustExec("INSERT INTO test VALUE (?)", reftime) + + // Retrieve time from DB + rows := dbt.mustQuery("SELECT ts FROM test") + defer rows.Close() + if !rows.Next() { + dbt.Fatal("did not get any rows out") + } + + var dbTime time.Time + err := rows.Scan(&dbTime) + if err != nil { + dbt.Fatal("Err", err) + } + + // Check that dates match + if reftime.Unix() != dbTime.Unix() { + dbt.Errorf("times do not match.\n") + dbt.Errorf(" Now(%v)=%v\n", newYorkTz, reftime) + dbt.Errorf(" Now(UTC)=%v\n", dbTime) + } + } + + for _, tz := range zones { + runTests(t, dsn+"&parseTime=true&loc="+url.QueryEscape(tz), tzTest) + } +} + +// Special cases + +func TestRowsClose(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + rows, err := dbt.db.Query("SELECT 1") + if err != nil { + dbt.Fatal(err) + } + + err = rows.Close() + if err != nil { + dbt.Fatal(err) + } + + if rows.Next() { + dbt.Fatal("unexpected row after rows.Close()") + } + + err = rows.Err() + if err != nil { + dbt.Fatal(err) + } + }) +} + +// dangling statements +// http://code.google.com/p/go/issues/detail?id=3865 +func TestCloseStmtBeforeRows(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + stmt, err := dbt.db.Prepare("SELECT 1") + if err != nil { + dbt.Fatal(err) + } + + rows, err := stmt.Query() + if err != nil { + stmt.Close() + dbt.Fatal(err) + } + defer rows.Close() + + err = stmt.Close() + if err != nil { + dbt.Fatal(err) + } + + if !rows.Next() { + dbt.Fatal("getting row failed") + } else { + err = rows.Err() + if err != nil { + dbt.Fatal(err) + } + + var out bool + err = rows.Scan(&out) + if err != nil { + dbt.Fatalf("error on rows.Scan(): %s", err.Error()) + } + if out != true { + dbt.Errorf("true != %t", out) + } + } + }) +} + +// It is valid to have multiple Rows for the same Stmt +// http://code.google.com/p/go/issues/detail?id=3734 +func TestStmtMultiRows(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + stmt, err := dbt.db.Prepare("SELECT 1 UNION SELECT 0") + if err != nil { + dbt.Fatal(err) + } + + rows1, err := stmt.Query() + if err != nil { + stmt.Close() + dbt.Fatal(err) + } + defer rows1.Close() + + rows2, err := stmt.Query() + if err != nil { + stmt.Close() + dbt.Fatal(err) + } + defer rows2.Close() + + var out bool + + // 1 + if !rows1.Next() { + dbt.Fatal("first rows1.Next failed") + } else { + err = rows1.Err() + if err != nil { + dbt.Fatal(err) + } + + err = rows1.Scan(&out) + if err != nil { + dbt.Fatalf("error on rows.Scan(): %s", err.Error()) + } + if out != true { + dbt.Errorf("true != %t", out) + } + } + + if !rows2.Next() { + dbt.Fatal("first rows2.Next failed") + } else { + err = rows2.Err() + if err != nil { + dbt.Fatal(err) + } + + err = rows2.Scan(&out) + if err != nil { + dbt.Fatalf("error on rows.Scan(): %s", err.Error()) + } + if out != true { + dbt.Errorf("true != %t", out) + } + } + + // 2 + if !rows1.Next() { + dbt.Fatal("second rows1.Next failed") + } else { + err = rows1.Err() + if err != nil { + dbt.Fatal(err) + } + + err = rows1.Scan(&out) + if err != nil { + dbt.Fatalf("error on rows.Scan(): %s", err.Error()) + } + if out != false { + dbt.Errorf("false != %t", out) + } + + if rows1.Next() { + dbt.Fatal("unexpected row on rows1") + } + err = rows1.Close() + if err != nil { + dbt.Fatal(err) + } + } + + if !rows2.Next() { + dbt.Fatal("second rows2.Next failed") + } else { + err = rows2.Err() + if err != nil { + dbt.Fatal(err) + } + + err = rows2.Scan(&out) + if err != nil { + dbt.Fatalf("error on rows.Scan(): %s", err.Error()) + } + if out != false { + dbt.Errorf("false != %t", out) + } + + if rows2.Next() { + dbt.Fatal("unexpected row on rows2") + } + err = rows2.Close() + if err != nil { + dbt.Fatal(err) + } + } + }) +} + +// Regression test for +// * more than 32 NULL parameters (issue 209) +// * more parameters than fit into the buffer (issue 201) +// * parameters * 64 > max_allowed_packet (issue 734) +func TestPreparedManyCols(t *testing.T) { + numParams := 65535 + runTests(t, dsn, func(dbt *DBTest) { + query := "SELECT ?" + strings.Repeat(",?", numParams-1) + stmt, err := dbt.db.Prepare(query) + if err != nil { + dbt.Fatal(err) + } + defer stmt.Close() + + // create more parameters than fit into the buffer + // which will take nil-values + params := make([]any, numParams) + rows, err := stmt.Query(params...) + if err != nil { + dbt.Fatal(err) + } + rows.Close() + + // Create 0byte string which we can't send via STMT_LONG_DATA. + for i := range numParams { + params[i] = "" + } + rows, err = stmt.Query(params...) + if err != nil { + dbt.Fatal(err) + } + rows.Close() + }) +} + +func TestConcurrent(t *testing.T) { + if enabled, _ := readBool(os.Getenv("MYSQL_TEST_CONCURRENT")); !enabled { + t.Skip("MYSQL_TEST_CONCURRENT env var not set") + } + + runTests(t, dsn, func(dbt *DBTest) { + // var version string + // if err := dbt.db.QueryRow("SELECT @@version").Scan(&version); err != nil { + // dbt.Fatal(err) + // } + // if strings.Contains(strings.ToLower(version), "mariadb") { + // t.Skip(`TODO: "fix commands out of sync. Did you run multiple statements at once?" on MariaDB`) + // } + + // Release idle connections after this subtest so subsequent subtests can + // use the server's full connection budget. Without this, the default pool + // retains a few idle connections that eat into max_connections, causing + // Windows to return a TCP timeout instead of MySQL error 1040 when the + // next subtest tries to open connections. + defer dbt.db.SetMaxIdleConns(0) + + var max int + err := dbt.db.QueryRow("SELECT @@max_connections").Scan(&max) + if err != nil { + dbt.Fatalf("%s", err.Error()) + } + dbt.Logf("testing up to %d concurrent connections \r\n", max) + + var remaining, succeeded int32 = int32(max), 0 + + var wg sync.WaitGroup + wg.Add(max) + + var fatalError string + var once sync.Once + fatalf := func(s string, vals ...any) { + once.Do(func() { + fatalError = fmt.Sprintf(s, vals...) + }) + } + + for i := range max { + go func(id int) { + defer wg.Done() + + tx, err := dbt.db.Begin() + + if err != nil { + if err.Error() != "Error 1040: Too many connections" { + fatalf("error on conn %d: %s", id, err.Error()) + } + return + } + + // keep the connection busy until all connections are open + for atomic.AddInt32(&remaining, -1) > 0 { + if _, err = tx.Exec("DO 1"); err != nil { + fatalf("error on conn %d: %s", id, err.Error()) + return + } + } + + if err = tx.Commit(); err != nil { + fatalf("error on conn %d: %s", id, err.Error()) + return + } + + // everything went fine with this connection + atomic.AddInt32(&succeeded, 1) + }(i) + } + + // wait until all connections are open + wg.Wait() + + if fatalError != "" { + dbt.Fatal(fatalError) + } + + dbt.Logf("reached %d concurrent connections\r\n", succeeded) + }) +} + +func testDialError(t *testing.T, dialErr error, expectErr error) { + RegisterDialContext("mydial", func(ctx context.Context, addr string) (net.Conn, error) { + return nil, dialErr + }) + + db, err := sql.Open(driverNameTest, fmt.Sprintf("%s:%s@mydial(%s)/%s?timeout=30s", user, pass, addr, dbname)) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + defer db.Close() + + _, err = db.Exec("DO 1") + if err != expectErr { + t.Fatalf("was expecting %s. Got: %s", dialErr, err) + } +} + +func TestDialUnknownError(t *testing.T) { + testErr := fmt.Errorf("test") + testDialError(t, testErr, testErr) +} + +func TestDialNonRetryableNetErr(t *testing.T) { + testErr := netErrorMock{} + testDialError(t, testErr, testErr) +} + +func TestDialTemporaryNetErr(t *testing.T) { + testErr := netErrorMock{temporary: true} + testDialError(t, testErr, testErr) +} + +// Tests custom dial functions +func TestCustomDial(t *testing.T) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + // our custom dial function which just wraps net.Dial here + RegisterDialContext("mydial", func(ctx context.Context, addr string) (net.Conn, error) { + var d net.Dialer + return d.DialContext(ctx, prot, addr) + }) + + db, err := sql.Open(driverNameTest, fmt.Sprintf("%s:%s@mydial(%s)/%s?timeout=30s", user, pass, addr, dbname)) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + defer db.Close() + + if _, err = db.Exec("DO 1"); err != nil { + t.Fatalf("connection failed: %s", err.Error()) + } +} + +func TestBeforeConnect(t *testing.T) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + // dbname is set in the BeforeConnect handle + cfg, err := ParseDSN(fmt.Sprintf("%s:%s@%s/%s?timeout=30s", user, pass, netAddr, "_")) + if err != nil { + t.Fatalf("error parsing DSN: %v", err) + } + + cfg.Apply(BeforeConnect(func(ctx context.Context, c *Config) error { + c.DBName = dbname + return nil + })) + + connector, err := NewConnector(cfg) + if err != nil { + t.Fatalf("error creating connector: %v", err) + } + + db := sql.OpenDB(connector) + defer db.Close() + + var connectedDb string + err = db.QueryRow("SELECT DATABASE();").Scan(&connectedDb) + if err != nil { + t.Fatalf("error executing query: %v", err) + } + if connectedDb != dbname { + t.Fatalf("expected to connect to DB %s, but connected to %s instead", dbname, connectedDb) + } +} + +func TestSQLInjection(t *testing.T) { + createTest := func(arg string) func(dbt *DBTest) { + return func(dbt *DBTest) { + dbt.mustExec("CREATE TABLE test (v INTEGER)") + dbt.mustExec("INSERT INTO test VALUES (?)", 1) + + var v int + // NULL can't be equal to anything, the idea here is to inject query so it returns row + // This test verifies that escapeQuotes and escapeBackslash are working properly + err := dbt.db.QueryRow("SELECT v FROM test WHERE NULL = ?", arg).Scan(&v) + if err == sql.ErrNoRows { + return // success, sql injection failed + } else if err == nil { + dbt.Errorf("sql injection successful with arg: %s", arg) + } else { + dbt.Errorf("error running query with arg: %s; err: %s", arg, err.Error()) + } + } + } + + dsns := []string{ + dsn, + dsn + "&sql_mode='NO_BACKSLASH_ESCAPES'", + } + for _, testdsn := range dsns { + runTests(t, testdsn, createTest("1 OR 1=1")) + runTests(t, testdsn, createTest("' OR '1'='1")) + } +} + +// Test if inserted data is correctly retrieved after being escaped +func TestInsertRetrieveEscapedData(t *testing.T) { + testData := func(dbt *DBTest) { + dbt.mustExec("CREATE TABLE test (v VARCHAR(255))") + + // All sequences that are escaped by escapeQuotes and escapeBackslash + v := "foo \x00\n\r\x1a\"'\\" + dbt.mustExec("INSERT INTO test VALUES (?)", v) + + var out string + err := dbt.db.QueryRow("SELECT v FROM test").Scan(&out) + if err != nil { + dbt.Fatalf("%s", err.Error()) + } + + if out != v { + dbt.Errorf("%q != %q", out, v) + } + } + + dsns := []string{ + dsn, + dsn + "&sql_mode='NO_BACKSLASH_ESCAPES'", + } + for _, testdsn := range dsns { + runTests(t, testdsn, testData) + } +} + +func TestUnixSocketAuthFail(t *testing.T) { + runTests(t, dsn, func(dbt *DBTest) { + // Save the current logger so we can restore it. + oldLogger := defaultLogger + + // Set a new logger so we can capture its output. + buffer := bytes.NewBuffer(make([]byte, 0, 64)) + newLogger := log.New(buffer, "prefix: ", 0) + SetLogger(newLogger) + + // Restore the logger. + defer SetLogger(oldLogger) + + // Make a new DSN that uses the MySQL socket file and a bad password, which + // we can make by simply appending any character to the real password. + badPass := pass + "x" + socket := "" + if prot == "unix" { + socket = addr + } else { + // Get socket file from MySQL. + err := dbt.db.QueryRow("SELECT @@socket").Scan(&socket) + if err != nil { + t.Fatalf("error on SELECT @@socket: %s", err.Error()) + } + } + t.Logf("socket: %s", socket) + badDSN := fmt.Sprintf("%s:%s@unix(%s)/%s?timeout=30s", user, badPass, socket, dbname) + db, err := sql.Open(driverNameTest, badDSN) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + defer db.Close() + + // Connect to MySQL for real. This will cause an auth failure. + err = db.Ping() + if err == nil { + t.Error("expected Ping() to return an error") + } + + // The driver should not log anything. + if actual := buffer.String(); actual != "" { + t.Errorf("expected no output, got %q", actual) + } + }) +} + +// See Issue #422 +func TestInterruptBySignal(t *testing.T) { + runTestsWithMultiStatement(t, dsn, func(dbt *DBTest) { + dbt.mustExec(` + DROP PROCEDURE IF EXISTS test_signal; + CREATE PROCEDURE test_signal(ret INT) + BEGIN + SELECT ret; + SIGNAL SQLSTATE + '45001' + SET + MESSAGE_TEXT = "an error", + MYSQL_ERRNO = 45001; + END + `) + defer dbt.mustExec("DROP PROCEDURE test_signal") + + var val int + + // text protocol + rows, err := dbt.db.Query("CALL test_signal(42)") + if err != nil { + dbt.Fatalf("error on text query: %s", err.Error()) + } + for rows.Next() { + if err := rows.Scan(&val); err != nil { + dbt.Error(err) + } else if val != 42 { + dbt.Errorf("expected val to be 42") + } + } + rows.Close() + + // binary protocol + rows, err = dbt.db.Query("CALL test_signal(?)", 42) + if err != nil { + dbt.Fatalf("error on binary query: %s", err.Error()) + } + for rows.Next() { + if err := rows.Scan(&val); err != nil { + dbt.Error(err) + } else if val != 42 { + dbt.Errorf("expected val to be 42") + } + } + rows.Close() + }) +} + +func TestColumnsReusesSlice(t *testing.T) { + rows := mysqlRows{ + rs: resultSet{ + columns: []mysqlField{ + { + tableName: "test", + name: "A", + }, + { + tableName: "test", + name: "B", + }, + }, + }, + } + + allocs := testing.AllocsPerRun(1, func() { + cols := rows.Columns() + + if len(cols) != 2 { + t.Fatalf("expected 2 columns, got %d", len(cols)) + } + }) + + if allocs != 0 { + t.Fatalf("expected 0 allocations, got %d", int(allocs)) + } + + if rows.rs.columnNames == nil { + t.Fatalf("expected columnNames to be set, got nil") + } +} + +func TestRejectReadOnly(t *testing.T) { + runTests(t, dsn, func(dbt *DBTest) { + // Create Table + dbt.mustExec("CREATE TABLE test (value BOOL)") + // Set the session to read-only. We didn't set the `rejectReadOnly` + // option, so any writes after this should fail. + _, err := dbt.db.Exec("SET SESSION TRANSACTION READ ONLY") + // Error 1193: Unknown system variable 'TRANSACTION' => skip test, + // MySQL server version is too old + maybeSkip(t, err, 1193) + if _, err := dbt.db.Exec("DROP TABLE test"); err == nil { + t.Fatalf("writing to DB in read-only session without " + + "rejectReadOnly did not error") + } + // Set the session back to read-write so runTests() can properly clean + // up the table `test`. + dbt.mustExec("SET SESSION TRANSACTION READ WRITE") + }) + + // Enable the `rejectReadOnly` option. + runTests(t, dsn+"&rejectReadOnly=true", func(dbt *DBTest) { + // Create Table + dbt.mustExec("CREATE TABLE test (value BOOL)") + // Set the session to read only. Any writes after this should error on + // a driver.ErrBadConn, and cause `database/sql` to initiate a new + // connection. + dbt.mustExec("SET SESSION TRANSACTION READ ONLY") + // This would error, but `database/sql` should automatically retry on a + // new connection which is not read-only, and eventually succeed. + dbt.mustExec("DROP TABLE test") + }) +} + +func TestPing(t *testing.T) { + ctx := context.Background() + runTests(t, dsn, func(dbt *DBTest) { + if err := dbt.db.Ping(); err != nil { + dbt.fail("Ping", "Ping", err) + } + }) + + runTests(t, dsn, func(dbt *DBTest) { + conn, err := dbt.db.Conn(ctx) + if err != nil { + dbt.fail("db", "Conn", err) + } + + // Check that affectedRows and insertIds are cleared after each call. + conn.Raw(func(conn any) error { + c := conn.(*mysqlConn) + + // Issue a query that sets affectedRows and insertIds. + q, err := c.Query(`SELECT 1`, nil) + if err != nil { + dbt.fail("Conn", "Query", err) + } + if got, want := c.result.affectedRows, []int64{0}; !reflect.DeepEqual(got, want) { + dbt.Fatalf("bad affectedRows: got %v, want=%v", got, want) + } + if got, want := c.result.insertIds, []int64{0}; !reflect.DeepEqual(got, want) { + dbt.Fatalf("bad insertIds: got %v, want=%v", got, want) + } + q.Close() + + // Verify that Ping() clears both fields. + for range 2 { + if err := c.Ping(ctx); err != nil { + dbt.fail("Pinger", "Ping", err) + } + if got, want := c.result.affectedRows, []int64(nil); !reflect.DeepEqual(got, want) { + t.Errorf("bad affectedRows: got %v, want=%v", got, want) + } + if got, want := c.result.insertIds, []int64(nil); !reflect.DeepEqual(got, want) { + t.Errorf("bad affectedRows: got %v, want=%v", got, want) + } + } + return nil + }) + }) +} + +// See Issue #799 +func TestEmptyPassword(t *testing.T) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + dsn := fmt.Sprintf("%s:%s@%s/%s?timeout=30s", user, "", netAddr, dbname) + db, err := sql.Open(driverNameTest, dsn) + if err == nil { + defer db.Close() + err = db.Ping() + } + + if pass == "" { + if err != nil { + t.Fatal(err.Error()) + } + } else { + if err == nil { + t.Fatal("expected authentication error") + } + if !strings.HasPrefix(err.Error(), "Error 1045") { + t.Fatal(err.Error()) + } + } +} + +// static interface implementation checks of mysqlConn +var ( + _ driver.ConnBeginTx = &mysqlConn{} + _ driver.ConnPrepareContext = &mysqlConn{} + _ driver.ExecerContext = &mysqlConn{} + _ driver.Pinger = &mysqlConn{} + _ driver.QueryerContext = &mysqlConn{} +) + +// static interface implementation checks of mysqlStmt +var ( + _ driver.StmtExecContext = &mysqlStmt{} + _ driver.StmtQueryContext = &mysqlStmt{} +) + +// Ensure that all the driver interfaces are implemented +var ( + // _ driver.RowsColumnTypeLength = &binaryRows{} + // _ driver.RowsColumnTypeLength = &textRows{} + _ driver.RowsColumnTypeDatabaseTypeName = &binaryRows{} + _ driver.RowsColumnTypeDatabaseTypeName = &textRows{} + _ driver.RowsColumnTypeNullable = &binaryRows{} + _ driver.RowsColumnTypeNullable = &textRows{} + _ driver.RowsColumnTypePrecisionScale = &binaryRows{} + _ driver.RowsColumnTypePrecisionScale = &textRows{} + _ driver.RowsColumnTypeScanType = &binaryRows{} + _ driver.RowsColumnTypeScanType = &textRows{} + _ driver.RowsNextResultSet = &binaryRows{} + _ driver.RowsNextResultSet = &textRows{} +) + +func TestMultiResultSet(t *testing.T) { + type result struct { + values [][]int + columns []string + } + + // checkRows is a helper test function to validate rows containing 3 result + // sets with specific values and columns. The basic query would look like this: + // + // SELECT 1 AS col1, 2 AS col2 UNION SELECT 3, 4; + // SELECT 0 UNION SELECT 1; + // SELECT 1 AS col1, 2 AS col2, 3 AS col3 UNION SELECT 4, 5, 6; + // + // to distinguish test cases the first string argument is put in front of + // every error or fatal message. + checkRows := func(desc string, rows *sql.Rows, dbt *DBTest) { + expected := []result{ + { + values: [][]int{{1, 2}, {3, 4}}, + columns: []string{"col1", "col2"}, + }, + { + values: [][]int{{1, 2, 3}, {4, 5, 6}}, + columns: []string{"col1", "col2", "col3"}, + }, + } + + var res1 result + for rows.Next() { + var res [2]int + if err := rows.Scan(&res[0], &res[1]); err != nil { + dbt.Fatal(err) + } + res1.values = append(res1.values, res[:]) + } + + cols, err := rows.Columns() + if err != nil { + dbt.Fatal(desc, err) + } + res1.columns = cols + + if !reflect.DeepEqual(expected[0], res1) { + dbt.Error(desc, "want =", expected[0], "got =", res1) + } + + if !rows.NextResultSet() { + dbt.Fatal(desc, "expected next result set") + } + + // ignoring one result set + + if !rows.NextResultSet() { + dbt.Fatal(desc, "expected next result set") + } + + var res2 result + cols, err = rows.Columns() + if err != nil { + dbt.Fatal(desc, err) + } + res2.columns = cols + + for rows.Next() { + var res [3]int + if err := rows.Scan(&res[0], &res[1], &res[2]); err != nil { + dbt.Fatal(desc, err) + } + res2.values = append(res2.values, res[:]) + } + + if !reflect.DeepEqual(expected[1], res2) { + dbt.Error(desc, "want =", expected[1], "got =", res2) + } + + if rows.NextResultSet() { + dbt.Error(desc, "unexpected next result set") + } + + if err := rows.Err(); err != nil { + dbt.Error(desc, err) + } + } + + runTestsWithMultiStatement(t, dsn, func(dbt *DBTest) { + rows := dbt.mustQuery(`DO 1; + SELECT 1 AS col1, 2 AS col2 UNION SELECT 3, 4; + DO 1; + SELECT 0 UNION SELECT 1; + SELECT 1 AS col1, 2 AS col2, 3 AS col3 UNION SELECT 4, 5, 6;`) + defer rows.Close() + checkRows("query: ", rows, dbt) + }) + + runTestsWithMultiStatement(t, dsn, func(dbt *DBTest) { + queries := []string{ + ` + DROP PROCEDURE IF EXISTS test_mrss; + CREATE PROCEDURE test_mrss() + BEGIN + DO 1; + SELECT 1 AS col1, 2 AS col2 UNION SELECT 3, 4; + DO 1; + SELECT 0 UNION SELECT 1; + SELECT 1 AS col1, 2 AS col2, 3 AS col3 UNION SELECT 4, 5, 6; + END + `, + ` + DROP PROCEDURE IF EXISTS test_mrss; + CREATE PROCEDURE test_mrss() + BEGIN + SELECT 1 AS col1, 2 AS col2 UNION SELECT 3, 4; + SELECT 0 UNION SELECT 1; + SELECT 1 AS col1, 2 AS col2, 3 AS col3 UNION SELECT 4, 5, 6; + END + `, + } + + defer dbt.mustExec("DROP PROCEDURE IF EXISTS test_mrss") + + for i, query := range queries { + dbt.mustExec(query) + + stmt, err := dbt.db.Prepare("CALL test_mrss()") + if err != nil { + dbt.Fatalf("%v (i=%d)", err, i) + } + defer stmt.Close() + + for j := range 2 { + rows, err := stmt.Query() + if err != nil { + dbt.Fatalf("%v (i=%d) (j=%d)", err, i, j) + } + checkRows(fmt.Sprintf("prepared stmt query (i=%d) (j=%d): ", i, j), rows, dbt) + } + } + }) +} + +func TestMultiResultSetNoSelect(t *testing.T) { + runTestsWithMultiStatement(t, dsn, func(dbt *DBTest) { + rows := dbt.mustQuery("DO 1; DO 2;") + defer rows.Close() + + if rows.Next() { + dbt.Error("unexpected row") + } + + if rows.NextResultSet() { + dbt.Error("unexpected next result set") + } + + if err := rows.Err(); err != nil { + dbt.Error("expected nil; got ", err) + } + }) +} + +func TestExecMultipleResults(t *testing.T) { + ctx := context.Background() + runTestsWithMultiStatement(t, dsn, func(dbt *DBTest) { + dbt.mustExec(` + CREATE TABLE test ( + id INT NOT NULL AUTO_INCREMENT, + value VARCHAR(255), + PRIMARY KEY (id) + )`) + conn, err := dbt.db.Conn(ctx) + if err != nil { + t.Fatalf("failed to connect: %v", err) + } + conn.Raw(func(conn any) error { + //lint:ignore SA1019 this is a test + ex := conn.(driver.Execer) + res, err := ex.Exec(` + INSERT INTO test (value) VALUES ('a'), ('b'); + INSERT INTO test (value) VALUES ('c'), ('d'), ('e'); + `, nil) + if err != nil { + t.Fatalf("insert statements failed: %v", err) + } + mres := res.(Result) + if got, want := mres.AllRowsAffected(), []int64{2, 3}; !reflect.DeepEqual(got, want) { + t.Errorf("bad AllRowsAffected: got %v, want=%v", got, want) + } + // For INSERTs containing multiple rows, LAST_INSERT_ID() returns the + // first inserted ID, not the last. + if got, want := mres.AllLastInsertIds(), []int64{1, 3}; !reflect.DeepEqual(got, want) { + t.Errorf("bad AllLastInsertIds: got %v, want %v", got, want) + } + return nil + }) + }) +} + +// tests if rows are set in a proper state if some results were ignored before +// calling rows.NextResultSet. +func TestSkipResults(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + rows := dbt.mustQuery("SELECT 1, 2") + defer rows.Close() + + if !rows.Next() { + dbt.Error("expected row") + } + + if rows.NextResultSet() { + dbt.Error("unexpected next result set") + } + + if err := rows.Err(); err != nil { + dbt.Error("expected nil; got ", err) + } + }) +} + +func TestQueryMultipleResults(t *testing.T) { + ctx := context.Background() + runTestsWithMultiStatement(t, dsn, func(dbt *DBTest) { + dbt.mustExec(` + CREATE TABLE test ( + id INT NOT NULL AUTO_INCREMENT, + value VARCHAR(255), + PRIMARY KEY (id) + )`) + conn, err := dbt.db.Conn(ctx) + if err != nil { + t.Fatalf("failed to connect: %v", err) + } + conn.Raw(func(conn any) error { + //lint:ignore SA1019 this is a test + qr := conn.(driver.Queryer) + c := conn.(*mysqlConn) + + // Demonstrate that repeated queries reset the affectedRows + for range 2 { + _, err := qr.Query(` + INSERT INTO test (value) VALUES ('a'), ('b'); + INSERT INTO test (value) VALUES ('c'), ('d'), ('e'); + `, nil) + if err != nil { + t.Fatalf("insert statements failed: %v", err) + } + if got, want := c.result.affectedRows, []int64{2, 3}; !reflect.DeepEqual(got, want) { + t.Errorf("bad affectedRows: got %v, want=%v", got, want) + } + } + return nil + }) + }) +} + +func TestPingContext(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if err := dbt.db.PingContext(ctx); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + }) +} + +func TestContextCancelExec(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (v INTEGER)") + ctx, cancel := context.WithCancel(context.Background()) + + // Delay execution for just a bit until db.ExecContext has begun. + defer time.AfterFunc(250*time.Millisecond, cancel).Stop() + + // This query will be canceled. + startTime := time.Now() + if _, err := dbt.db.ExecContext(ctx, "INSERT INTO "+tbl+" VALUES (SLEEP(1))"); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + if d := time.Since(startTime); d > 500*time.Millisecond { + dbt.Errorf("too long execution time: %s", d) + } + + // Wait for the INSERT query to be done. + time.Sleep(time.Second) + + // Check how many times the query is executed. + var v int + if err := dbt.db.QueryRow("SELECT COUNT(*) FROM " + tbl).Scan(&v); err != nil { + dbt.Fatalf("%s", err.Error()) + } + if v != 1 { // TODO: need to kill the query, and v should be 0. + dbt.Skipf("[WARN] expected val to be 1, got %d", v) + } + + // Context is already canceled, so error should come before execution. + if _, err := dbt.db.ExecContext(ctx, "INSERT INTO "+tbl+" VALUES (1)"); err == nil { + dbt.Error("expected error") + } else if err.Error() != "context canceled" { + dbt.Fatalf("unexpected error: %s", err) + } + + // The second insert query will fail, so the table has no changes. + if err := dbt.db.QueryRow("SELECT COUNT(*) FROM " + tbl).Scan(&v); err != nil { + dbt.Fatalf("%s", err.Error()) + } + if v != 1 { + dbt.Skipf("[WARN] expected val to be 1, got %d", v) + } + }) +} + +func TestContextCancelQuery(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (v INTEGER)") + ctx, cancel := context.WithCancel(context.Background()) + + // Delay execution for just a bit until db.ExecContext has begun. + defer time.AfterFunc(250*time.Millisecond, cancel).Stop() + + // This query will be canceled. + startTime := time.Now() + if _, err := dbt.db.QueryContext(ctx, "INSERT INTO "+tbl+" VALUES (SLEEP(1))"); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + if d := time.Since(startTime); d > 500*time.Millisecond { + dbt.Errorf("too long execution time: %s", d) + } + + // Wait for the INSERT query to be done. + time.Sleep(time.Second) + + // Check how many times the query is executed. + var v int + if err := dbt.db.QueryRow("SELECT COUNT(*) FROM " + tbl).Scan(&v); err != nil { + dbt.Fatalf("%s", err.Error()) + } + if v != 1 { // TODO: need to kill the query, and v should be 0. + dbt.Skipf("[WARN] expected val to be 1, got %d", v) + } + + // Context is already canceled, so error should come before execution. + if _, err := dbt.db.QueryContext(ctx, "INSERT INTO "+tbl+" VALUES (1)"); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + + // The second insert query will fail, so the table has no changes. + if err := dbt.db.QueryRow("SELECT COUNT(*) FROM " + tbl).Scan(&v); err != nil { + dbt.Fatalf("%s", err.Error()) + } + if v != 1 { + dbt.Skipf("[WARN] expected val to be 1, got %d", v) + } + }) +} + +func TestContextCancelQueryRow(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (v INTEGER)") + dbt.mustExec("INSERT INTO " + tbl + " VALUES (1), (2), (3)") + ctx, cancel := context.WithCancel(context.Background()) + + rows, err := dbt.db.QueryContext(ctx, "SELECT v FROM "+tbl) + if err != nil { + dbt.Fatalf("%s", err.Error()) + } + + // the first row will be succeed. + var v int + if !rows.Next() { + dbt.Fatalf("unexpected end") + } + if err := rows.Scan(&v); err != nil { + dbt.Fatalf("%s", err.Error()) + } + + cancel() + // make sure the driver receives the cancel request. + time.Sleep(100 * time.Millisecond) + + if rows.Next() { + dbt.Errorf("expected end, but not") + } + if err := rows.Err(); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + }) +} + +func TestContextCancelPrepare(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, _ string) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if _, err := dbt.db.PrepareContext(ctx, "SELECT 1"); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + }) +} + +func TestContextCancelStmtExec(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (v INTEGER)") + ctx, cancel := context.WithCancel(context.Background()) + stmt, err := dbt.db.PrepareContext(ctx, "INSERT INTO "+tbl+" VALUES (SLEEP(1))") + if err != nil { + dbt.Fatalf("unexpected error: %v", err) + } + + // Delay execution for just a bit until db.ExecContext has begun. + defer time.AfterFunc(250*time.Millisecond, cancel).Stop() + + // This query will be canceled. + startTime := time.Now() + if _, err := stmt.ExecContext(ctx); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + if d := time.Since(startTime); d > 500*time.Millisecond { + dbt.Errorf("too long execution time: %s", d) + } + + // Wait for the INSERT query to be done. + time.Sleep(time.Second) + + // Check how many times the query is executed. + var v int + if err := dbt.db.QueryRow("SELECT COUNT(*) FROM " + tbl).Scan(&v); err != nil { + dbt.Fatalf("%s", err.Error()) + } + if v != 1 { // TODO: need to kill the query, and v should be 0. + dbt.Skipf("[WARN] expected val to be 1, got %d", v) + } + }) +} + +func TestContextCancelStmtQuery(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (v INTEGER)") + ctx, cancel := context.WithCancel(context.Background()) + stmt, err := dbt.db.PrepareContext(ctx, "INSERT INTO "+tbl+" VALUES (SLEEP(1))") + if err != nil { + dbt.Fatalf("unexpected error: %v", err) + } + + // Delay execution for just a bit until db.ExecContext has begun. + defer time.AfterFunc(250*time.Millisecond, cancel).Stop() + + // This query will be canceled. + startTime := time.Now() + if _, err := stmt.QueryContext(ctx); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + if d := time.Since(startTime); d > 500*time.Millisecond { + dbt.Errorf("too long execution time: %s", d) + } + + // Wait for the INSERT query has done. + time.Sleep(time.Second) + + // Check how many times the query is executed. + var v int + if err := dbt.db.QueryRow("SELECT COUNT(*) FROM " + tbl).Scan(&v); err != nil { + dbt.Fatalf("%s", err.Error()) + } + if v != 1 { // TODO: need to kill the query, and v should be 0. + dbt.Skipf("[WARN] expected val to be 1, got %d", v) + } + }) +} + +func TestContextCancelBegin(t *testing.T) { + if runtime.GOOS == "windows" || runtime.GOOS == "darwin" { + t.Skip(`FIXME: it sometime fails with "expected driver.ErrBadConn, got sql: connection is already closed" on windows and macOS`) + } + + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (v INTEGER)") + ctx, cancel := context.WithCancel(context.Background()) + conn, err := dbt.db.Conn(ctx) + if err != nil { + dbt.Fatal(err) + } + defer conn.Close() + tx, err := conn.BeginTx(ctx, nil) + if err != nil { + dbt.Fatal(err) + } + + // Delay execution for just a bit until db.ExecContext has begun. + defer time.AfterFunc(100*time.Millisecond, cancel).Stop() + + // This query will be canceled. + startTime := time.Now() + if _, err := tx.ExecContext(ctx, "INSERT INTO "+tbl+" VALUES (SLEEP(1))"); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + if d := time.Since(startTime); d > 500*time.Millisecond { + dbt.Errorf("too long execution time: %s", d) + } + + // Transaction is canceled, so expect an error. + switch err := tx.Commit(); err { + case sql.ErrTxDone: + // because the transaction has already been rollbacked. + // the database/sql package watches ctx + // and rollbacks when ctx is canceled. + case context.Canceled: + // the database/sql package rollbacks on another goroutine, + // so the transaction may not be rollbacked depending on goroutine scheduling. + default: + dbt.Errorf("expected sql.ErrTxDone or context.Canceled, got %v", err) + } + + // The connection is now in an inoperable state - so performing other + // operations should fail with ErrBadConn + // Important to exercise isolation level too - it runs SET TRANSACTION ISOLATION + // LEVEL XXX first, which needs to return ErrBadConn if the connection's context + // is cancelled + _, err = conn.BeginTx(context.Background(), &sql.TxOptions{Isolation: sql.LevelReadCommitted}) + if err != driver.ErrBadConn { + dbt.Errorf("expected driver.ErrBadConn, got %v", err) + } + + // cannot begin a transaction (on a different conn) with a canceled context + if _, err := dbt.db.BeginTx(ctx, nil); err != context.Canceled { + dbt.Errorf("expected context.Canceled, got %v", err) + } + }) +} + +func TestContextBeginIsolationLevel(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (v INTEGER)") + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + tx1, err := dbt.db.BeginTx(ctx, &sql.TxOptions{ + Isolation: sql.LevelRepeatableRead, + }) + if err != nil { + dbt.Fatal(err) + } + + tx2, err := dbt.db.BeginTx(ctx, &sql.TxOptions{ + Isolation: sql.LevelReadCommitted, + }) + if err != nil { + dbt.Fatal(err) + } + + _, err = tx1.ExecContext(ctx, "INSERT INTO "+tbl+" VALUES (1)") + if err != nil { + dbt.Fatal(err) + } + + var v int + row := tx2.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+tbl) + if err := row.Scan(&v); err != nil { + dbt.Fatal(err) + } + // Because writer transaction wasn't committed yet, it should be available + if v != 0 { + dbt.Errorf("expected val to be 0, got %d", v) + } + + err = tx1.Commit() + if err != nil { + dbt.Fatal(err) + } + + row = tx2.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+tbl) + if err := row.Scan(&v); err != nil { + dbt.Fatal(err) + } + // Data written by writer transaction is already committed, it should be selectable + if v != 1 { + dbt.Errorf("expected val to be 1, got %d", v) + } + tx2.Commit() + }) +} + +func TestContextBeginReadOnly(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (v INTEGER)") + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + tx, err := dbt.db.BeginTx(ctx, &sql.TxOptions{ + ReadOnly: true, + }) + if _, ok := err.(*MySQLError); ok { + dbt.Skip("It seems that your MySQL does not support READ ONLY transactions") + return + } else if err != nil { + dbt.Fatal(err) + } + + // INSERT queries fail in a READ ONLY transaction. + _, err = tx.ExecContext(ctx, "INSERT INTO "+tbl+" VALUES (1)") + if _, ok := err.(*MySQLError); !ok { + dbt.Errorf("expected MySQLError, got %v", err) + } + + // SELECT queries can be executed. + var v int + row := tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+tbl) + if err := row.Scan(&v); err != nil { + dbt.Fatal(err) + } + if v != 0 { + dbt.Errorf("expected val to be 0, got %d", v) + } + + if err := tx.Commit(); err != nil { + dbt.Fatal(err) + } + }) +} + +func TestRowsColumnTypes(t *testing.T) { + niNULL := sql.NullInt64{Int64: 0, Valid: false} + ni0 := sql.NullInt64{Int64: 0, Valid: true} + ni1 := sql.NullInt64{Int64: 1, Valid: true} + ni42 := sql.NullInt64{Int64: 42, Valid: true} + nfNULL := sql.NullFloat64{Float64: 0.0, Valid: false} + nf0 := sql.NullFloat64{Float64: 0.0, Valid: true} + nf1337 := sql.NullFloat64{Float64: 13.37, Valid: true} + nt0 := sql.NullTime{Time: time.Date(2006, 01, 02, 15, 04, 05, 0, time.UTC), Valid: true} + nt1 := sql.NullTime{Time: time.Date(2006, 01, 02, 15, 04, 05, 100000000, time.UTC), Valid: true} + nt2 := sql.NullTime{Time: time.Date(2006, 01, 02, 15, 04, 05, 110000000, time.UTC), Valid: true} + nt6 := sql.NullTime{Time: time.Date(2006, 01, 02, 15, 04, 05, 111111000, time.UTC), Valid: true} + nd1 := sql.NullTime{Time: time.Date(2006, 01, 02, 0, 0, 0, 0, time.UTC), Valid: true} + nd2 := sql.NullTime{Time: time.Date(2006, 03, 04, 0, 0, 0, 0, time.UTC), Valid: true} + ndNULL := sql.NullTime{Time: time.Time{}, Valid: false} + bNULL := []byte(nil) + nsNULL := sql.NullString{String: "", Valid: false} + // Helper function to build NullString from string literal. + ns := func(s string) sql.NullString { return sql.NullString{String: s, Valid: true} } + ns0 := ns("0") + b0 := []byte("0") + b42 := []byte("42") + nsTest := ns("Test") + bTest := []byte("Test") + b0pad4 := []byte("0\x00\x00\x00") // BINARY right-pads values with 0x00 + bx0 := []byte("\x00") + bx42 := []byte("\x42") + + var columns = []struct { + name string + fieldType string // type used when creating table schema + databaseTypeName string // actual type used by MySQL + scanType reflect.Type + nullable bool + precision int64 // 0 if not ok + scale int64 + valuesIn [3]string + valuesOut [3]any + }{ + {"bit8null", "BIT(8)", "BIT", scanTypeBytes, true, 0, 0, [3]string{"0x0", "NULL", "0x42"}, [3]any{bx0, bNULL, bx42}}, + {"boolnull", "BOOL", "TINYINT", scanTypeNullInt, true, 0, 0, [3]string{"NULL", "true", "0"}, [3]any{niNULL, ni1, ni0}}, + {"bool", "BOOL NOT NULL", "TINYINT", scanTypeInt8, false, 0, 0, [3]string{"1", "0", "FALSE"}, [3]any{int8(1), int8(0), int8(0)}}, + {"intnull", "INTEGER", "INT", scanTypeNullInt, true, 0, 0, [3]string{"0", "NULL", "42"}, [3]any{ni0, niNULL, ni42}}, + {"smallint", "SMALLINT NOT NULL", "SMALLINT", scanTypeInt16, false, 0, 0, [3]string{"0", "-32768", "32767"}, [3]any{int16(0), int16(-32768), int16(32767)}}, + {"smallintnull", "SMALLINT", "SMALLINT", scanTypeNullInt, true, 0, 0, [3]string{"0", "NULL", "42"}, [3]any{ni0, niNULL, ni42}}, + {"int3null", "INT(3)", "INT", scanTypeNullInt, true, 0, 0, [3]string{"0", "NULL", "42"}, [3]any{ni0, niNULL, ni42}}, + {"int7", "INT(7) NOT NULL", "INT", scanTypeInt32, false, 0, 0, [3]string{"0", "-1337", "42"}, [3]any{int32(0), int32(-1337), int32(42)}}, + {"mediumintnull", "MEDIUMINT", "MEDIUMINT", scanTypeNullInt, true, 0, 0, [3]string{"0", "42", "NULL"}, [3]any{ni0, ni42, niNULL}}, + {"bigint", "BIGINT NOT NULL", "BIGINT", scanTypeInt64, false, 0, 0, [3]string{"0", "65535", "-42"}, [3]any{int64(0), int64(65535), int64(-42)}}, + {"bigintnull", "BIGINT", "BIGINT", scanTypeNullInt, true, 0, 0, [3]string{"NULL", "1", "42"}, [3]any{niNULL, ni1, ni42}}, + {"tinyuint", "TINYINT UNSIGNED NOT NULL", "UNSIGNED TINYINT", scanTypeUint8, false, 0, 0, [3]string{"0", "255", "42"}, [3]any{uint8(0), uint8(255), uint8(42)}}, + {"smalluint", "SMALLINT UNSIGNED NOT NULL", "UNSIGNED SMALLINT", scanTypeUint16, false, 0, 0, [3]string{"0", "65535", "42"}, [3]any{uint16(0), uint16(65535), uint16(42)}}, + {"biguint", "BIGINT UNSIGNED NOT NULL", "UNSIGNED BIGINT", scanTypeUint64, false, 0, 0, [3]string{"0", "65535", "42"}, [3]any{uint64(0), uint64(65535), uint64(42)}}, + {"mediumuint", "MEDIUMINT UNSIGNED NOT NULL", "UNSIGNED MEDIUMINT", scanTypeUint32, false, 0, 0, [3]string{"0", "16777215", "42"}, [3]any{uint32(0), uint32(16777215), uint32(42)}}, + {"uint13", "INT(13) UNSIGNED NOT NULL", "UNSIGNED INT", scanTypeUint32, false, 0, 0, [3]string{"0", "1337", "42"}, [3]any{uint32(0), uint32(1337), uint32(42)}}, + {"float", "FLOAT NOT NULL", "FLOAT", scanTypeFloat32, false, math.MaxInt64, math.MaxInt64, [3]string{"0", "42", "13.37"}, [3]any{float32(0), float32(42), float32(13.37)}}, + {"floatnull", "FLOAT", "FLOAT", scanTypeNullFloat, true, math.MaxInt64, math.MaxInt64, [3]string{"0", "NULL", "13.37"}, [3]any{nf0, nfNULL, nf1337}}, + {"float74null", "FLOAT(7,4)", "FLOAT", scanTypeNullFloat, true, math.MaxInt64, 4, [3]string{"0", "NULL", "13.37"}, [3]any{nf0, nfNULL, nf1337}}, + {"double", "DOUBLE NOT NULL", "DOUBLE", scanTypeFloat64, false, math.MaxInt64, math.MaxInt64, [3]string{"0", "42", "13.37"}, [3]any{float64(0), float64(42), float64(13.37)}}, + {"doublenull", "DOUBLE", "DOUBLE", scanTypeNullFloat, true, math.MaxInt64, math.MaxInt64, [3]string{"0", "NULL", "13.37"}, [3]any{nf0, nfNULL, nf1337}}, + {"decimal1", "DECIMAL(10,6) NOT NULL", "DECIMAL", scanTypeString, false, 10, 6, [3]string{"0", "13.37", "1234.123456"}, [3]any{"0.000000", "13.370000", "1234.123456"}}, + {"decimal1null", "DECIMAL(10,6)", "DECIMAL", scanTypeNullString, true, 10, 6, [3]string{"0", "NULL", "1234.123456"}, [3]any{ns("0.000000"), nsNULL, ns("1234.123456")}}, + {"decimal2", "DECIMAL(8,4) NOT NULL", "DECIMAL", scanTypeString, false, 8, 4, [3]string{"0", "13.37", "1234.123456"}, [3]any{"0.0000", "13.3700", "1234.1235"}}, + {"decimal2null", "DECIMAL(8,4)", "DECIMAL", scanTypeNullString, true, 8, 4, [3]string{"0", "NULL", "1234.123456"}, [3]any{ns("0.0000"), nsNULL, ns("1234.1235")}}, + {"decimal3", "DECIMAL(5,0) NOT NULL", "DECIMAL", scanTypeString, false, 5, 0, [3]string{"0", "13.37", "-12345.123456"}, [3]any{"0", "13", "-12345"}}, + {"decimal3null", "DECIMAL(5,0)", "DECIMAL", scanTypeNullString, true, 5, 0, [3]string{"0", "NULL", "-12345.123456"}, [3]any{ns0, nsNULL, ns("-12345")}}, + {"char25null", "CHAR(25)", "CHAR", scanTypeNullString, true, 0, 0, [3]string{"0", "NULL", "'Test'"}, [3]any{ns0, nsNULL, nsTest}}, + {"varchar42", "VARCHAR(42) NOT NULL", "VARCHAR", scanTypeString, false, 0, 0, [3]string{"0", "'Test'", "42"}, [3]any{"0", "Test", "42"}}, + {"binary4null", "BINARY(4)", "BINARY", scanTypeBytes, true, 0, 0, [3]string{"0", "NULL", "'Test'"}, [3]any{b0pad4, bNULL, bTest}}, + {"varbinary42", "VARBINARY(42) NOT NULL", "VARBINARY", scanTypeBytes, false, 0, 0, [3]string{"0", "'Test'", "42"}, [3]any{b0, bTest, b42}}, + {"tinyblobnull", "TINYBLOB", "BLOB", scanTypeBytes, true, 0, 0, [3]string{"0", "NULL", "'Test'"}, [3]any{b0, bNULL, bTest}}, + {"tinytextnull", "TINYTEXT", "TEXT", scanTypeNullString, true, 0, 0, [3]string{"0", "NULL", "'Test'"}, [3]any{ns0, nsNULL, nsTest}}, + {"blobnull", "BLOB", "BLOB", scanTypeBytes, true, 0, 0, [3]string{"0", "NULL", "'Test'"}, [3]any{b0, bNULL, bTest}}, + {"textnull", "TEXT", "TEXT", scanTypeNullString, true, 0, 0, [3]string{"0", "NULL", "'Test'"}, [3]any{ns0, nsNULL, nsTest}}, + {"mediumblob", "MEDIUMBLOB NOT NULL", "BLOB", scanTypeBytes, false, 0, 0, [3]string{"0", "'Test'", "42"}, [3]any{b0, bTest, b42}}, + {"mediumtext", "MEDIUMTEXT NOT NULL", "TEXT", scanTypeString, false, 0, 0, [3]string{"0", "'Test'", "42"}, [3]any{"0", "Test", "42"}}, + {"longblob", "LONGBLOB NOT NULL", "BLOB", scanTypeBytes, false, 0, 0, [3]string{"0", "'Test'", "42"}, [3]any{b0, bTest, b42}}, + {"longtext", "LONGTEXT NOT NULL", "TEXT", scanTypeString, false, 0, 0, [3]string{"0", "'Test'", "42"}, [3]any{"0", "Test", "42"}}, + {"datetime", "DATETIME", "DATETIME", scanTypeNullTime, true, 0, 0, [3]string{"'2006-01-02 15:04:05'", "'2006-01-02 15:04:05.1'", "'2006-01-02 15:04:05.111111'"}, [3]any{nt0, nt0, nt0}}, + {"datetime2", "DATETIME(2)", "DATETIME", scanTypeNullTime, true, 2, 2, [3]string{"'2006-01-02 15:04:05'", "'2006-01-02 15:04:05.1'", "'2006-01-02 15:04:05.111111'"}, [3]any{nt0, nt1, nt2}}, + {"datetime6", "DATETIME(6)", "DATETIME", scanTypeNullTime, true, 6, 6, [3]string{"'2006-01-02 15:04:05'", "'2006-01-02 15:04:05.1'", "'2006-01-02 15:04:05.111111'"}, [3]any{nt0, nt1, nt6}}, + {"date", "DATE", "DATE", scanTypeNullTime, true, 0, 0, [3]string{"'2006-01-02'", "NULL", "'2006-03-04'"}, [3]any{nd1, ndNULL, nd2}}, + {"year", "YEAR NOT NULL", "YEAR", scanTypeUint16, false, 0, 0, [3]string{"2006", "2000", "1994"}, [3]any{uint16(2006), uint16(2000), uint16(1994)}}, + {"enum", "ENUM('', 'v1', 'v2')", "ENUM", scanTypeNullString, true, 0, 0, [3]string{"''", "'v1'", "'v2'"}, [3]any{ns(""), ns("v1"), ns("v2")}}, + {"set", "set('', 'v1', 'v2')", "SET", scanTypeNullString, true, 0, 0, [3]string{"''", "'v1'", "'v1,v2'"}, [3]any{ns(""), ns("v1"), ns("v1,v2")}}, + } + + schema := "" + values1 := "" + values2 := "" + values3 := "" + for _, column := range columns { + schema += fmt.Sprintf("`%s` %s, ", column.name, column.fieldType) + values1 += column.valuesIn[0] + ", " + values2 += column.valuesIn[1] + ", " + values3 += column.valuesIn[2] + ", " + } + schema = schema[:len(schema)-2] + values1 = values1[:len(values1)-2] + values2 = values2[:len(values2)-2] + values3 = values3[:len(values3)-2] + + runTests(t, dsn+"&parseTime=true", func(dbt *DBTest) { + dbt.mustExec("CREATE TABLE test (" + schema + ")") + dbt.mustExec("INSERT INTO test VALUES (" + values1 + "), (" + values2 + "), (" + values3 + ")") + + rows, err := dbt.db.Query("SELECT * FROM test") + if err != nil { + t.Fatalf("Query: %v", err) + } + + tt, err := rows.ColumnTypes() + if err != nil { + t.Fatalf("ColumnTypes: %v", err) + } + + if len(tt) != len(columns) { + t.Fatalf("unexpected number of columns: expected %d, got %d", len(columns), len(tt)) + } + + types := make([]reflect.Type, len(tt)) + for i, tp := range tt { + column := columns[i] + + // Name + name := tp.Name() + if name != column.name { + t.Errorf("column name mismatch %s != %s", name, column.name) + continue + } + + // DatabaseTypeName + databaseTypeName := tp.DatabaseTypeName() + if databaseTypeName != column.databaseTypeName { + t.Errorf("databasetypename name mismatch for column %q: %s != %s", name, databaseTypeName, column.databaseTypeName) + continue + } + + // ScanType + scanType := tp.ScanType() + if scanType != column.scanType { + if scanType == nil { + t.Errorf("scantype is null for column %q", name) + } else { + t.Errorf("scantype mismatch for column %q: %s != %s", name, scanType.Name(), column.scanType.Name()) + } + continue + } + types[i] = scanType + + // Nullable + nullable, ok := tp.Nullable() + if !ok { + t.Errorf("nullable not ok %q", name) + continue + } + if nullable != column.nullable { + t.Errorf("nullable mismatch for column %q: %t != %t", name, nullable, column.nullable) + } + + // Length + // length, ok := tp.Length() + // if length != column.length { + // if !ok { + // t.Errorf("length not ok for column %q", name) + // } else { + // t.Errorf("length mismatch for column %q: %d != %d", name, length, column.length) + // } + // continue + // } + + // Precision and Scale + precision, scale, ok := tp.DecimalSize() + if precision != column.precision { + if !ok { + t.Errorf("precision not ok for column %q", name) + } else { + t.Errorf("precision mismatch for column %q: %d != %d", name, precision, column.precision) + } + continue + } + if scale != column.scale { + if !ok { + t.Errorf("scale not ok for column %q", name) + } else { + t.Errorf("scale mismatch for column %q: %d != %d", name, scale, column.scale) + } + continue + } + } + // Avoid panic caused by nil scantype. + if t.Failed() { + return + } + values := make([]any, len(tt)) + for i := range values { + values[i] = reflect.New(types[i]).Interface() + } + i := 0 + for rows.Next() { + err = rows.Scan(values...) + if err != nil { + t.Fatalf("failed to scan values in %v", err) + } + for j, value := range values { + value := reflect.ValueOf(value).Elem().Interface() + if !reflect.DeepEqual(value, columns[j].valuesOut[i]) { + t.Errorf("row %d, column %d: %v != %v", i, j, value, columns[j].valuesOut[i]) + } + } + i++ + } + if i != 3 { + t.Errorf("expected 3 rows, got %d", i) + } + + if err := rows.Close(); err != nil { + t.Errorf("error closing rows: %s", err) + } + }) +} + +func TestValuerWithValueReceiverGivenNilValue(t *testing.T) { + runTestsParallel(t, dsn, func(dbt *DBTest, tbl string) { + dbt.mustExec("CREATE TABLE " + tbl + " (value VARCHAR(255))") + dbt.db.Exec("INSERT INTO "+tbl+" VALUES (?)", (*testValuer)(nil)) + // This test will panic on the INSERT if ConvertValue() does not check for typed nil before calling Value() + }) +} + +// TestRawBytesAreNotModified checks for a race condition that arises when a query context +// is canceled while a user is calling rows.Scan. This is a more stringent test than the one +// proposed in https://github.com/golang/go/issues/23519. Here we're explicitly using +// `sql.RawBytes` to check the contents of our internal buffers are not modified after an implicit +// call to `Rows.Close`, so Context cancellation should **not** invalidate the backing buffers. +func TestRawBytesAreNotModified(t *testing.T) { + const blob = "abcdefghijklmnop" + const contextRaceIterations = 20 + const blobSize = defaultBufSize * 3 / 4 // Second row overwrites first row. + const insertRows = 4 + + var sqlBlobs = [2]string{ + strings.Repeat(blob, blobSize/len(blob)), + strings.Repeat(strings.ToUpper(blob), blobSize/len(blob)), + } + + runTests(t, dsn, func(dbt *DBTest) { + dbt.mustExec("CREATE TABLE test (id int, value BLOB) CHARACTER SET utf8") + for i := range insertRows { + dbt.mustExec("INSERT INTO test VALUES (?, ?)", i+1, sqlBlobs[i&1]) + } + + for i := range contextRaceIterations { + func() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + rows, err := dbt.db.QueryContext(ctx, `SELECT id, value FROM test`) + if err != nil { + dbt.Fatal(err) + } + defer rows.Close() + + var b int + var raw sql.RawBytes + if !rows.Next() { + dbt.Fatal("expected at least one row") + } + if err := rows.Scan(&b, &raw); err != nil { + dbt.Fatal(err) + } + + before := string(raw) + // Ensure cancelling the query does not corrupt the contents of `raw` + cancel() + time.Sleep(time.Microsecond * 100) + after := string(raw) + + if before != after { + dbt.Fatalf("the backing storage for sql.RawBytes has been modified (i=%v)", i) + } + }() + } + }) +} + +var _ driver.DriverContext = &MySQLDriver{} + +type dialCtxKey struct{} + +func TestConnectorObeysDialTimeouts(t *testing.T) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + RegisterDialContext("dialctxtest", func(ctx context.Context, addr string) (net.Conn, error) { + var d net.Dialer + if !ctx.Value(dialCtxKey{}).(bool) { + return nil, fmt.Errorf("test error: query context is not propagated to our dialer") + } + return d.DialContext(ctx, prot, addr) + }) + + db, err := sql.Open(driverNameTest, fmt.Sprintf("%s:%s@dialctxtest(%s)/%s?timeout=30s", user, pass, addr, dbname)) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + defer db.Close() + + ctx := context.WithValue(context.Background(), dialCtxKey{}, true) + + _, err = db.ExecContext(ctx, "DO 1") + if err != nil { + t.Fatal(err) + } +} + +func configForTests(t *testing.T) *Config { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + mycnf := NewConfig() + mycnf.User = user + mycnf.Passwd = pass + mycnf.Addr = addr + mycnf.Net = prot + mycnf.DBName = dbname + return mycnf +} + +func TestNewConnector(t *testing.T) { + mycnf := configForTests(t) + conn, err := NewConnector(mycnf) + if err != nil { + t.Fatal(err) + } + + db := sql.OpenDB(conn) + defer db.Close() + + if err := db.Ping(); err != nil { + t.Fatal(err) + } +} + +type slowConnection struct { + net.Conn + slowdown time.Duration +} + +func (sc *slowConnection) Read(b []byte) (int, error) { + time.Sleep(sc.slowdown) + return sc.Conn.Read(b) +} + +type connectorHijack struct { + driver.Connector + connErr error +} + +func (cw *connectorHijack) Connect(ctx context.Context) (driver.Conn, error) { + var conn driver.Conn + conn, cw.connErr = cw.Connector.Connect(ctx) + return conn, cw.connErr +} + +func TestConnectorTimeoutsDuringOpen(t *testing.T) { + RegisterDialContext("slowconn", func(ctx context.Context, addr string) (net.Conn, error) { + var d net.Dialer + conn, err := d.DialContext(ctx, prot, addr) + if err != nil { + return nil, err + } + return &slowConnection{Conn: conn, slowdown: 100 * time.Millisecond}, nil + }) + + mycnf := configForTests(t) + mycnf.Net = "slowconn" + + conn, err := NewConnector(mycnf) + if err != nil { + t.Fatal(err) + } + + hijack := &connectorHijack{Connector: conn} + + db := sql.OpenDB(hijack) + defer db.Close() + + ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) + defer cancel() + + _, err = db.ExecContext(ctx, "DO 1") + if err != context.DeadlineExceeded { + t.Fatalf("ExecContext should have timed out") + } + if hijack.connErr != context.DeadlineExceeded { + t.Fatalf("(*Connector).Connect should have timed out") + } +} + +// A connection which can only be closed. +type dummyConnection struct { + net.Conn + closed bool +} + +func (d *dummyConnection) Close() error { + d.closed = true + return nil +} + +func TestConnectorTimeoutsWatchCancel(t *testing.T) { + var ( + cancel func() // Used to cancel the context just after connecting. + created *dummyConnection // The created connection. + ) + + RegisterDialContext("TestConnectorTimeoutsWatchCancel", func(ctx context.Context, addr string) (net.Conn, error) { + // Canceling at this time triggers the watchCancel error branch in Connect(). + cancel() + created = &dummyConnection{} + return created, nil + }) + + mycnf := NewConfig() + mycnf.User = "root" + mycnf.Addr = "foo" + mycnf.Net = "TestConnectorTimeoutsWatchCancel" + + conn, err := NewConnector(mycnf) + if err != nil { + t.Fatal(err) + } + + db := sql.OpenDB(conn) + defer db.Close() + + var ctx context.Context + ctx, cancel = context.WithCancel(context.Background()) + defer cancel() + + if _, err := db.Conn(ctx); err != context.Canceled { + t.Errorf("got %v, want context.Canceled", err) + } + + if created == nil { + t.Fatal("no connection created") + } + if !created.closed { + t.Errorf("connection not closed") + } +} + +func TestConnectionAttributes(t *testing.T) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + + defaultAttrs := []string{ + connAttrClientName, + connAttrOS, + connAttrPlatform, + connAttrPid, + connAttrServerHost, + } + host, _, _ := net.SplitHostPort(addr) + defaultAttrValues := []string{ + connAttrClientNameValue, + connAttrOSValue, + connAttrPlatformValue, + strconv.Itoa(os.Getpid()), + host, + } + + customAttrs := []string{"attr1", "fo/o"} + customAttrValues := []string{"value1", "bo/o"} + + customAttrStrs := make([]string, len(customAttrs)) + for i := range customAttrs { + customAttrStrs[i] = fmt.Sprintf("%s:%s", customAttrs[i], customAttrValues[i]) + } + dsn += "&connectionAttributes=" + url.QueryEscape(strings.Join(customAttrStrs, ",")) + + var db *sql.DB + if _, err := ParseDSN(dsn); err != errInvalidDSNUnsafeCollation { + db, err = sql.Open(driverNameTest, dsn) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + defer db.Close() + } + + dbt := &DBTest{t, db} + + var varName string + var varValue string + err := dbt.db.QueryRow("SHOW VARIABLES LIKE 'performance_schema'").Scan(&varName, &varValue) + if err != nil { + t.Fatalf("error: %s", err.Error()) + } + if varValue != "ON" { + t.Skipf("Performance schema is not enabled. skipping") + } + queryString := "SELECT ATTR_NAME, ATTR_VALUE FROM performance_schema.session_account_connect_attrs WHERE PROCESSLIST_ID = CONNECTION_ID()" + rows := dbt.mustQuery(queryString) + defer rows.Close() + + rowsMap := make(map[string]string) + for rows.Next() { + var attrName, attrValue string + rows.Scan(&attrName, &attrValue) + rowsMap[attrName] = attrValue + } + + connAttrs := slices.Concat(defaultAttrs, customAttrs) + expectedAttrValues := slices.Concat(defaultAttrValues, customAttrValues) + for i := range connAttrs { + if gotValue := rowsMap[connAttrs[i]]; gotValue != expectedAttrValues[i] { + dbt.Errorf("expected %q, got %q", expectedAttrValues[i], gotValue) + } + } +} + +func TestErrorInMultiResult(t *testing.T) { + if !available { + t.Skipf("MySQL server not running on %s", netAddr) + } + // https://github.com/go-sql-driver/mysql/issues/1361 + var db *sql.DB + if _, err := ParseDSN(dsn); err != errInvalidDSNUnsafeCollation { + db, err = sql.Open("mysql", dsn) + if err != nil { + t.Fatalf("error connecting: %s", err.Error()) + } + defer db.Close() + } + + dbt := &DBTest{t, db} + query := ` +CREATE PROCEDURE test_proc1() +BEGIN + SELECT 1,2; + SELECT 3,4; + SIGNAL SQLSTATE '10000' SET MESSAGE_TEXT = "some error", MYSQL_ERRNO = 10000; +END; +` + runCallCommand(dbt, query, "test_proc1") +} + +func runCallCommand(dbt *DBTest, query, name string) { + dbt.mustExec(fmt.Sprintf("DROP PROCEDURE IF EXISTS %s", name)) + dbt.mustExec(query) + defer dbt.mustExec("DROP PROCEDURE " + name) + rows, err := dbt.db.Query(fmt.Sprintf("CALL %s", name)) + if err != nil { + return + } + defer rows.Close() + + for rows.Next() { + } + for rows.NextResultSet() { + for rows.Next() { + } + } +} + +func TestIssue1567(t *testing.T) { + // enable TLS. + runTests(t, dsn+"&tls=skip-verify", func(dbt *DBTest) { + var max int + err := dbt.db.QueryRow("SELECT @@max_connections").Scan(&max) + if err != nil { + dbt.Fatalf("%s", err.Error()) + } + + // disable connection pooling. + // data race happens when new connection is created. + dbt.db.SetMaxIdleConns(0) + + // estimate round trip time. + start := time.Now() + if err := dbt.db.PingContext(context.Background()); err != nil { + t.Fatal(err) + } + rtt := time.Since(start) + if rtt <= 0 { + // In some environments, rtt may become 0, so set it to at least 1ms. + rtt = time.Millisecond + } + + count := 1000 + if testing.Short() { + count = 10 + } + if count > max { + count = max + } + + for range count { + timeout := time.Duration(mrand.Int63n(int64(rtt))) + ctx, cancel := context.WithTimeout(context.Background(), timeout) + dbt.db.PingContext(ctx) + cancel() + } + }) +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/dsn.go b/_LICENSES/github.com/go-sql-driver/mysql/dsn.go new file mode 100644 index 0000000000..41463a503d --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/dsn.go @@ -0,0 +1,706 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2016 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "context" + "crypto/rsa" + "crypto/tls" + "errors" + "fmt" + "maps" + "math/big" + "net" + "net/url" + "sort" + "strconv" + "strings" + "time" +) + +var ( + errInvalidDSNUnescaped = errors.New("invalid DSN: did you forget to escape a param value?") + errInvalidDSNAddr = errors.New("invalid DSN: network address not terminated (missing closing brace)") + errInvalidDSNNoSlash = errors.New("invalid DSN: missing the slash separating the database name") + errInvalidDSNUnsafeCollation = errors.New("invalid DSN: interpolateParams can not be used with unsafe collations") +) + +// Config is a configuration parsed from a DSN string. +// If a new Config is created instead of being parsed from a DSN string, +// the NewConfig function should be used, which sets default values. +type Config struct { + // non boolean fields + + User string // Username + Passwd string // Password (requires User) + Net string // Network (e.g. "tcp", "tcp6", "unix". default: "tcp") + Addr string // Address (default: "127.0.0.1:3306" for "tcp" and "/tmp/mysql.sock" for "unix") + DBName string // Database name + Params map[string]string // Connection parameters + ConnectionAttributes string // Connection Attributes, comma-delimited string of user-defined "key:value" pairs + Collation string // Connection collation. When set, this will be set in SET NAMES COLLATE query + Loc *time.Location // Location for time.Time values + MaxAllowedPacket int // Max packet size allowed + ServerPubKey string // Server public key name + TLSConfig string // TLS configuration name + TLS *tls.Config // TLS configuration, its priority is higher than TLSConfig + Timeout time.Duration // Dial timeout + ReadTimeout time.Duration // I/O read timeout + WriteTimeout time.Duration // I/O write timeout + Logger Logger // Logger + // DialFunc specifies the dial function for creating connections + DialFunc func(ctx context.Context, network, addr string) (net.Conn, error) + + // boolean fields + + AllowAllFiles bool // Allow all files to be used with LOAD DATA LOCAL INFILE + AllowCleartextPasswords bool // Allows the cleartext client side plugin + AllowFallbackToPlaintext bool // Allows fallback to unencrypted connection if server does not support TLS + AllowNativePasswords bool // Allows the native password authentication method + AllowOldPasswords bool // Allows the old insecure password method + CheckConnLiveness bool // Check connections for liveness before using them + ClientFoundRows bool // Return number of matching rows instead of rows changed + ColumnsWithAlias bool // Prepend table alias to column names + InterpolateParams bool // Interpolate placeholders into query string + MultiStatements bool // Allow multiple statements in one query + ParseTime bool // Parse time values to time.Time + RejectReadOnly bool // Reject read-only connections + + // unexported fields. new options should be come here. + // boolean first. alphabetical order. + + compress bool // Enable zlib compression + + beforeConnect func(context.Context, *Config) error // Invoked before a connection is established + pubKey *rsa.PublicKey // Server public key + timeTruncate time.Duration // Truncate time.Time values to the specified duration + charsets []string // Connection charset. When set, this will be set in SET NAMES query +} + +// Functional Options Pattern +// https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis +type Option func(*Config) error + +// NewConfig creates a new Config and sets default values. +func NewConfig() *Config { + cfg := &Config{ + Loc: time.UTC, + MaxAllowedPacket: defaultMaxAllowedPacket, + Logger: defaultLogger, + AllowNativePasswords: true, + CheckConnLiveness: true, + } + return cfg +} + +// Apply applies the given options to the Config object. +func (c *Config) Apply(opts ...Option) error { + for _, opt := range opts { + err := opt(c) + if err != nil { + return err + } + } + return nil +} + +// TimeTruncate sets the time duration to truncate time.Time values in +// query parameters. +func TimeTruncate(d time.Duration) Option { + return func(cfg *Config) error { + cfg.timeTruncate = d + return nil + } +} + +// BeforeConnect sets the function to be invoked before a connection is established. +func BeforeConnect(fn func(context.Context, *Config) error) Option { + return func(cfg *Config) error { + cfg.beforeConnect = fn + return nil + } +} + +// EnableCompress sets the compression mode. +func EnableCompression(yes bool) Option { + return func(cfg *Config) error { + cfg.compress = yes + return nil + } +} + +// Charset sets the connection charset and collation. +// +// charset is the connection charset. +// collation is the connection collation. It can be null or empty string. +// +// When collation is not specified, `SET NAMES ` command is sent when the connection is established. +// When collation is specified, `SET NAMES COLLATE ` command is sent when the connection is established. +func Charset(charset, collation string) Option { + return func(cfg *Config) error { + cfg.charsets = []string{charset} + cfg.Collation = collation + return nil + } +} + +func (cfg *Config) Clone() *Config { + cp := *cfg + if cp.TLS != nil { + cp.TLS = cfg.TLS.Clone() + } + if len(cp.Params) > 0 { + cp.Params = make(map[string]string, len(cfg.Params)) + maps.Copy(cp.Params, cfg.Params) + } + if cfg.pubKey != nil { + cp.pubKey = &rsa.PublicKey{ + N: new(big.Int).Set(cfg.pubKey.N), + E: cfg.pubKey.E, + } + } + return &cp +} + +func (cfg *Config) normalize() error { + if cfg.InterpolateParams && cfg.Collation != "" && unsafeCollations[cfg.Collation] { + return errInvalidDSNUnsafeCollation + } + + // Set default network if empty + if cfg.Net == "" { + cfg.Net = "tcp" + } + + // Set default address if empty + if cfg.Addr == "" { + switch cfg.Net { + case "tcp": + cfg.Addr = "127.0.0.1:3306" + case "unix": + cfg.Addr = "/tmp/mysql.sock" + default: + return errors.New("default addr for network '" + cfg.Net + "' unknown") + } + } else if cfg.Net == "tcp" { + cfg.Addr = ensureHavePort(cfg.Addr) + } + + if cfg.TLS == nil { + switch cfg.TLSConfig { + case "false", "": + // don't set anything + case "true": + cfg.TLS = &tls.Config{} + case "skip-verify": + cfg.TLS = &tls.Config{InsecureSkipVerify: true} + case "preferred": + cfg.TLS = &tls.Config{InsecureSkipVerify: true} + cfg.AllowFallbackToPlaintext = true + default: + cfg.TLS = getTLSConfigClone(cfg.TLSConfig) + if cfg.TLS == nil { + return errors.New("invalid value / unknown config name: " + cfg.TLSConfig) + } + } + } + + if cfg.TLS != nil && cfg.TLS.ServerName == "" && !cfg.TLS.InsecureSkipVerify { + host, _, err := net.SplitHostPort(cfg.Addr) + if err == nil { + cfg.TLS.ServerName = host + } + } + + if cfg.ServerPubKey != "" { + cfg.pubKey = getServerPubKey(cfg.ServerPubKey) + if cfg.pubKey == nil { + return errors.New("invalid value / unknown server pub key name: " + cfg.ServerPubKey) + } + } + + if cfg.Logger == nil { + cfg.Logger = defaultLogger + } + + return nil +} + +func writeDSNParam(buf *bytes.Buffer, hasParam *bool, name, value string) { + buf.Grow(1 + len(name) + 1 + len(value)) + if !*hasParam { + *hasParam = true + buf.WriteByte('?') + } else { + buf.WriteByte('&') + } + buf.WriteString(name) + buf.WriteByte('=') + buf.WriteString(value) +} + +// FormatDSN formats the given Config into a DSN string which can be passed to +// the driver. +// +// Note: use [NewConnector] and [database/sql.OpenDB] to open a connection from a [*Config]. +func (cfg *Config) FormatDSN() string { + var buf bytes.Buffer + + // [username[:password]@] + if len(cfg.User) > 0 { + buf.WriteString(cfg.User) + if len(cfg.Passwd) > 0 { + buf.WriteByte(':') + buf.WriteString(cfg.Passwd) + } + buf.WriteByte('@') + } + + // [protocol[(address)]] + if len(cfg.Addr) > 0 { + net := cfg.Net + if net == "" { + net = "tcp" + } + buf.WriteString(net) + buf.WriteByte('(') + buf.WriteString(cfg.Addr) + buf.WriteByte(')') + } else if cfg.Net != "" && cfg.Net != "tcp" { + // Preserve an explicit non-default protocol when there's no + // address, so e.g. Net="unix" still round-trips. + buf.WriteString(cfg.Net) + } + + // /dbname + buf.WriteByte('/') + buf.WriteString(url.PathEscape(cfg.DBName)) + + // [?param1=value1&...¶mN=valueN] + hasParam := false + + if cfg.AllowAllFiles { + hasParam = true + buf.WriteString("?allowAllFiles=true") + } + + if cfg.AllowCleartextPasswords { + writeDSNParam(&buf, &hasParam, "allowCleartextPasswords", "true") + } + + if cfg.AllowFallbackToPlaintext { + writeDSNParam(&buf, &hasParam, "allowFallbackToPlaintext", "true") + } + + if !cfg.AllowNativePasswords { + writeDSNParam(&buf, &hasParam, "allowNativePasswords", "false") + } + + if cfg.AllowOldPasswords { + writeDSNParam(&buf, &hasParam, "allowOldPasswords", "true") + } + + if !cfg.CheckConnLiveness { + writeDSNParam(&buf, &hasParam, "checkConnLiveness", "false") + } + + if cfg.ClientFoundRows { + writeDSNParam(&buf, &hasParam, "clientFoundRows", "true") + } + + if charsets := cfg.charsets; len(charsets) > 0 { + writeDSNParam(&buf, &hasParam, "charset", strings.Join(charsets, ",")) + } + + if col := cfg.Collation; col != "" { + writeDSNParam(&buf, &hasParam, "collation", col) + } + + if cfg.ColumnsWithAlias { + writeDSNParam(&buf, &hasParam, "columnsWithAlias", "true") + } + + if cfg.ConnectionAttributes != "" { + writeDSNParam(&buf, &hasParam, "connectionAttributes", url.QueryEscape(cfg.ConnectionAttributes)) + } + + if cfg.compress { + writeDSNParam(&buf, &hasParam, "compress", "true") + } + + if cfg.InterpolateParams { + writeDSNParam(&buf, &hasParam, "interpolateParams", "true") + } + + if cfg.Loc != time.UTC && cfg.Loc != nil { + writeDSNParam(&buf, &hasParam, "loc", url.QueryEscape(cfg.Loc.String())) + } + + if cfg.MultiStatements { + writeDSNParam(&buf, &hasParam, "multiStatements", "true") + } + + if cfg.ParseTime { + writeDSNParam(&buf, &hasParam, "parseTime", "true") + } + + if cfg.timeTruncate > 0 { + writeDSNParam(&buf, &hasParam, "timeTruncate", cfg.timeTruncate.String()) + } + + if cfg.ReadTimeout > 0 { + writeDSNParam(&buf, &hasParam, "readTimeout", cfg.ReadTimeout.String()) + } + + if cfg.RejectReadOnly { + writeDSNParam(&buf, &hasParam, "rejectReadOnly", "true") + } + + if len(cfg.ServerPubKey) > 0 { + writeDSNParam(&buf, &hasParam, "serverPubKey", url.QueryEscape(cfg.ServerPubKey)) + } + + if cfg.Timeout > 0 { + writeDSNParam(&buf, &hasParam, "timeout", cfg.Timeout.String()) + } + + if len(cfg.TLSConfig) > 0 { + writeDSNParam(&buf, &hasParam, "tls", url.QueryEscape(cfg.TLSConfig)) + } + + if cfg.WriteTimeout > 0 { + writeDSNParam(&buf, &hasParam, "writeTimeout", cfg.WriteTimeout.String()) + } + + if cfg.MaxAllowedPacket != defaultMaxAllowedPacket { + writeDSNParam(&buf, &hasParam, "maxAllowedPacket", strconv.Itoa(cfg.MaxAllowedPacket)) + } + + // other params + if cfg.Params != nil { + var params []string + for param := range cfg.Params { + params = append(params, param) + } + sort.Strings(params) + for _, param := range params { + writeDSNParam(&buf, &hasParam, param, url.QueryEscape(cfg.Params[param])) + } + } + + return buf.String() +} + +// ParseDSN parses the DSN string to a Config +func ParseDSN(dsn string) (cfg *Config, err error) { + // New config with some default values + cfg = NewConfig() + + // [user[:password]@][net[(addr)]]/dbname[?param1=value1¶mN=valueN] + // Find the last '/' (since the password or the net addr might contain a '/') + foundSlash := false + for i := len(dsn) - 1; i >= 0; i-- { + if dsn[i] == '/' { + foundSlash = true + var j, k int + + // left part is empty if i <= 0 + if i > 0 { + // [username[:password]@][protocol[(address)]] + // Find the last '@' in dsn[:i] + for j = i; j >= 0; j-- { + if dsn[j] == '@' { + // username[:password] + // Find the first ':' in dsn[:j] + for k = 0; k < j; k++ { // We cannot use k = range j here, because we use dsn[:k] below + if dsn[k] == ':' { + cfg.Passwd = dsn[k+1 : j] + break + } + } + cfg.User = dsn[:k] + + break + } + } + + // [protocol[(address)]] + // Find the first '(' in dsn[j+1:i] + for k = j + 1; k < i; k++ { + if dsn[k] == '(' { + // dsn[i-1] must be == ')' if an address is specified + if dsn[i-1] != ')' { + if strings.ContainsRune(dsn[k+1:i], ')') { + return nil, errInvalidDSNUnescaped + } + return nil, errInvalidDSNAddr + } + cfg.Addr = dsn[k+1 : i-1] + break + } + } + cfg.Net = dsn[j+1 : k] + } + + // dbname[?param1=value1&...¶mN=valueN] + // Find the first '?' in dsn[i+1:] + for j = i + 1; j < len(dsn); j++ { + if dsn[j] == '?' { + if err = parseDSNParams(cfg, dsn[j+1:]); err != nil { + return + } + break + } + } + + dbname := dsn[i+1 : j] + if cfg.DBName, err = url.PathUnescape(dbname); err != nil { + return nil, fmt.Errorf("invalid dbname %q: %w", dbname, err) + } + + break + } + } + + if !foundSlash && len(dsn) > 0 { + return nil, errInvalidDSNNoSlash + } + + if err = cfg.normalize(); err != nil { + return nil, err + } + return +} + +// parseDSNParams parses the DSN "query string" +// Values must be url.QueryEscape'ed +func parseDSNParams(cfg *Config, params string) (err error) { + for v := range strings.SplitSeq(params, "&") { + key, value, found := strings.Cut(v, "=") + if !found { + continue + } + + // cfg params + switch key { + // Disable INFILE allowlist / enable all files + case "allowAllFiles": + var isBool bool + cfg.AllowAllFiles, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Use cleartext authentication mode (MySQL 5.5.10+) + case "allowCleartextPasswords": + var isBool bool + cfg.AllowCleartextPasswords, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Allow fallback to unencrypted connection if server does not support TLS + case "allowFallbackToPlaintext": + var isBool bool + cfg.AllowFallbackToPlaintext, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Use native password authentication + case "allowNativePasswords": + var isBool bool + cfg.AllowNativePasswords, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Use old authentication mode (pre MySQL 4.1) + case "allowOldPasswords": + var isBool bool + cfg.AllowOldPasswords, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Check connections for Liveness before using them + case "checkConnLiveness": + var isBool bool + cfg.CheckConnLiveness, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Switch "rowsAffected" mode + case "clientFoundRows": + var isBool bool + cfg.ClientFoundRows, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // charset + case "charset": + cfg.charsets = strings.Split(value, ",") + + // Collation + case "collation": + cfg.Collation = value + + case "columnsWithAlias": + var isBool bool + cfg.ColumnsWithAlias, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Compression + case "compress": + var isBool bool + cfg.compress, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Enable client side placeholder substitution + case "interpolateParams": + var isBool bool + cfg.InterpolateParams, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Time Location + case "loc": + if value, err = url.QueryUnescape(value); err != nil { + return + } + cfg.Loc, err = time.LoadLocation(value) + if err != nil { + return + } + + // multiple statements in one query + case "multiStatements": + var isBool bool + cfg.MultiStatements, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // time.Time parsing + case "parseTime": + var isBool bool + cfg.ParseTime, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // time.Time truncation + case "timeTruncate": + cfg.timeTruncate, err = time.ParseDuration(value) + if err != nil { + return fmt.Errorf("invalid timeTruncate value: %v, error: %w", value, err) + } + + // I/O read Timeout + case "readTimeout": + cfg.ReadTimeout, err = time.ParseDuration(value) + if err != nil { + return + } + + // Reject read-only connections + case "rejectReadOnly": + var isBool bool + cfg.RejectReadOnly, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Server public key + case "serverPubKey": + name, err := url.QueryUnescape(value) + if err != nil { + return fmt.Errorf("invalid value for server pub key name: %v", err) + } + cfg.ServerPubKey = name + + // Strict mode + case "strict": + panic("strict mode has been removed. See https://github.com/go-sql-driver/mysql/wiki/strict-mode") + + // Dial Timeout + case "timeout": + cfg.Timeout, err = time.ParseDuration(value) + if err != nil { + return + } + + // TLS-Encryption + case "tls": + boolValue, isBool := readBool(value) + if isBool { + if boolValue { + cfg.TLSConfig = "true" + } else { + cfg.TLSConfig = "false" + } + } else if vl := strings.ToLower(value); vl == "skip-verify" || vl == "preferred" { + cfg.TLSConfig = vl + } else { + name, err := url.QueryUnescape(value) + if err != nil { + return fmt.Errorf("invalid value for TLS config name: %v", err) + } + cfg.TLSConfig = name + } + + // I/O write Timeout + case "writeTimeout": + cfg.WriteTimeout, err = time.ParseDuration(value) + if err != nil { + return + } + case "maxAllowedPacket": + cfg.MaxAllowedPacket, err = strconv.Atoi(value) + if err != nil { + return + } + + // Connection attributes + case "connectionAttributes": + connectionAttributes, err := url.QueryUnescape(value) + if err != nil { + return fmt.Errorf("invalid connectionAttributes value: %v", err) + } + cfg.ConnectionAttributes = connectionAttributes + + default: + // lazy init + if cfg.Params == nil { + cfg.Params = make(map[string]string) + } + + if cfg.Params[key], err = url.QueryUnescape(value); err != nil { + return + } + } + } + + return +} + +func ensureHavePort(addr string) string { + if _, _, err := net.SplitHostPort(addr); err != nil { + return net.JoinHostPort(addr, "3306") + } + return addr +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/dsn_fuzz_test.go b/_LICENSES/github.com/go-sql-driver/mysql/dsn_fuzz_test.go new file mode 100644 index 0000000000..c5613332ef --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/dsn_fuzz_test.go @@ -0,0 +1,46 @@ +//go:build go1.18 + +package mysql + +import ( + "net" + "testing" +) + +func FuzzFormatDSN(f *testing.F) { + for _, test := range testDSNs { // See dsn_test.go + f.Add(test.in) + } + + f.Fuzz(func(t *testing.T, dsn1 string) { + // Do not waste resources + if len(dsn1) > 1000 { + t.Skip("ignore: too long") + } + + cfg1, err := ParseDSN(dsn1) + if err != nil { + t.Skipf("invalid DSN: %v", err) + } + + dsn2 := cfg1.FormatDSN() + if dsn2 == dsn1 { + return + } + + // Skip known cases of bad config that are not strictly checked by ParseDSN + if _, _, err := net.SplitHostPort(cfg1.Addr); err != nil { + t.Skipf("invalid addr %q: %v", cfg1.Addr, err) + } + + cfg2, err := ParseDSN(dsn2) + if err != nil { + t.Fatalf("%q rewritten as %q: %v", dsn1, dsn2, err) + } + + dsn3 := cfg2.FormatDSN() + if dsn3 != dsn2 { + t.Errorf("%q rewritten as %q", dsn2, dsn3) + } + }) +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/dsn_test.go b/_LICENSES/github.com/go-sql-driver/mysql/dsn_test.go new file mode 100644 index 0000000000..0c8ac7a048 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/dsn_test.go @@ -0,0 +1,479 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2016 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "crypto/tls" + "fmt" + "net/url" + "reflect" + "testing" + "time" +) + +var testDSNs = []struct { + in string + out *Config +}{{ + "username:password@protocol(address)/dbname?param=value", + &Config{User: "username", Passwd: "password", Net: "protocol", Addr: "address", DBName: "dbname", Params: map[string]string{"param": "value"}, Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "username:password@protocol(address)/dbname?param=value&columnsWithAlias=true", + &Config{User: "username", Passwd: "password", Net: "protocol", Addr: "address", DBName: "dbname", Params: map[string]string{"param": "value"}, Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true, ColumnsWithAlias: true}, +}, { + "username:password@protocol(address)/dbname?param=value&columnsWithAlias=true&multiStatements=true", + &Config{User: "username", Passwd: "password", Net: "protocol", Addr: "address", DBName: "dbname", Params: map[string]string{"param": "value"}, Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true, ColumnsWithAlias: true, MultiStatements: true}, +}, { + "user@unix(/path/to/socket)/dbname?charset=utf8", + &Config{User: "user", Net: "unix", Addr: "/path/to/socket", DBName: "dbname", charsets: []string{"utf8"}, Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "user:password@tcp(localhost:5555)/dbname?charset=utf8&tls=true", + &Config{User: "user", Passwd: "password", Net: "tcp", Addr: "localhost:5555", DBName: "dbname", charsets: []string{"utf8"}, Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true, TLSConfig: "true"}, +}, { + "user:password@tcp(localhost:5555)/dbname?charset=utf8mb4,utf8&tls=skip-verify", + &Config{User: "user", Passwd: "password", Net: "tcp", Addr: "localhost:5555", DBName: "dbname", charsets: []string{"utf8mb4", "utf8"}, Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true, TLSConfig: "skip-verify"}, +}, { + "user:password@/dbname?loc=UTC&timeout=30s&readTimeout=1s&writeTimeout=1s&allowAllFiles=1&clientFoundRows=true&allowOldPasswords=TRUE&collation=utf8mb4_unicode_ci&maxAllowedPacket=16777216&tls=false&allowCleartextPasswords=true&parseTime=true&rejectReadOnly=true", + &Config{User: "user", Passwd: "password", Net: "tcp", Addr: "127.0.0.1:3306", DBName: "dbname", Collation: "utf8mb4_unicode_ci", Loc: time.UTC, TLSConfig: "false", AllowCleartextPasswords: true, AllowNativePasswords: true, Timeout: 30 * time.Second, ReadTimeout: time.Second, WriteTimeout: time.Second, Logger: defaultLogger, AllowAllFiles: true, AllowOldPasswords: true, CheckConnLiveness: true, ClientFoundRows: true, MaxAllowedPacket: 16777216, ParseTime: true, RejectReadOnly: true}, +}, { + "user:password@/dbname?allowNativePasswords=false&checkConnLiveness=false&maxAllowedPacket=0&allowFallbackToPlaintext=true", + &Config{User: "user", Passwd: "password", Net: "tcp", Addr: "127.0.0.1:3306", DBName: "dbname", Loc: time.UTC, MaxAllowedPacket: 0, Logger: defaultLogger, AllowFallbackToPlaintext: true, AllowNativePasswords: false, CheckConnLiveness: false}, +}, { + "user:p@ss(word)@tcp([de:ad:be:ef::ca:fe]:80)/dbname?loc=Local", + &Config{User: "user", Passwd: "p@ss(word)", Net: "tcp", Addr: "[de:ad:be:ef::ca:fe]:80", DBName: "dbname", Loc: time.Local, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "/dbname", + &Config{Net: "tcp", Addr: "127.0.0.1:3306", DBName: "dbname", Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "/dbname%2Fwithslash", + &Config{Net: "tcp", Addr: "127.0.0.1:3306", DBName: "dbname/withslash", Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "@/", + &Config{Net: "tcp", Addr: "127.0.0.1:3306", Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "/", + &Config{Net: "tcp", Addr: "127.0.0.1:3306", Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "", + &Config{Net: "tcp", Addr: "127.0.0.1:3306", Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "user:p@/ssword@/", + &Config{User: "user", Passwd: "p@/ssword", Net: "tcp", Addr: "127.0.0.1:3306", Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "unix/?arg=%2Fsome%2Fpath.ext", + &Config{Net: "unix", Addr: "/tmp/mysql.sock", Params: map[string]string{"arg": "/some/path.ext"}, Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "tcp(127.0.0.1)/dbname", + &Config{Net: "tcp", Addr: "127.0.0.1:3306", DBName: "dbname", Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "tcp(de:ad:be:ef::ca:fe)/dbname", + &Config{Net: "tcp", Addr: "[de:ad:be:ef::ca:fe]:3306", DBName: "dbname", Loc: time.UTC, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true}, +}, { + "user:password@/dbname?loc=UTC&timeout=30s&parseTime=true&timeTruncate=1h", + &Config{User: "user", Passwd: "password", Net: "tcp", Addr: "127.0.0.1:3306", DBName: "dbname", Loc: time.UTC, Timeout: 30 * time.Second, ParseTime: true, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true, timeTruncate: time.Hour}, +}, { + "foo:bar@tcp(192.168.1.50:3307)/baz?timeout=10s&connectionAttributes=program_name:MySQLGoDriver%2FTest,program_version:1.2.3", + &Config{User: "foo", Passwd: "bar", Net: "tcp", Addr: "192.168.1.50:3307", DBName: "baz", Loc: time.UTC, Timeout: 10 * time.Second, MaxAllowedPacket: defaultMaxAllowedPacket, Logger: defaultLogger, AllowNativePasswords: true, CheckConnLiveness: true, ConnectionAttributes: "program_name:MySQLGoDriver/Test,program_version:1.2.3"}, +}, +} + +func TestDSNParser(t *testing.T) { + for i, tst := range testDSNs { + t.Run(tst.in, func(t *testing.T) { + cfg, err := ParseDSN(tst.in) + if err != nil { + t.Error(err.Error()) + return + } + + // pointer not static + cfg.TLS = nil + + if !reflect.DeepEqual(cfg, tst.out) { + t.Errorf("%d. ParseDSN(%q) mismatch:\ngot %+v\nwant %+v", i, tst.in, cfg, tst.out) + } + }) + } +} + +func TestDSNParserInvalid(t *testing.T) { + var invalidDSNs = []string{ + "@net(addr/", // no closing brace + "@tcp(/", // no closing brace + "tcp(/", // no closing brace + "(/", // no closing brace + "net(addr)//", // unescaped + "User:pass@tcp(1.2.3.4:3306)", // no trailing slash + "net()/", // unknown default addr + "user:pass@tcp(127.0.0.1:3306)/db/name", // invalid dbname + "user:password@/dbname?allowFallbackToPlaintext=PREFERRED", // wrong bool flag + "user:password@/dbname?connectionAttributes=attr1:/unescaped/value", // unescaped + //"/dbname?arg=/some/unescaped/path", + } + + for i, tst := range invalidDSNs { + if _, err := ParseDSN(tst); err == nil { + t.Errorf("invalid DSN #%d. (%s) didn't error!", i, tst) + } + } +} + +func TestDSNReformat(t *testing.T) { + for i, tst := range testDSNs { + t.Run(tst.in, func(t *testing.T) { + dsn1 := tst.in + cfg1, err := ParseDSN(dsn1) + if err != nil { + t.Error(err.Error()) + return + } + cfg1.TLS = nil // pointer not static + res1 := fmt.Sprintf("%+v", cfg1) + + dsn2 := cfg1.FormatDSN() + if dsn2 != dsn1 { + // Just log + t.Logf("%d. %q reformatted as %q", i, dsn1, dsn2) + } + + cfg2, err := ParseDSN(dsn2) + if err != nil { + t.Error(err.Error()) + return + } + cfg2.TLS = nil // pointer not static + res2 := fmt.Sprintf("%+v", cfg2) + + if res1 != res2 { + t.Errorf("%d. %q does not match %q", i, res2, res1) + } + + dsn3 := cfg2.FormatDSN() + if dsn3 != dsn2 { + t.Errorf("%d. %q does not match %q", i, dsn2, dsn3) + } + }) + } +} + +func TestDSNServerPubKey(t *testing.T) { + baseDSN := "User:password@tcp(localhost:5555)/dbname?serverPubKey=" + + RegisterServerPubKey("testKey", testPubKeyRSA) + defer DeregisterServerPubKey("testKey") + + tst := baseDSN + "testKey" + cfg, err := ParseDSN(tst) + if err != nil { + t.Error(err.Error()) + } + + if cfg.ServerPubKey != "testKey" { + t.Errorf("unexpected cfg.ServerPubKey value: %v", cfg.ServerPubKey) + } + if cfg.pubKey != testPubKeyRSA { + t.Error("pub key pointer doesn't match") + } + + // Key is missing + tst = baseDSN + "invalid_name" + cfg, err = ParseDSN(tst) + if err == nil { + t.Errorf("invalid name in DSN (%s) but did not error. Got config: %#v", tst, cfg) + } +} + +func TestDSNServerPubKeyQueryEscape(t *testing.T) { + const name = "&%!:" + dsn := "User:password@tcp(localhost:5555)/dbname?serverPubKey=" + url.QueryEscape(name) + + RegisterServerPubKey(name, testPubKeyRSA) + defer DeregisterServerPubKey(name) + + cfg, err := ParseDSN(dsn) + if err != nil { + t.Error(err.Error()) + } + + if cfg.pubKey != testPubKeyRSA { + t.Error("pub key pointer doesn't match") + } +} + +func TestDSNWithCustomTLS(t *testing.T) { + baseDSN := "User:password@tcp(localhost:5555)/dbname?tls=" + tlsCfg := tls.Config{} + + RegisterTLSConfig("utils_test", &tlsCfg) + defer DeregisterTLSConfig("utils_test") + + // Custom TLS is missing + tst := baseDSN + "invalid_tls" + cfg, err := ParseDSN(tst) + if err == nil { + t.Errorf("invalid custom TLS in DSN (%s) but did not error. Got config: %#v", tst, cfg) + } + + tst = baseDSN + "utils_test" + + // Custom TLS with a server name + name := "foohost" + tlsCfg.ServerName = name + cfg, err = ParseDSN(tst) + + if err != nil { + t.Error(err.Error()) + } else if cfg.TLS.ServerName != name { + t.Errorf("did not get the correct TLS ServerName (%s) parsing DSN (%s).", name, tst) + } + + // Custom TLS without a server name + name = "localhost" + tlsCfg.ServerName = "" + cfg, err = ParseDSN(tst) + + if err != nil { + t.Error(err.Error()) + } else if cfg.TLS.ServerName != name { + t.Errorf("did not get the correct ServerName (%s) parsing DSN (%s).", name, tst) + } else if tlsCfg.ServerName != "" { + t.Errorf("tlsCfg was mutated ServerName (%s) should be empty parsing DSN (%s).", name, tst) + } +} + +func TestDSNTLSConfig(t *testing.T) { + expectedServerName := "example.com" + dsn := "tcp(example.com:1234)/?tls=true" + + cfg, err := ParseDSN(dsn) + if err != nil { + t.Error(err.Error()) + } + if cfg.TLS == nil { + t.Error("cfg.tls should not be nil") + } + if cfg.TLS.ServerName != expectedServerName { + t.Errorf("cfg.tls.ServerName should be %q, got %q (host with port)", expectedServerName, cfg.TLS.ServerName) + } + + dsn = "tcp(example.com)/?tls=true" + cfg, err = ParseDSN(dsn) + if err != nil { + t.Error(err.Error()) + } + if cfg.TLS == nil { + t.Error("cfg.tls should not be nil") + } + if cfg.TLS.ServerName != expectedServerName { + t.Errorf("cfg.tls.ServerName should be %q, got %q (host without port)", expectedServerName, cfg.TLS.ServerName) + } +} + +func TestDSNWithCustomTLSQueryEscape(t *testing.T) { + const configKey = "&%!:" + dsn := "User:password@tcp(localhost:5555)/dbname?tls=" + url.QueryEscape(configKey) + name := "foohost" + tlsCfg := tls.Config{ServerName: name} + + RegisterTLSConfig(configKey, &tlsCfg) + defer DeregisterTLSConfig(configKey) + + cfg, err := ParseDSN(dsn) + + if err != nil { + t.Error(err.Error()) + } else if cfg.TLS.ServerName != name { + t.Errorf("did not get the correct TLS ServerName (%s) parsing DSN (%s).", name, dsn) + } +} + +func TestDSNUnsafeCollation(t *testing.T) { + _, err := ParseDSN("/dbname?collation=gbk_chinese_ci&interpolateParams=true") + if err != errInvalidDSNUnsafeCollation { + t.Errorf("expected %v, got %v", errInvalidDSNUnsafeCollation, err) + } + + _, err = ParseDSN("/dbname?collation=gbk_chinese_ci&interpolateParams=false") + if err != nil { + t.Errorf("expected %v, got %v", nil, err) + } + + _, err = ParseDSN("/dbname?collation=gbk_chinese_ci") + if err != nil { + t.Errorf("expected %v, got %v", nil, err) + } + + _, err = ParseDSN("/dbname?collation=ascii_bin&interpolateParams=true") + if err != nil { + t.Errorf("expected %v, got %v", nil, err) + } + + _, err = ParseDSN("/dbname?collation=latin1_german1_ci&interpolateParams=true") + if err != nil { + t.Errorf("expected %v, got %v", nil, err) + } + + _, err = ParseDSN("/dbname?collation=utf8_general_ci&interpolateParams=true") + if err != nil { + t.Errorf("expected %v, got %v", nil, err) + } + + _, err = ParseDSN("/dbname?collation=utf8mb4_general_ci&interpolateParams=true") + if err != nil { + t.Errorf("expected %v, got %v", nil, err) + } +} + +func TestFormatDSN_NetWithoutAddr(t *testing.T) { + // An explicit non-default Net should still appear in the formatted + // DSN when Addr is empty, so the Config round-trips. + cases := []struct { + name string + net string + want string + }{ + {"unix without addr", "unix", "unix/"}, + // tcp is the default; dropping it is a no-op on parse. + {"tcp without addr", "tcp", "/"}, + {"empty net empty addr", "", "/"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + cfg := NewConfig() + cfg.Net = tc.net + cfg.Addr = "" + if got := cfg.FormatDSN(); got != tc.want { + t.Errorf("FormatDSN() = %q, want %q", got, tc.want) + } + }) + } +} + +func TestFormatDSN_AddrWithoutNet(t *testing.T) { + // Direct check of the bug from #1616: an Addr-only Config should + // format with the default tcp protocol so it round-trips. + cfg := NewConfig() + cfg.Addr = "myhost:3306" + got := cfg.FormatDSN() + want := "tcp(myhost:3306)/" + if got != want { + t.Errorf("FormatDSN() = %q, want %q", got, want) + } +} + +func TestParamsAreSorted(t *testing.T) { + expected := "/dbname?interpolateParams=true&foobar=baz&quux=loo" + cfg := NewConfig() + cfg.DBName = "dbname" + cfg.InterpolateParams = true + cfg.Params = map[string]string{ + "quux": "loo", + "foobar": "baz", + } + actual := cfg.FormatDSN() + if actual != expected { + t.Errorf("generic Config.Params were not sorted: want %#v, got %#v", expected, actual) + } +} + +func TestCloneConfig(t *testing.T) { + RegisterServerPubKey("testKey", testPubKeyRSA) + defer DeregisterServerPubKey("testKey") + + expectedServerName := "example.com" + dsn := "tcp(example.com:1234)/?tls=true&foobar=baz&serverPubKey=testKey" + cfg, err := ParseDSN(dsn) + if err != nil { + t.Fatal(err.Error()) + } + + cfg2 := cfg.Clone() + if cfg == cfg2 { + t.Errorf("Config.Clone did not create a separate config struct") + } + + if cfg2.TLS.ServerName != expectedServerName { + t.Errorf("cfg.tls.ServerName should be %q, got %q (host with port)", expectedServerName, cfg.TLS.ServerName) + } + + cfg2.TLS.ServerName = "example2.com" + if cfg.TLS.ServerName == cfg2.TLS.ServerName { + t.Errorf("changed cfg.tls.Server name should not propagate to original Config") + } + + if _, ok := cfg2.Params["foobar"]; !ok { + t.Errorf("cloned Config is missing custom params") + } + + delete(cfg2.Params, "foobar") + + if _, ok := cfg.Params["foobar"]; !ok { + t.Errorf("custom params in cloned Config should not propagate to original Config") + } + + if !reflect.DeepEqual(cfg.pubKey, cfg2.pubKey) { + t.Errorf("public key in Config should be identical") + } +} + +func TestNormalizeTLSConfig(t *testing.T) { + tt := []struct { + tlsConfig string + want *tls.Config + }{ + {"", nil}, + {"false", nil}, + {"true", &tls.Config{ServerName: "myserver"}}, + {"skip-verify", &tls.Config{InsecureSkipVerify: true}}, + {"preferred", &tls.Config{InsecureSkipVerify: true}}, + {"test_tls_config", &tls.Config{ServerName: "myServerName"}}, + } + + RegisterTLSConfig("test_tls_config", &tls.Config{ServerName: "myServerName"}) + defer func() { DeregisterTLSConfig("test_tls_config") }() + + for _, tc := range tt { + t.Run(tc.tlsConfig, func(t *testing.T) { + cfg := &Config{ + Addr: "myserver:3306", + TLSConfig: tc.tlsConfig, + } + + cfg.normalize() + + if cfg.TLS == nil { + if tc.want != nil { + t.Fatal("wanted a tls config but got nil instead") + } + return + } + + if cfg.TLS.ServerName != tc.want.ServerName { + t.Errorf("tls.ServerName doesn't match (want: '%s', got: '%s')", + tc.want.ServerName, cfg.TLS.ServerName) + } + if cfg.TLS.InsecureSkipVerify != tc.want.InsecureSkipVerify { + t.Errorf("tls.InsecureSkipVerify doesn't match (want: %T, got :%T)", + tc.want.InsecureSkipVerify, cfg.TLS.InsecureSkipVerify) + } + }) + } +} + +func BenchmarkParseDSN(b *testing.B) { + b.ReportAllocs() + + for b.Loop() { + for _, tst := range testDSNs { + if _, err := ParseDSN(tst.in); err != nil { + b.Error(err.Error()) + } + } + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/errors.go b/_LICENSES/github.com/go-sql-driver/mysql/errors.go new file mode 100644 index 0000000000..584617b118 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/errors.go @@ -0,0 +1,83 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "errors" + "fmt" + "log" + "os" +) + +// Various errors the driver might return. Can change between driver versions. +var ( + ErrInvalidConn = errors.New("invalid connection") + ErrMalformPkt = errors.New("malformed packet") + ErrNoTLS = errors.New("TLS requested but server does not support TLS") + ErrCleartextPassword = errors.New("this user requires clear text authentication. If you still want to use it, please add 'allowCleartextPasswords=1' to your DSN") + ErrNativePassword = errors.New("this user requires mysql native password authentication") + ErrOldPassword = errors.New("this user requires old password authentication. If you still want to use it, please add 'allowOldPasswords=1' to your DSN. See also https://github.com/go-sql-driver/mysql/wiki/old_passwords") + ErrUnknownPlugin = errors.New("this authentication plugin is not supported") + ErrOldProtocol = errors.New("MySQL server does not support required protocol 41+") + ErrPktSync = errors.New("commands out of sync. You can't run this command now") + ErrPktSyncMul = errors.New("commands out of sync. Did you run multiple statements at once?") + ErrPktTooLarge = errors.New("packet for query is too large. Try adjusting the `Config.MaxAllowedPacket`") + ErrBusyBuffer = errors.New("busy buffer") + + // errBadConnNoWrite is used for connection errors where nothing was sent to the database yet. + // If this happens first in a function starting a database interaction, it should be replaced by driver.ErrBadConn + // to trigger a resend. Use mc.markBadConn(err) to do this. + // See https://github.com/go-sql-driver/mysql/pull/302 + errBadConnNoWrite = errors.New("bad connection") +) + +var defaultLogger = Logger(log.New(os.Stderr, "[mysql] ", log.Ldate|log.Ltime)) + +// Logger is used to log critical error messages. +type Logger interface { + Print(v ...any) +} + +// NopLogger is a nop implementation of the Logger interface. +type NopLogger struct{} + +// Print implements Logger interface. +func (nl *NopLogger) Print(_ ...any) {} + +// SetLogger is used to set the default logger for critical errors. +// The initial logger is os.Stderr. +func SetLogger(logger Logger) error { + if logger == nil { + return errors.New("logger is nil") + } + defaultLogger = logger + return nil +} + +// MySQLError is an error type which represents a single MySQL error +type MySQLError struct { + Number uint16 + SQLState [5]byte + Message string +} + +func (me *MySQLError) Error() string { + if me.SQLState != [5]byte{} { + return fmt.Sprintf("Error %d (%s): %s", me.Number, me.SQLState, me.Message) + } + + return fmt.Sprintf("Error %d: %s", me.Number, me.Message) +} + +func (me *MySQLError) Is(err error) bool { + if merr, ok := err.(*MySQLError); ok { + return merr.Number == me.Number + } + return false +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/errors_test.go b/_LICENSES/github.com/go-sql-driver/mysql/errors_test.go new file mode 100644 index 0000000000..53d634454a --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/errors_test.go @@ -0,0 +1,61 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "errors" + "log" + "testing" +) + +func TestErrorsSetLogger(t *testing.T) { + previous := defaultLogger + defer func() { + defaultLogger = previous + }() + + // set up logger + const expected = "prefix: test\n" + buffer := bytes.NewBuffer(make([]byte, 0, 64)) + logger := log.New(buffer, "prefix: ", 0) + + // print + SetLogger(logger) + defaultLogger.Print("test") + + // check result + if actual := buffer.String(); actual != expected { + t.Errorf("expected %q, got %q", expected, actual) + } +} + +func TestErrorsStrictIgnoreNotes(t *testing.T) { + runTests(t, dsn+"&sql_notes=false", func(dbt *DBTest) { + dbt.mustExec("DROP TABLE IF EXISTS does_not_exist") + }) +} + +func TestMySQLErrIs(t *testing.T) { + infraErr := &MySQLError{Number: 1234, Message: "the server is on fire"} + otherInfraErr := &MySQLError{Number: 1234, Message: "the datacenter is flooded"} + if !errors.Is(infraErr, otherInfraErr) { + t.Errorf("expected errors to be the same: %+v %+v", infraErr, otherInfraErr) + } + + differentCodeErr := &MySQLError{Number: 5678, Message: "the server is on fire"} + if errors.Is(infraErr, differentCodeErr) { + t.Fatalf("expected errors to be different: %+v %+v", infraErr, differentCodeErr) + } + + nonMysqlErr := errors.New("not a mysql error") + if errors.Is(infraErr, nonMysqlErr) { + t.Fatalf("expected errors to be different: %+v %+v", infraErr, nonMysqlErr) + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/fields.go b/_LICENSES/github.com/go-sql-driver/mysql/fields.go new file mode 100644 index 0000000000..ee9d964171 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/fields.go @@ -0,0 +1,228 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2017 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql" + "reflect" +) + +func (mf *mysqlField) typeDatabaseName() string { + switch mf.fieldType { + case fieldTypeBit: + return "BIT" + case fieldTypeBLOB: + if mf.charSet != binaryCollationID { + return "TEXT" + } + return "BLOB" + case fieldTypeDate: + return "DATE" + case fieldTypeDateTime: + return "DATETIME" + case fieldTypeDecimal: + return "DECIMAL" + case fieldTypeDouble: + return "DOUBLE" + case fieldTypeEnum: + return "ENUM" + case fieldTypeFloat: + return "FLOAT" + case fieldTypeGeometry: + return "GEOMETRY" + case fieldTypeInt24: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED MEDIUMINT" + } + return "MEDIUMINT" + case fieldTypeJSON: + return "JSON" + case fieldTypeLong: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED INT" + } + return "INT" + case fieldTypeLongBLOB: + if mf.charSet != binaryCollationID { + return "LONGTEXT" + } + return "LONGBLOB" + case fieldTypeLongLong: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED BIGINT" + } + return "BIGINT" + case fieldTypeMediumBLOB: + if mf.charSet != binaryCollationID { + return "MEDIUMTEXT" + } + return "MEDIUMBLOB" + case fieldTypeNewDate: + return "DATE" + case fieldTypeNewDecimal: + return "DECIMAL" + case fieldTypeNULL: + return "NULL" + case fieldTypeSet: + return "SET" + case fieldTypeShort: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED SMALLINT" + } + return "SMALLINT" + case fieldTypeString: + if mf.flags&flagEnum != 0 { + return "ENUM" + } else if mf.flags&flagSet != 0 { + return "SET" + } + if mf.charSet == binaryCollationID { + return "BINARY" + } + return "CHAR" + case fieldTypeTime: + return "TIME" + case fieldTypeTimestamp: + return "TIMESTAMP" + case fieldTypeTiny: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED TINYINT" + } + return "TINYINT" + case fieldTypeTinyBLOB: + if mf.charSet != binaryCollationID { + return "TINYTEXT" + } + return "TINYBLOB" + case fieldTypeVarChar: + if mf.charSet == binaryCollationID { + return "VARBINARY" + } + return "VARCHAR" + case fieldTypeVarString: + if mf.charSet == binaryCollationID { + return "VARBINARY" + } + return "VARCHAR" + case fieldTypeYear: + return "YEAR" + case fieldTypeVector: + return "VECTOR" + default: + return "" + } +} + +var ( + scanTypeFloat32 = reflect.TypeFor[float32]() + scanTypeFloat64 = reflect.TypeFor[float64]() + scanTypeInt8 = reflect.TypeFor[int8]() + scanTypeInt16 = reflect.TypeFor[int16]() + scanTypeInt32 = reflect.TypeFor[int32]() + scanTypeInt64 = reflect.TypeFor[int64]() + scanTypeNullFloat = reflect.TypeFor[sql.NullFloat64]() + scanTypeNullInt = reflect.TypeFor[sql.NullInt64]() + scanTypeNullUint = reflect.TypeFor[sql.Null[uint64]]() + scanTypeNullTime = reflect.TypeFor[sql.NullTime]() + scanTypeUint8 = reflect.TypeFor[uint8]() + scanTypeUint16 = reflect.TypeFor[uint16]() + scanTypeUint32 = reflect.TypeFor[uint32]() + scanTypeUint64 = reflect.TypeFor[uint64]() + scanTypeString = reflect.TypeFor[string]() + scanTypeNullString = reflect.TypeFor[sql.NullString]() + scanTypeBytes = reflect.TypeFor[[]byte]() + scanTypeUnknown = reflect.TypeFor[*any]() +) + +type mysqlField struct { + tableName string + name string + length uint32 + flags fieldFlag + fieldType fieldType + decimals byte + charSet uint8 +} + +func (mf *mysqlField) scanType() reflect.Type { + switch mf.fieldType { + case fieldTypeTiny: + if mf.flags&flagNotNULL != 0 { + if mf.flags&flagUnsigned != 0 { + return scanTypeUint8 + } + return scanTypeInt8 + } + return scanTypeNullInt + + case fieldTypeShort, fieldTypeYear: + if mf.flags&flagNotNULL != 0 { + if mf.flags&flagUnsigned != 0 { + return scanTypeUint16 + } + return scanTypeInt16 + } + return scanTypeNullInt + + case fieldTypeInt24, fieldTypeLong: + if mf.flags&flagNotNULL != 0 { + if mf.flags&flagUnsigned != 0 { + return scanTypeUint32 + } + return scanTypeInt32 + } + return scanTypeNullInt + + case fieldTypeLongLong: + if mf.flags&flagNotNULL != 0 { + if mf.flags&flagUnsigned != 0 { + return scanTypeUint64 + } + return scanTypeInt64 + } + if mf.flags&flagUnsigned != 0 { + return scanTypeNullUint + } + return scanTypeNullInt + + case fieldTypeFloat: + if mf.flags&flagNotNULL != 0 { + return scanTypeFloat32 + } + return scanTypeNullFloat + + case fieldTypeDouble: + if mf.flags&flagNotNULL != 0 { + return scanTypeFloat64 + } + return scanTypeNullFloat + + case fieldTypeBit, fieldTypeTinyBLOB, fieldTypeMediumBLOB, fieldTypeLongBLOB, + fieldTypeBLOB, fieldTypeVarString, fieldTypeString, fieldTypeGeometry, fieldTypeVector: + if mf.charSet == binaryCollationID { + return scanTypeBytes + } + fallthrough + case fieldTypeDecimal, fieldTypeNewDecimal, fieldTypeVarChar, + fieldTypeEnum, fieldTypeSet, fieldTypeJSON, fieldTypeTime: + if mf.flags&flagNotNULL != 0 { + return scanTypeString + } + return scanTypeNullString + + case fieldTypeDate, fieldTypeNewDate, + fieldTypeTimestamp, fieldTypeDateTime: + // NullTime is always returned for more consistent behavior as it can + // handle both cases of parseTime regardless if the field is nullable. + return scanTypeNullTime + + default: + return scanTypeUnknown + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/go.mod b/_LICENSES/github.com/go-sql-driver/mysql/go.mod new file mode 100644 index 0000000000..728ab9fc77 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/go.mod @@ -0,0 +1,5 @@ +module github.com/go-sql-driver/mysql + +go 1.24.0 + +require filippo.io/edwards25519 v1.2.0 diff --git a/_LICENSES/github.com/go-sql-driver/mysql/go.sum b/_LICENSES/github.com/go-sql-driver/mysql/go.sum new file mode 100644 index 0000000000..277ea2e531 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/go.sum @@ -0,0 +1,2 @@ +filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo= +filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc= diff --git a/_LICENSES/github.com/go-sql-driver/mysql/infile.go b/_LICENSES/github.com/go-sql-driver/mysql/infile.go new file mode 100644 index 0000000000..597b5e7f67 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/infile.go @@ -0,0 +1,181 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "fmt" + "io" + "os" + "strings" + "sync" +) + +var ( + fileRegister map[string]struct{} + fileRegisterLock sync.RWMutex + readerRegister map[string]func() io.Reader + readerRegisterLock sync.RWMutex +) + +// RegisterLocalFile adds the given file to the file allowlist, +// so that it can be used by "LOAD DATA LOCAL INFILE ". +// Alternatively you can allow the use of all local files with +// the DSN parameter 'allowAllFiles=true' +// +// filePath := "/home/gopher/data.csv" +// mysql.RegisterLocalFile(filePath) +// err := db.Exec("LOAD DATA LOCAL INFILE '" + filePath + "' INTO TABLE foo") +// if err != nil { +// ... +func RegisterLocalFile(filePath string) { + fileRegisterLock.Lock() + // lazy map init + if fileRegister == nil { + fileRegister = make(map[string]struct{}) + } + + fileRegister[strings.Trim(filePath, `"`)] = struct{}{} + fileRegisterLock.Unlock() +} + +// DeregisterLocalFile removes the given filepath from the allowlist. +func DeregisterLocalFile(filePath string) { + fileRegisterLock.Lock() + delete(fileRegister, strings.Trim(filePath, `"`)) + fileRegisterLock.Unlock() +} + +// RegisterReaderHandler registers a handler function which is used +// to receive a io.Reader. +// The Reader can be used by "LOAD DATA LOCAL INFILE Reader::". +// If the handler returns a io.ReadCloser Close() is called when the +// request is finished. +// +// mysql.RegisterReaderHandler("data", func() io.Reader { +// var csvReader io.Reader // Some Reader that returns CSV data +// ... // Open Reader here +// return csvReader +// }) +// err := db.Exec("LOAD DATA LOCAL INFILE 'Reader::data' INTO TABLE foo") +// if err != nil { +// ... +func RegisterReaderHandler(name string, handler func() io.Reader) { + readerRegisterLock.Lock() + // lazy map init + if readerRegister == nil { + readerRegister = make(map[string]func() io.Reader) + } + + readerRegister[name] = handler + readerRegisterLock.Unlock() +} + +// DeregisterReaderHandler removes the ReaderHandler function with +// the given name from the registry. +func DeregisterReaderHandler(name string) { + readerRegisterLock.Lock() + delete(readerRegister, name) + readerRegisterLock.Unlock() +} + +func deferredClose(err *error, closer io.Closer) { + closeErr := closer.Close() + if *err == nil { + *err = closeErr + } +} + +const defaultPacketSize = 16 * 1024 // 16KB is small enough for disk readahead and large enough for TCP + +func (mc *okHandler) handleInFileRequest(name string) (err error) { + var rdr io.Reader + packetSize := min(mc.maxWriteSize, defaultPacketSize) + + if idx := strings.Index(name, "Reader::"); idx == 0 || (idx > 0 && name[idx-1] == '/') { // io.Reader + // The server might return an an absolute path. See issue #355. + name = name[idx+8:] + + readerRegisterLock.RLock() + handler, inMap := readerRegister[name] + readerRegisterLock.RUnlock() + + if inMap { + rdr = handler() + if rdr != nil { + if cl, ok := rdr.(io.Closer); ok { + defer deferredClose(&err, cl) + } + } else { + err = fmt.Errorf("reader '%s' is ", name) + } + } else { + err = fmt.Errorf("reader '%s' is not registered", name) + } + } else { // File + name = strings.Trim(name, `"`) + fileRegisterLock.RLock() + _, exists := fileRegister[name] + fileRegisterLock.RUnlock() + if mc.cfg.AllowAllFiles || exists { + var file *os.File + var fi os.FileInfo + + if file, err = os.Open(name); err == nil { + defer deferredClose(&err, file) + + // get file size + if fi, err = file.Stat(); err == nil { + rdr = file + if fileSize := int(fi.Size()); fileSize < packetSize { + packetSize = fileSize + } + } + } + } else { + err = fmt.Errorf("local file '%s' is not registered", name) + } + } + + // send content packets + var data []byte + + // if packetSize == 0, the Reader contains no data + if err == nil && packetSize > 0 { + data = make([]byte, 4+packetSize) + var n int + for err == nil { + n, err = rdr.Read(data[4:]) + if n > 0 { + if ioErr := mc.conn().writePacket(data[:4+n]); ioErr != nil { + return ioErr + } + } + } + if err == io.EOF { + err = nil + } + } + + // send empty packet (termination) + if data == nil { + data = make([]byte, 4) + } + if ioErr := mc.conn().writePacket(data[:4]); ioErr != nil { + return ioErr + } + mc.conn().syncSequence() + + // read OK packet + if err == nil { + return mc.readResultOK() + } + + mc.conn().readPacket() + return err +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/nulltime.go b/_LICENSES/github.com/go-sql-driver/mysql/nulltime.go new file mode 100644 index 0000000000..316a48aae1 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/nulltime.go @@ -0,0 +1,71 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql" + "database/sql/driver" + "fmt" + "time" +) + +// NullTime represents a time.Time that may be NULL. +// NullTime implements the Scanner interface so +// it can be used as a scan destination: +// +// var nt NullTime +// err := db.QueryRow("SELECT time FROM foo WHERE id=?", id).Scan(&nt) +// ... +// if nt.Valid { +// // use nt.Time +// } else { +// // NULL value +// } +// +// # This NullTime implementation is not driver-specific +// +// Deprecated: NullTime doesn't honor the loc DSN parameter. +// NullTime.Scan interprets a time as UTC, not the loc DSN parameter. +// Use sql.NullTime instead. +type NullTime sql.NullTime + +// Scan implements the Scanner interface. +// The value type must be time.Time or string / []byte (formatted time-string), +// otherwise Scan fails. +func (nt *NullTime) Scan(value any) (err error) { + if value == nil { + nt.Time, nt.Valid = time.Time{}, false + return + } + + switch v := value.(type) { + case time.Time: + nt.Time, nt.Valid = v, true + return + case []byte: + nt.Time, err = parseDateTime(v, time.UTC) + nt.Valid = (err == nil) + return + case string: + nt.Time, err = parseDateTime([]byte(v), time.UTC) + nt.Valid = (err == nil) + return + } + + nt.Valid = false + return fmt.Errorf("can't convert %T to time.Time", value) +} + +// Value implements the driver Valuer interface. +func (nt NullTime) Value() (driver.Value, error) { + if !nt.Valid { + return nil, nil + } + return nt.Time, nil +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/nulltime_test.go b/_LICENSES/github.com/go-sql-driver/mysql/nulltime_test.go new file mode 100644 index 0000000000..4f1d9029e0 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/nulltime_test.go @@ -0,0 +1,62 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql" + "database/sql/driver" + "testing" + "time" +) + +var ( + // Check implementation of interfaces + _ driver.Valuer = NullTime{} + _ sql.Scanner = (*NullTime)(nil) +) + +func TestScanNullTime(t *testing.T) { + var scanTests = []struct { + in any + error bool + valid bool + time time.Time + }{ + {tDate, false, true, tDate}, + {sDate, false, true, tDate}, + {[]byte(sDate), false, true, tDate}, + {tDateTime, false, true, tDateTime}, + {sDateTime, false, true, tDateTime}, + {[]byte(sDateTime), false, true, tDateTime}, + {tDate0, false, true, tDate0}, + {sDate0, false, true, tDate0}, + {[]byte(sDate0), false, true, tDate0}, + {sDateTime0, false, true, tDate0}, + {[]byte(sDateTime0), false, true, tDate0}, + {"", true, false, tDate0}, + {"1234", true, false, tDate0}, + {0, true, false, tDate0}, + } + + var nt = NullTime{} + var err error + + for _, tst := range scanTests { + err = nt.Scan(tst.in) + if (err != nil) != tst.error { + t.Errorf("%v: expected error status %t, got %t", tst.in, tst.error, (err != nil)) + } + if nt.Valid != tst.valid { + t.Errorf("%v: expected valid status %t, got %t", tst.in, tst.valid, nt.Valid) + } + if nt.Time != tst.time { + t.Errorf("%v: expected time %v, got %v", tst.in, tst.time, nt.Time) + } + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/packets.go b/_LICENSES/github.com/go-sql-driver/mysql/packets.go new file mode 100644 index 0000000000..d0b21b06c9 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/packets.go @@ -0,0 +1,1451 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "crypto/tls" + "database/sql/driver" + "encoding/binary" + "encoding/json" + "fmt" + "io" + "math" + "os" + "strconv" + "time" +) + +// MySQL client/server protocol documentations. +// https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_PROTOCOL.html +// https://mariadb.com/kb/en/clientserver-protocol/ + +// read n bytes from mc.buf +func (mc *mysqlConn) readNext(n int) ([]byte, error) { + if mc.buf.len() < n { + err := mc.buf.fill(n, mc.readWithTimeout) + if err != nil { + return nil, err + } + } + return mc.buf.readNext(n), nil +} + +// Read packet to buffer 'data' +func (mc *mysqlConn) readPacket() ([]byte, error) { + var prevData []byte + invalidSequence := false + + readNext := mc.readNext + if mc.compress { + readNext = mc.compIO.readNext + } + + for { + // read packet header + data, err := readNext(4) + if err != nil { + mc.close() + if cerr := mc.canceled.Value(); cerr != nil { + return nil, cerr + } + mc.log(err) + return nil, ErrInvalidConn + } + + // packet length [24 bit] + pktLen := getUint24(data[:3]) + seq := data[3] + + // check packet sync [8 bit] + if seq != mc.sequence { + mc.log(fmt.Sprintf("[warn] unexpected sequence nr: expected %v, got %v", mc.sequence, seq)) + // MySQL and MariaDB doesn't check packet nr in compressed packet. + if !mc.compress { + // For large packets, we stop reading as soon as sync error. + if len(prevData) > 0 { + mc.close() + return nil, ErrPktSyncMul + } + invalidSequence = true + } + } + mc.sequence = seq + 1 + + // packets with length 0 terminate a previous packet which is a + // multiple of (2^24)-1 bytes long + if pktLen == 0 { + // there was no previous packet + if prevData == nil { + mc.log(ErrMalformPkt) + mc.close() + return nil, ErrInvalidConn + } + return prevData, nil + } + + // read packet body [pktLen bytes] + data, err = readNext(pktLen) + if err != nil { + mc.close() + if cerr := mc.canceled.Value(); cerr != nil { + return nil, cerr + } + mc.log(err) + return nil, ErrInvalidConn + } + + // return data if this was the last packet + if pktLen < maxPacketSize { + // zero allocations for non-split packets + if prevData != nil { + data = append(prevData, data...) + } + if invalidSequence { + mc.close() + // return sync error only for regular packet. + // error packets may have wrong sequence number. + if data[0] != iERR { + return nil, ErrPktSync + } + } + return data, nil + } + + prevData = append(prevData, data...) + } +} + +// Write packet buffer 'data' +func (mc *mysqlConn) writePacket(data []byte) error { + pktLen := len(data) - 4 + if pktLen > mc.maxAllowedPacket { + return ErrPktTooLarge + } + + writeFunc := mc.writeWithTimeout + if mc.compress { + writeFunc = mc.compIO.writePackets + } + + for { + size := min(maxPacketSize, pktLen) + putUint24(data[:3], size) + data[3] = mc.sequence + + // Write packet + if debug { + fmt.Fprintf(os.Stderr, "writePacket: size=%v seq=%v\n", size, mc.sequence) + } + + n, err := writeFunc(data[:4+size]) + if err != nil { + mc.cleanup() + if cerr := mc.canceled.Value(); cerr != nil { + return cerr + } + if n == 0 && pktLen == len(data)-4 { + // only for the first loop iteration when nothing was written yet + mc.log(err) + return errBadConnNoWrite + } else { + return err + } + } + if n != 4+size { + // io.Writer(b) must return a non-nil error if it cannot write len(b) bytes. + // The io.ErrShortWrite error is used to indicate that this rule has not been followed. + mc.cleanup() + return io.ErrShortWrite + } + + mc.sequence++ + if size != maxPacketSize { + return nil + } + pktLen -= size + data = data[size:] + } +} + +/****************************************************************************** +* Initialization Process * +******************************************************************************/ + +// Handshake Initialization Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_handshake_v10.html +// https://mariadb.com/kb/en/connection/#initial-handshake-packet +func (mc *mysqlConn) readHandshakePacket() (data []byte, capabilities capabilityFlag, extendedCapabilities extendedCapabilityFlag, plugin string, err error) { + data, err = mc.readPacket() + if err != nil { + return + } + + if data[0] == iERR { + err = mc.handleErrorPacket(data) + return + } + + // protocol version [1 byte] + if data[0] < minProtocolVersion { + return nil, 0, 0, "", fmt.Errorf( + "unsupported protocol version %d. Version %d or higher is required", + data[0], + minProtocolVersion, + ) + } + + // server version [null terminated string] + // connection id [4 bytes] + pos := 1 + bytes.IndexByte(data[1:], 0x00) + 1 + 4 + + // first part of the password cipher [8 bytes] + authData := data[pos : pos+8] + + // (filler) always 0x00 [1 byte] + pos += 8 + 1 + + // capability flags (lower 2 bytes) [2 bytes] + capabilities = capabilityFlag(binary.LittleEndian.Uint16(data[pos : pos+2])) + if capabilities&clientProtocol41 == 0 { + return nil, capabilities, 0, "", ErrOldProtocol + } + if capabilities&clientSSL == 0 && mc.cfg.TLS != nil { + if mc.cfg.AllowFallbackToPlaintext { + mc.cfg.TLS = nil + } else { + return nil, capabilities, 0, "", ErrNoTLS + } + } + pos += 2 + + if len(data) > pos { + // character set [1 byte] + // status flags [2 bytes] + pos += 3 + // capability flags (upper 2 bytes) [2 bytes] + capabilities |= capabilityFlag(binary.LittleEndian.Uint16(data[pos:pos+2])) << 16 + pos += 2 + // length of auth-plugin-data [1 byte] + // reserved (all [00]) [6 bytes] + pos += 7 + if capabilities&clientMySQL == 0 { + // MariaDB server extended flag + extendedCapabilities = extendedCapabilityFlag(binary.LittleEndian.Uint32(data[pos : pos+4])) + } + pos += 4 + + // second part of the password cipher [minimum 13 bytes], + // where len=MAX(13, length of auth-plugin-data - 8) + // + // The web documentation is ambiguous about the length. However, + // according to mysql-5.7/sql/auth/sql_authentication.cc line 538, + // the 13th byte is "\0 byte, terminating the second part of + // a scramble". So the second part of the password cipher is + // a NULL terminated string that's at least 13 bytes with the + // last byte being NULL. + // + // The official Python library uses the fixed length 12 + // which seems to work but technically could have a hidden bug. + authData = append(authData, data[pos:pos+12]...) + pos += 13 + + // EOF if version (>= 5.5.7 and < 5.5.10) or (>= 5.6.0 and < 5.6.2) + // \NUL otherwise + if end := bytes.IndexByte(data[pos:], 0x00); end != -1 { + plugin = string(data[pos : pos+end]) + } else { + plugin = string(data[pos:]) + } + + // make a memory safe copy of the cipher slice + var b [20]byte + copy(b[:], authData) + return b[:], capabilities, extendedCapabilities, plugin, nil + } + + // make a memory safe copy of the cipher slice + var b [8]byte + copy(b[:], authData) + return b[:], capabilities, 0, plugin, nil +} + +// initCapabilities initializes the capabilities based on server support and configuration +func (mc *mysqlConn) initCapabilities(serverCapabilities capabilityFlag, serverExtCapabilities extendedCapabilityFlag, cfg *Config) { + clientCapabilities := + clientMySQL | + clientLongFlag | + clientProtocol41 | + clientSecureConn | + clientTransactions | + clientPluginAuthLenEncClientData | + clientLocalFiles | + clientPluginAuth | + clientMultiResults | + clientConnectAttrs | + clientDeprecateEOF + + if cfg.ClientFoundRows { + clientCapabilities |= clientFoundRows + } + if cfg.compress { + clientCapabilities |= clientCompress + } + // To enable TLS / SSL + if mc.cfg.TLS != nil { + clientCapabilities |= clientSSL + } + + if mc.cfg.MultiStatements { + clientCapabilities |= clientMultiStatements + } + if n := len(cfg.DBName); n > 0 { + clientCapabilities |= clientConnectWithDB + } + + // only keep client capabilities that server have + mc.capabilities = clientCapabilities & serverCapabilities + + // set MariaDB extended clientCacheMetadata capability if server support it + mc.extCapabilities = clientCacheMetadata & serverExtCapabilities +} + +// Client Authentication Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_handshake_response.html +func (mc *mysqlConn) writeHandshakeResponsePacket(authResp []byte, plugin string) error { + // packet header 4 + // capabilities 4 + // maxPacketSize 4 + // collation id 1 + // filler 23 + data, err := mc.buf.takeSmallBuffer(4*3 + 24) + if err != nil { + mc.cleanup() + return err + } + _ = data[4*3+23] // boundery check + + // clientCapabilities [32 bit] + binary.LittleEndian.PutUint32(data[4:], uint32(mc.capabilities)) + + // MaxPacketSize [32 bit] (none) + binary.LittleEndian.PutUint32(data[8:], 0) + + // Collation ID [1 byte] + data[12] = defaultCollationID + if cname := mc.cfg.Collation; cname != "" { + colID, ok := collations[cname] + if ok { + data[12] = colID + } else if len(mc.cfg.charsets) > 0 { + // When cfg.charset is set, the collation is set by `SET NAMES COLLATE `. + return fmt.Errorf("unknown collation: %q", cname) + } + } + + // Filler [23 bytes] (all 0x00) + // or filler 19bytes + mariadb extCapabilities + pos := 13 + if mc.capabilities&clientMySQL == 0 { + for ; pos < 13+19; pos++ { + data[pos] = 0 + } + // MariaDB Extended Capabilities + binary.LittleEndian.PutUint32(data[13+19:], uint32(mc.extCapabilities)) + } else { + for ; pos < 13+23; pos++ { + data[pos] = 0 + } + } + + // SSL Connection Request Packet + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_ssl_request.html + // https://mariadb.com/kb/en/connection/#sslrequest-packet + if mc.cfg.TLS != nil { + // Send TLS / SSL request packet + if err := mc.writePacket(data); err != nil { + return err + } + + // Switch to TLS + tlsConn := tls.Client(mc.netConn, mc.cfg.TLS) + if err := tlsConn.Handshake(); err != nil { + if cerr := mc.canceled.Value(); cerr != nil { + return cerr + } + return err + } + mc.netConn = tlsConn + } + + // User [null terminated string] + if len(mc.cfg.User) > 0 { + data = append(data, mc.cfg.User...) + } + data = append(data, 0) + + // Auth Data [length encoded integer] + data = appendLengthEncodedInteger(data, uint64(len(authResp))) + data = append(data, authResp...) + + // Database name [null terminated string] + if mc.capabilities&clientConnectWithDB != 0 { + data = append(data, mc.cfg.DBName...) + data = append(data, 0) + } + + data = append(data, plugin...) + data = append(data, 0) + + // Connection Attributes + if mc.capabilities&clientConnectAttrs != 0 { + connAttrsLen := len(mc.connector.encodedAttributes) + data = appendLengthEncodedInteger(data, uint64(connAttrsLen)) + data = append(data, mc.connector.encodedAttributes...) + } + + // Send Auth packet + return mc.writePacket(data) +} + +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_auth_switch_response.html +func (mc *mysqlConn) writeAuthSwitchPacket(authData []byte) error { + pktLen := 4 + len(authData) + data, err := mc.buf.takeBuffer(pktLen) + if err != nil { + mc.cleanup() + return err + } + + // Add the auth data [EOF] + copy(data[4:], authData) + return mc.writePacket(data) +} + +/****************************************************************************** +* Command Packets * +******************************************************************************/ + +func (mc *mysqlConn) writeCommandPacket(command byte) error { + // Reset Packet Sequence + mc.resetSequence() + + data, err := mc.buf.takeSmallBuffer(4 + 1) + if err != nil { + return err + } + + // Add command byte + data[4] = command + + // Send CMD packet + err = mc.writePacket(data) + mc.syncSequence() + return err +} + +func (mc *mysqlConn) writeCommandPacketStr(command byte, arg string) error { + // Reset Packet Sequence + mc.resetSequence() + + pktLen := 1 + len(arg) + data, err := mc.buf.takeBuffer(pktLen + 4) + if err != nil { + return err + } + + // Add command byte + data[4] = command + + // Add arg + copy(data[5:], arg) + + // Send CMD packet + err = mc.writePacket(data) + mc.syncSequence() + return err +} + +func (mc *mysqlConn) writeCommandPacketUint32(command byte, arg uint32) error { + // Reset Packet Sequence + mc.resetSequence() + + data, err := mc.buf.takeSmallBuffer(4 + 1 + 4) + if err != nil { + return err + } + + // Add command byte + data[4] = command + + // Add arg [32 bit] + binary.LittleEndian.PutUint32(data[5:], arg) + + // Send CMD packet + err = mc.writePacket(data) + mc.syncSequence() + return err +} + +/****************************************************************************** +* Result Packets * +******************************************************************************/ + +func (mc *mysqlConn) readAuthResult() ([]byte, string, error) { + data, err := mc.readPacket() + if err != nil { + return nil, "", err + } + + // packet indicator + switch data[0] { + + case iOK: + // resultUnchanged, since auth happens before any queries or + // commands have been executed. + return nil, "", mc.resultUnchanged().handleOkPacket(data) + + case iAuthMoreData: + return data[1:], "", err + + case iEOF: + if len(data) == 1 { + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_old_auth_switch_request.html + return nil, "mysql_old_password", nil + } + pluginEndIndex := bytes.IndexByte(data, 0x00) + if pluginEndIndex < 0 { + return nil, "", ErrMalformPkt + } + plugin := string(data[1:pluginEndIndex]) + authData := data[pluginEndIndex+1:] + if len(authData) > 0 && authData[len(authData)-1] == 0 { + authData = authData[:len(authData)-1] + } + return authData, plugin, nil + + default: // Error otherwise + return nil, "", mc.handleErrorPacket(data) + } +} + +// Returns error if Packet is not a 'Result OK'-Packet +func (mc *okHandler) readResultOK() error { + data, err := mc.conn().readPacket() + if err != nil { + return err + } + + if data[0] == iOK { + return mc.handleOkPacket(data) + } + return mc.conn().handleErrorPacket(data) +} + +// Result Set Header Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response.html +func (mc *okHandler) readResultSetHeaderPacket() (int, bool, error) { + // handleOkPacket replaces both values; other cases leave the values unchanged. + mc.result.affectedRows = append(mc.result.affectedRows, 0) + mc.result.insertIds = append(mc.result.insertIds, 0) + + data, err := mc.conn().readPacket() + if err != nil { + return 0, false, err + } + + switch data[0] { + case iOK: + return 0, false, mc.handleOkPacket(data) + + case iERR: + return 0, false, mc.conn().handleErrorPacket(data) + + case iLocalInFile: + return 0, false, mc.handleInFileRequest(string(data[1:])) + } + + // column count + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset.html + // https://mariadb.com/kb/en/result-set-packets/#column-count-packet + num, _, len := readLengthEncodedInteger(data) + + if mc.extCapabilities&clientCacheMetadata != 0 { + return int(num), data[len] == 0x01, nil + } + // ignore remaining data in the packet. see #1478. + return int(num), true, nil +} + +// Error Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_err_packet.html +func (mc *mysqlConn) handleErrorPacket(data []byte) error { + if data[0] != iERR { + return ErrMalformPkt + } + + // 0xff [1 byte] + + // Error Number [16 bit uint] + errno := binary.LittleEndian.Uint16(data[1:3]) + + // 1792: ER_CANT_EXECUTE_IN_READ_ONLY_TRANSACTION + // 1290: ER_OPTION_PREVENTS_STATEMENT (returned by Aurora during failover) + // 1836: ER_READ_ONLY_MODE + if (errno == 1792 || errno == 1290 || errno == 1836) && mc.cfg.RejectReadOnly { + // Oops; we are connected to a read-only connection, and won't be able + // to issue any write statements. Since RejectReadOnly is configured, + // we throw away this connection hoping this one would have write + // permission. This is specifically for a possible race condition + // during failover (e.g. on AWS Aurora). See README.md for more. + // + // We explicitly close the connection before returning + // driver.ErrBadConn to ensure that `database/sql` purges this + // connection and initiates a new one for next statement next time. + mc.Close() + return driver.ErrBadConn + } + + me := &MySQLError{Number: errno} + + pos := 3 + + // SQL State [optional: # + 5bytes string] + if data[3] == 0x23 { + copy(me.SQLState[:], data[4:4+5]) + pos = 9 + } + + // Error Message [string] + me.Message = string(data[pos:]) + + return me +} + +func readStatus(b []byte) statusFlag { + return statusFlag(b[0]) | statusFlag(b[1])<<8 +} + +// Returns an instance of okHandler for codepaths where mysqlConn.result doesn't +// need to be cleared first (e.g. during authentication, or while additional +// resultsets are being fetched.) +func (mc *mysqlConn) resultUnchanged() *okHandler { + return (*okHandler)(mc) +} + +// okHandler represents the state of the connection when mysqlConn.result has +// been prepared for processing of OK packets. +// +// To correctly populate mysqlConn.result (updated by handleOkPacket()), all +// callpaths must either: +// +// 1. first clear it using clearResult(), or +// 2. confirm that they don't need to (by calling resultUnchanged()). +// +// Both return an instance of type *okHandler. +type okHandler mysqlConn + +// Exposes the underlying type's methods. +func (mc *okHandler) conn() *mysqlConn { + return (*mysqlConn)(mc) +} + +// clearResult clears the connection's stored affectedRows and insertIds +// fields. +// +// It returns a handler that can process OK responses. +func (mc *mysqlConn) clearResult() *okHandler { + mc.result = mysqlResult{} + return (*okHandler)(mc) +} + +// Ok Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_ok_packet.html +func (mc *okHandler) handleOkPacket(data []byte) error { + var n, m int + var affectedRows, insertId uint64 + + // 0x00 [1 byte] + + // Affected rows [Length Coded Binary] + affectedRows, _, n = readLengthEncodedInteger(data[1:]) + + // Insert id [Length Coded Binary] + insertId, _, m = readLengthEncodedInteger(data[1+n:]) + + // Update for the current statement result (only used by + // readResultSetHeaderPacket). + if len(mc.result.affectedRows) > 0 { + mc.result.affectedRows[len(mc.result.affectedRows)-1] = int64(affectedRows) + } + if len(mc.result.insertIds) > 0 { + mc.result.insertIds[len(mc.result.insertIds)-1] = int64(insertId) + } + + // server_status [2 bytes] + mc.status = readStatus(data[1+n+m : 1+n+m+2]) + if mc.status&statusMoreResultsExists != 0 { + return nil + } + + // warning count [2 bytes] + + return nil +} + +// Read Packets as Field Packets until EOF-Packet or an Error appears +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset_column_definition.html#sect_protocol_com_query_response_text_resultset_column_definition_41 +func (mc *mysqlConn) readColumns(count int, old []mysqlField) ([]mysqlField, error) { + columns := make([]mysqlField, count) + if len(old) != count { + old = nil + } + + for i := range count { + data, err := mc.readPacket() + if err != nil { + return nil, err + } + + // Catalog + pos, err := skipLengthEncodedString(data) + if err != nil { + return nil, err + } + + // Database [len coded string] + n, err := skipLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + + // Table [len coded string] + if mc.cfg.ColumnsWithAlias { + tableName, _, n, err := readLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + if old != nil && old[i].tableName == string(tableName) { + // avoid allocating new string + columns[i].tableName = old[i].tableName + } else { + columns[i].tableName = string(tableName) + } + } else { + n, err = skipLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + } + + // Original table [len coded string] + n, err = skipLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + + // Name [len coded string] + name, _, n, err := readLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + if old != nil && old[i].name == string(name) { + // avoid allocating new string + columns[i].name = old[i].name + } else { + columns[i].name = string(name) + } + pos += n + + // Original name [len coded string] + n, err = skipLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + + // Filler [uint8] + pos++ + + // Charset [charset, collation uint8] + columns[i].charSet = data[pos] + pos += 2 + + // Length [uint32] + columns[i].length = binary.LittleEndian.Uint32(data[pos : pos+4]) + pos += 4 + + // Field type [uint8] + columns[i].fieldType = fieldType(data[pos]) + pos++ + + // Flags [uint16] + columns[i].flags = fieldFlag(binary.LittleEndian.Uint16(data[pos : pos+2])) + pos += 2 + + // Decimals [uint8] + columns[i].decimals = data[pos] + } + + // skip EOF packet if client does not support deprecateEOF + if err := mc.skipEof(); err != nil { + return nil, err + } + return columns, nil +} + +// Read Packets as Field Packets until EOF-Packet or an Error appears +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset_row.html +func (rows *textRows) readRow(dest []driver.Value) error { + mc := rows.mc + + if rows.rs.done { + return io.EOF + } + + data, err := mc.readPacket() + if err != nil { + return err + } + + // EOF Packet + // text row packets may starts with LengthEncodedString. + // In such case, 0xFE can mean string larger than 0xffffff. + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_dt_integers.html#sect_protocol_basic_dt_int_le + if data[0] == iEOF && len(data) <= 0xffffff { + if mc.capabilities&clientDeprecateEOF == 0 { + // Deprecated EOF packet + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_eof_packet.html + mc.status = readStatus(data[3:]) + } else { + // Ok Packet with an 0xFE header + _, _, n := readLengthEncodedInteger(data[1:]) // affected_rows + _, _, m := readLengthEncodedInteger(data[1+n:]) // last_insert_id + mc.status = readStatus(data[1+n+m:]) + } + rows.rs.done = true + if !rows.HasNextResultSet() { + rows.mc = nil + } + return io.EOF + } + if data[0] == iERR { + rows.mc = nil + return mc.handleErrorPacket(data) + } + + // RowSet Packet + var ( + n int + isNull bool + pos int = 0 + ) + + for i := range dest { + // Read bytes and convert to string + var buf []byte + buf, isNull, n, err = readLengthEncodedString(data[pos:]) + pos += n + + if err != nil { + return err + } + + if isNull { + dest[i] = nil + continue + } + + switch rows.rs.columns[i].fieldType { + case fieldTypeTimestamp, + fieldTypeDateTime, + fieldTypeDate, + fieldTypeNewDate: + if mc.parseTime { + dest[i], err = parseDateTime(buf, mc.cfg.Loc) + } else { + dest[i] = buf + } + + case fieldTypeTiny, fieldTypeShort, fieldTypeInt24, fieldTypeYear, fieldTypeLong: + dest[i], err = strconv.ParseInt(string(buf), 10, 64) + + case fieldTypeLongLong: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + dest[i], err = strconv.ParseUint(string(buf), 10, 64) + } else { + dest[i], err = strconv.ParseInt(string(buf), 10, 64) + } + + case fieldTypeFloat: + var d float64 + d, err = strconv.ParseFloat(string(buf), 32) + dest[i] = float32(d) + + case fieldTypeDouble: + dest[i], err = strconv.ParseFloat(string(buf), 64) + + default: + dest[i] = buf + } + if err != nil { + return err + } + } + + return nil +} + +func (mc *mysqlConn) skipPackets(n int) error { + for range n { + if _, err := mc.readPacket(); err != nil { + return err + } + } + return nil +} + +// skips EOF packet after n * ColumnDefinition packets when clientDeprecateEOF is not set +func (mc *mysqlConn) skipEof() error { + if mc.capabilities&clientDeprecateEOF == 0 { + if _, err := mc.readPacket(); err != nil { + return err + } + } + return nil +} + +func (mc *mysqlConn) skipColumns(n int) error { + if err := mc.skipPackets(n); err != nil { + return err + } + return mc.skipEof() +} + +// Reads Packets until EOF-Packet or an Error appears. +func (mc *mysqlConn) skipRows() error { + for { + data, err := mc.readPacket() + if err != nil { + return err + } + + switch data[0] { + case iERR: + return mc.handleErrorPacket(data) + case iEOF: + // text row packets may starts with LengthEncodedString. + // In such case, 0xFE can mean string larger than 0xffffff. + if len(data) <= 0xffffff { + if mc.capabilities&clientDeprecateEOF == 0 { + // EOF packet + mc.status = readStatus(data[3:]) + } else { + // OK packet with an 0xFE header + _, _, n := readLengthEncodedInteger(data[1:]) // affected_rows + _, _, m := readLengthEncodedInteger(data[1+n:]) // last_insert_id + mc.status = readStatus(data[1+n+m:]) + } + return nil + } + } + } +} + +/****************************************************************************** +* Prepared Statements * +******************************************************************************/ + +// Prepare Result Packets +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_prepare.html#sect_protocol_com_stmt_prepare_response +func (stmt *mysqlStmt) readPrepareResultPacket() (uint16, error) { + data, err := stmt.mc.readPacket() + if err == nil { + // packet indicator [1 byte] + if data[0] != iOK { + return 0, stmt.mc.handleErrorPacket(data) + } + + // statement id [4 bytes] + stmt.id = binary.LittleEndian.Uint32(data[1:5]) + + // Column count [16 bit uint] + columnCount := binary.LittleEndian.Uint16(data[5:7]) + + // Param count [16 bit uint] + stmt.paramCount = int(binary.LittleEndian.Uint16(data[7:9])) + + // Reserved [8 bit] + + // Warning count [16 bit uint] + + return columnCount, nil + } + return 0, err +} + +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_send_long_data.html +func (stmt *mysqlStmt) writeCommandLongData(paramID int, arg []byte) error { + maxLen := stmt.mc.maxAllowedPacket - 1 + pktLen := maxLen + + // After the header (bytes 0-3) follows before the data: + // 1 byte command + // 4 bytes stmtID + // 2 bytes paramID + const dataOffset = 1 + 4 + 2 + + // Cannot use the write buffer since + // a) the buffer is too small + // b) it is in use + data := make([]byte, 4+1+4+2+len(arg)) + + copy(data[4+dataOffset:], arg) + + for argLen := len(arg); argLen > 0; argLen -= pktLen - dataOffset { + if dataOffset+argLen < maxLen { + pktLen = dataOffset + argLen + } + + // Add command byte [1 byte] + data[4] = comStmtSendLongData + + // Add stmtID [32 bit] + binary.LittleEndian.PutUint32(data[5:], stmt.id) + + // Add paramID [16 bit] + binary.LittleEndian.PutUint16(data[9:], uint16(paramID)) + + // Send CMD packet + err := stmt.mc.writePacket(data[:4+pktLen]) + // Every COM_LONG_DATA packet reset Packet Sequence + stmt.mc.resetSequence() + if err == nil { + data = data[pktLen-dataOffset:] + continue + } + return err + } + + return nil +} + +// Execute Prepared Statement +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_execute.html +func (stmt *mysqlStmt) writeExecutePacket(args []driver.Value) error { + if len(args) != stmt.paramCount { + return fmt.Errorf( + "argument count mismatch (got: %d; has: %d)", + len(args), + stmt.paramCount, + ) + } + + const minPktLen = 4 + 1 + 4 + 1 + 4 + mc := stmt.mc + + // Determine threshold dynamically to avoid packet size shortage. + longDataSize := max(mc.maxAllowedPacket/(stmt.paramCount+1), 64) + + // Reset packet-sequence + mc.resetSequence() + + var data []byte + var err error + + if len(args) == 0 { + data, err = mc.buf.takeBuffer(minPktLen) + } else { + data, err = mc.buf.takeCompleteBuffer() + // In this case the len(data) == cap(data) which is used to optimise the flow below. + } + if err != nil { + return err + } + + // command [1 byte] + data[4] = comStmtExecute + + // statement_id [4 bytes] + binary.LittleEndian.PutUint32(data[5:], stmt.id) + + // flags (0: CURSOR_TYPE_NO_CURSOR) [1 byte] + data[9] = 0x00 + + // iteration_count (uint32(1)) [4 bytes] + binary.LittleEndian.PutUint32(data[10:], 1) + + if len(args) > 0 { + pos := minPktLen + + var nullMask []byte + if maskLen, typesLen := (len(args)+7)/8, 1+2*len(args); pos+maskLen+typesLen >= cap(data) { + // buffer has to be extended but we don't know by how much so + // we depend on append after all data with known sizes fit. + // We stop at that because we deal with a lot of columns here + // which makes the required allocation size hard to guess. + tmp := make([]byte, pos+maskLen+typesLen) + copy(tmp[:pos], data[:pos]) + data = tmp + nullMask = data[pos : pos+maskLen] + // No need to clean nullMask as make ensures that. + pos += maskLen + } else { + nullMask = data[pos : pos+maskLen] + for i := range nullMask { + nullMask[i] = 0 + } + pos += maskLen + } + + // newParameterBoundFlag 1 [1 byte] + data[pos] = 0x01 + pos++ + + // type of each parameter [len(args)*2 bytes] + paramTypes := data[pos:] + pos += len(args) * 2 + + // value of each parameter [n bytes] + paramValues := data[pos:pos] + valuesCap := cap(paramValues) + + for i, arg := range args { + // build NULL-bitmap + if arg == nil { + nullMask[i/8] |= 1 << (uint(i) & 7) + paramTypes[i+i] = byte(fieldTypeNULL) + paramTypes[i+i+1] = 0x00 + continue + } + + if v, ok := arg.(json.RawMessage); ok { + arg = []byte(v) + } + // cache types and values + switch v := arg.(type) { + case int64: + paramTypes[i+i] = byte(fieldTypeLongLong) + paramTypes[i+i+1] = 0x00 + paramValues = binary.LittleEndian.AppendUint64(paramValues, uint64(v)) + + case uint64: + paramTypes[i+i] = byte(fieldTypeLongLong) + paramTypes[i+i+1] = 0x80 // type is unsigned + paramValues = binary.LittleEndian.AppendUint64(paramValues, uint64(v)) + + case float64: + paramTypes[i+i] = byte(fieldTypeDouble) + paramTypes[i+i+1] = 0x00 + paramValues = binary.LittleEndian.AppendUint64(paramValues, math.Float64bits(v)) + + case bool: + paramTypes[i+i] = byte(fieldTypeTiny) + paramTypes[i+i+1] = 0x00 + + if v { + paramValues = append(paramValues, 0x01) + } else { + paramValues = append(paramValues, 0x00) + } + + case []byte: + // Common case (non-nil value) first + if v != nil { + paramTypes[i+i] = byte(fieldTypeString) + paramTypes[i+i+1] = 0x00 + + if len(v) < longDataSize { + paramValues = appendLengthEncodedInteger(paramValues, + uint64(len(v)), + ) + paramValues = append(paramValues, v...) + } else { + if err := stmt.writeCommandLongData(i, v); err != nil { + return err + } + } + continue + } + + // Handle []byte(nil) as a NULL value + nullMask[i/8] |= 1 << (uint(i) & 7) + paramTypes[i+i] = byte(fieldTypeNULL) + paramTypes[i+i+1] = 0x00 + + case string: + paramTypes[i+i] = byte(fieldTypeString) + paramTypes[i+i+1] = 0x00 + + if len(v) < longDataSize { + paramValues = appendLengthEncodedInteger(paramValues, + uint64(len(v)), + ) + paramValues = append(paramValues, v...) + } else { + if err := stmt.writeCommandLongData(i, []byte(v)); err != nil { + return err + } + } + + case time.Time: + paramTypes[i+i] = byte(fieldTypeString) + paramTypes[i+i+1] = 0x00 + + var a [64]byte + var b = a[:0] + + if v.IsZero() { + b = append(b, "0000-00-00"...) + } else { + b, err = appendDateTime(b, v.In(mc.cfg.Loc), mc.cfg.timeTruncate) + if err != nil { + return err + } + } + + paramValues = appendLengthEncodedInteger(paramValues, + uint64(len(b)), + ) + paramValues = append(paramValues, b...) + + default: + return fmt.Errorf("cannot convert type: %T", arg) + } + } + + // Check if param values exceeded the available buffer + // In that case we must build the data packet with the new values buffer + if valuesCap != cap(paramValues) { + data = append(data[:pos], paramValues...) + mc.buf.store(data) // allow this buffer to be reused + } + + pos += len(paramValues) + data = data[:pos] + } + + err = mc.writePacket(data) + mc.syncSequence() + return err +} + +// For each remaining resultset in the stream, discards its rows and updates +// mc.affectedRows and mc.insertIds. +func (mc *okHandler) discardResults() error { + for mc.status&statusMoreResultsExists != 0 { + resLen, _, err := mc.readResultSetHeaderPacket() + if err != nil { + return err + } + if resLen > 0 { + // columns + if err := mc.conn().skipColumns(resLen); err != nil { + return err + } + // rows + if err := mc.conn().skipRows(); err != nil { + return err + } + } + } + return nil +} + +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_binary_resultset.html#sect_protocol_binary_resultset_row +func (rows *binaryRows) readRow(dest []driver.Value) error { + data, err := rows.mc.readPacket() + if err != nil { + return err + } + + // packet indicator [1 byte] + if data[0] != iOK { + // EOF/OK Packet + if data[0] == iEOF { + if rows.mc.capabilities&clientDeprecateEOF == 0 { + // EOF packet + rows.mc.status = readStatus(data[3:]) + } else { + // OK Packet with an 0xFE header + _, _, n := readLengthEncodedInteger(data[1:]) + _, _, m := readLengthEncodedInteger(data[1+n:]) + rows.mc.status = readStatus(data[1+n+m:]) + } + rows.rs.done = true + if !rows.HasNextResultSet() { + rows.mc = nil + } + return io.EOF + } + mc := rows.mc + rows.mc = nil + + // Error otherwise + return mc.handleErrorPacket(data) + } + + // NULL-bitmap, [(column-count + 7 + 2) / 8 bytes] + pos := 1 + (len(dest)+7+2)>>3 + nullMask := data[1:pos] + + for i := range dest { + // Field is NULL + // (byte >> bit-pos) % 2 == 1 + if ((nullMask[(i+2)>>3] >> uint((i+2)&7)) & 1) == 1 { + dest[i] = nil + continue + } + + // Convert to byte-coded string + switch rows.rs.columns[i].fieldType { + case fieldTypeNULL: + dest[i] = nil + continue + + // Numeric Types + case fieldTypeTiny: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + dest[i] = int64(data[pos]) + } else { + dest[i] = int64(int8(data[pos])) + } + pos++ + continue + + case fieldTypeShort, fieldTypeYear: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + dest[i] = int64(binary.LittleEndian.Uint16(data[pos : pos+2])) + } else { + dest[i] = int64(int16(binary.LittleEndian.Uint16(data[pos : pos+2]))) + } + pos += 2 + continue + + case fieldTypeInt24, fieldTypeLong: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + dest[i] = int64(binary.LittleEndian.Uint32(data[pos : pos+4])) + } else { + dest[i] = int64(int32(binary.LittleEndian.Uint32(data[pos : pos+4]))) + } + pos += 4 + continue + + case fieldTypeLongLong: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + val := binary.LittleEndian.Uint64(data[pos : pos+8]) + if val > math.MaxInt64 { + dest[i] = uint64ToString(val) + } else { + dest[i] = int64(val) + } + } else { + dest[i] = int64(binary.LittleEndian.Uint64(data[pos : pos+8])) + } + pos += 8 + continue + + case fieldTypeFloat: + dest[i] = math.Float32frombits(binary.LittleEndian.Uint32(data[pos : pos+4])) + pos += 4 + continue + + case fieldTypeDouble: + dest[i] = math.Float64frombits(binary.LittleEndian.Uint64(data[pos : pos+8])) + pos += 8 + continue + + // Length coded Binary Strings + case fieldTypeDecimal, fieldTypeNewDecimal, fieldTypeVarChar, + fieldTypeBit, fieldTypeEnum, fieldTypeSet, fieldTypeTinyBLOB, + fieldTypeMediumBLOB, fieldTypeLongBLOB, fieldTypeBLOB, + fieldTypeVarString, fieldTypeString, fieldTypeGeometry, fieldTypeJSON, + fieldTypeVector: + var isNull bool + var n int + dest[i], isNull, n, err = readLengthEncodedString(data[pos:]) + pos += n + if err == nil { + if !isNull { + continue + } else { + dest[i] = nil + continue + } + } + return err + + case + fieldTypeDate, fieldTypeNewDate, // Date YYYY-MM-DD + fieldTypeTime, // Time [-][H]HH:MM:SS[.fractal] + fieldTypeTimestamp, fieldTypeDateTime: // Timestamp YYYY-MM-DD HH:MM:SS[.fractal] + + num, isNull, n := readLengthEncodedInteger(data[pos:]) + pos += n + + switch { + case isNull: + dest[i] = nil + continue + case rows.rs.columns[i].fieldType == fieldTypeTime: + // database/sql does not support an equivalent to TIME, return a string + var dstlen uint8 + switch decimals := rows.rs.columns[i].decimals; decimals { + case 0x00, 0x1f: + dstlen = 8 + case 1, 2, 3, 4, 5, 6: + dstlen = 8 + 1 + decimals + default: + return fmt.Errorf( + "protocol error, illegal decimals value %d", + rows.rs.columns[i].decimals, + ) + } + dest[i], err = formatBinaryTime(data[pos:pos+int(num)], dstlen) + case rows.mc.parseTime: + dest[i], err = parseBinaryDateTime(num, data[pos:], rows.mc.cfg.Loc) + default: + var dstlen uint8 + if rows.rs.columns[i].fieldType == fieldTypeDate { + dstlen = 10 + } else { + switch decimals := rows.rs.columns[i].decimals; decimals { + case 0x00, 0x1f: + dstlen = 19 + case 1, 2, 3, 4, 5, 6: + dstlen = 19 + 1 + decimals + default: + return fmt.Errorf( + "protocol error, illegal decimals value %d", + rows.rs.columns[i].decimals, + ) + } + } + dest[i], err = formatBinaryDateTime(data[pos:pos+int(num)], dstlen) + } + + if err == nil { + pos += int(num) + continue + } else { + return err + } + + // Please report if this happens! + default: + return fmt.Errorf("unknown field type %d", rows.rs.columns[i].fieldType) + } + } + + return nil +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/packets_test.go b/_LICENSES/github.com/go-sql-driver/mysql/packets_test.go new file mode 100644 index 0000000000..b487051e2d --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/packets_test.go @@ -0,0 +1,357 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2016 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "errors" + "net" + "testing" + "time" +) + +var ( + errConnClosed = errors.New("connection is closed") + errConnTooManyReads = errors.New("too many reads") + errConnTooManyWrites = errors.New("too many writes") +) + +// struct to mock a net.Conn for testing purposes +type mockConn struct { + laddr net.Addr + raddr net.Addr + data []byte + written []byte + queuedReplies [][]byte + closed bool + read int + reads int + writes int + maxReads int + maxWrites int +} + +func (m *mockConn) Read(b []byte) (n int, err error) { + if m.closed { + return 0, errConnClosed + } + + m.reads++ + if m.maxReads > 0 && m.reads > m.maxReads { + return 0, errConnTooManyReads + } + + n = copy(b, m.data) + m.read += n + m.data = m.data[n:] + return +} +func (m *mockConn) Write(b []byte) (n int, err error) { + if m.closed { + return 0, errConnClosed + } + + m.writes++ + if m.maxWrites > 0 && m.writes > m.maxWrites { + return 0, errConnTooManyWrites + } + + n = len(b) + m.written = append(m.written, b...) + + if n > 0 && len(m.queuedReplies) > 0 { + m.data = m.queuedReplies[0] + m.queuedReplies = m.queuedReplies[1:] + } + return +} +func (m *mockConn) Close() error { + m.closed = true + return nil +} +func (m *mockConn) LocalAddr() net.Addr { + return m.laddr +} +func (m *mockConn) RemoteAddr() net.Addr { + return m.raddr +} +func (m *mockConn) SetDeadline(t time.Time) error { + return nil +} +func (m *mockConn) SetReadDeadline(t time.Time) error { + return nil +} +func (m *mockConn) SetWriteDeadline(t time.Time) error { + return nil +} + +// make sure mockConn implements the net.Conn interface +var _ net.Conn = new(mockConn) + +func newRWMockConn(sequence uint8) (*mockConn, *mysqlConn) { + conn := new(mockConn) + connector := newConnector(NewConfig()) + mc := &mysqlConn{ + buf: newBuffer(), + cfg: connector.cfg, + connector: connector, + netConn: conn, + closech: make(chan struct{}), + maxAllowedPacket: defaultMaxAllowedPacket, + sequence: sequence, + } + return conn, mc +} + +func TestReadPacketSingleByte(t *testing.T) { + conn := new(mockConn) + mc := &mysqlConn{ + netConn: conn, + buf: newBuffer(), + cfg: NewConfig(), + } + + conn.data = []byte{0x01, 0x00, 0x00, 0x00, 0xff} + conn.maxReads = 1 + packet, err := mc.readPacket() + if err != nil { + t.Fatal(err) + } + if len(packet) != 1 { + t.Fatalf("unexpected packet length: expected %d, got %d", 1, len(packet)) + } + if packet[0] != 0xff { + t.Fatalf("unexpected packet content: expected %x, got %x", 0xff, packet[0]) + } +} + +func TestReadPacketWrongSequenceID(t *testing.T) { + for _, testCase := range []struct { + ClientSequenceID byte + ServerSequenceID byte + ExpectedErr error + }{ + { + ClientSequenceID: 1, + ServerSequenceID: 0, + ExpectedErr: ErrPktSync, + }, + { + ClientSequenceID: 0, + ServerSequenceID: 0x42, + ExpectedErr: ErrPktSync, + }, + } { + conn, mc := newRWMockConn(testCase.ClientSequenceID) + + conn.data = []byte{0x01, 0x00, 0x00, testCase.ServerSequenceID, 0x22} + _, err := mc.readPacket() + if err != testCase.ExpectedErr { + t.Errorf("expected %v, got %v", testCase.ExpectedErr, err) + } + + // connection should not be returned to the pool in this state + if mc.IsValid() { + t.Errorf("expected IsValid() to be false") + } + } +} + +func TestReadPacketSplit(t *testing.T) { + conn := new(mockConn) + mc := &mysqlConn{ + netConn: conn, + buf: newBuffer(), + cfg: NewConfig(), + } + + data := make([]byte, maxPacketSize*2+4*3) + const pkt2ofs = maxPacketSize + 4 + const pkt3ofs = 2 * (maxPacketSize + 4) + + // case 1: payload has length maxPacketSize + data = data[:pkt2ofs+4] + + // 1st packet has maxPacketSize length and sequence id 0 + // ff ff ff 00 ... + data[0] = 0xff + data[1] = 0xff + data[2] = 0xff + + // mark the payload start and end of 1st packet so that we can check if the + // content was correctly appended + data[4] = 0x11 + data[maxPacketSize+3] = 0x22 + + // 2nd packet has payload length 0 and sequence id 1 + // 00 00 00 01 + data[pkt2ofs+3] = 0x01 + + conn.data = data + conn.maxReads = 3 + packet, err := mc.readPacket() + if err != nil { + t.Fatal(err) + } + if len(packet) != maxPacketSize { + t.Fatalf("unexpected packet length: expected %d, got %d", maxPacketSize, len(packet)) + } + if packet[0] != 0x11 { + t.Fatalf("unexpected payload start: expected %x, got %x", 0x11, packet[0]) + } + if packet[maxPacketSize-1] != 0x22 { + t.Fatalf("unexpected payload end: expected %x, got %x", 0x22, packet[maxPacketSize-1]) + } + + // case 2: payload has length which is a multiple of maxPacketSize + data = data[:cap(data)] + + // 2nd packet now has maxPacketSize length + data[pkt2ofs] = 0xff + data[pkt2ofs+1] = 0xff + data[pkt2ofs+2] = 0xff + + // mark the payload start and end of the 2nd packet + data[pkt2ofs+4] = 0x33 + data[pkt2ofs+maxPacketSize+3] = 0x44 + + // 3rd packet has payload length 0 and sequence id 2 + // 00 00 00 02 + data[pkt3ofs+3] = 0x02 + + conn.data = data + conn.reads = 0 + conn.maxReads = 5 + mc.sequence = 0 + packet, err = mc.readPacket() + if err != nil { + t.Fatal(err) + } + if len(packet) != 2*maxPacketSize { + t.Fatalf("unexpected packet length: expected %d, got %d", 2*maxPacketSize, len(packet)) + } + if packet[0] != 0x11 { + t.Fatalf("unexpected payload start: expected %x, got %x", 0x11, packet[0]) + } + if packet[2*maxPacketSize-1] != 0x44 { + t.Fatalf("unexpected payload end: expected %x, got %x", 0x44, packet[2*maxPacketSize-1]) + } + + // case 3: payload has a length larger maxPacketSize, which is not an exact + // multiple of it + data = data[:pkt2ofs+4+42] + data[pkt2ofs] = 0x2a + data[pkt2ofs+1] = 0x00 + data[pkt2ofs+2] = 0x00 + data[pkt2ofs+4+41] = 0x44 + + conn.data = data + conn.reads = 0 + conn.maxReads = 4 + mc.sequence = 0 + packet, err = mc.readPacket() + if err != nil { + t.Fatal(err) + } + if len(packet) != maxPacketSize+42 { + t.Fatalf("unexpected packet length: expected %d, got %d", maxPacketSize+42, len(packet)) + } + if packet[0] != 0x11 { + t.Fatalf("unexpected payload start: expected %x, got %x", 0x11, packet[0]) + } + if packet[maxPacketSize+41] != 0x44 { + t.Fatalf("unexpected payload end: expected %x, got %x", 0x44, packet[maxPacketSize+41]) + } +} + +func TestReadPacketFail(t *testing.T) { + conn := new(mockConn) + mc := &mysqlConn{ + netConn: conn, + buf: newBuffer(), + closech: make(chan struct{}), + cfg: NewConfig(), + } + + // illegal empty (stand-alone) packet + conn.data = []byte{0x00, 0x00, 0x00, 0x00} + conn.maxReads = 1 + _, err := mc.readPacket() + if err != ErrInvalidConn { + t.Errorf("expected ErrInvalidConn, got %v", err) + } + + // reset + conn.reads = 0 + mc.sequence = 0 + mc.buf = newBuffer() + + // fail to read header + conn.closed = true + _, err = mc.readPacket() + if err != ErrInvalidConn { + t.Errorf("expected ErrInvalidConn, got %v", err) + } + + // reset + conn.closed = false + conn.reads = 0 + mc.sequence = 0 + mc.buf = newBuffer() + + // fail to read body + conn.maxReads = 1 + _, err = mc.readPacket() + if err != ErrInvalidConn { + t.Errorf("expected ErrInvalidConn, got %v", err) + } +} + +// https://github.com/go-sql-driver/mysql/pull/801 +// not-NUL terminated plugin_name in init packet +func TestRegression801(t *testing.T) { + conn := new(mockConn) + mc := &mysqlConn{ + netConn: conn, + buf: newBuffer(), + cfg: new(Config), + sequence: 42, + closech: make(chan struct{}), + } + + conn.data = []byte{72, 0, 0, 42, 10, 53, 46, 53, 46, 56, 0, 165, 0, 0, 0, + 60, 70, 63, 58, 68, 104, 34, 97, 0, 223, 247, 33, 2, 0, 15, 128, 21, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 98, 120, 114, 47, 85, 75, 109, 99, 51, 77, + 50, 64, 0, 109, 121, 115, 113, 108, 95, 110, 97, 116, 105, 118, 101, 95, + 112, 97, 115, 115, 119, 111, 114, 100} + conn.maxReads = 1 + + authData, serverCapabilities, serverExtendedCapabilities, pluginName, err := mc.readHandshakePacket() + if err != nil { + t.Fatalf("got error: %v", err) + } + + if serverCapabilities != 2148530143 { + t.Fatalf("expected serverCapabilities to be 2148530143, got %v", serverCapabilities) + } + + if serverExtendedCapabilities != 0 { + t.Fatalf("expected serverExtendedCapabilities to be 0, got %v", serverExtendedCapabilities) + } + + if pluginName != "mysql_native_password" { + t.Errorf("expected plugin name 'mysql_native_password', got '%s'", pluginName) + } + + expectedAuthData := []byte{60, 70, 63, 58, 68, 104, 34, 97, 98, 120, 114, + 47, 85, 75, 109, 99, 51, 77, 50, 64} + if !bytes.Equal(authData, expectedAuthData) { + t.Errorf("expected authData '%v', got '%v'", expectedAuthData, authData) + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/result.go b/_LICENSES/github.com/go-sql-driver/mysql/result.go new file mode 100644 index 0000000000..82dc0f9b6c --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/result.go @@ -0,0 +1,52 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import "slices" + +import "database/sql/driver" + +// Result exposes data not available through *connection.Result. +// +// This is accessible by executing statements using sql.Conn.Raw() and +// downcasting the returned result: +// +// res, err := rawConn.Exec(...) +// res.(mysql.Result).AllRowsAffected() +type Result interface { + driver.Result + // AllRowsAffected returns a slice containing the affected rows for each + // executed statement. + AllRowsAffected() []int64 + // AllLastInsertIds returns a slice containing the last inserted ID for each + // executed statement. + AllLastInsertIds() []int64 +} + +type mysqlResult struct { + // One entry in both slices is created for every executed statement result. + affectedRows []int64 + insertIds []int64 +} + +func (res *mysqlResult) LastInsertId() (int64, error) { + return res.insertIds[len(res.insertIds)-1], nil +} + +func (res *mysqlResult) RowsAffected() (int64, error) { + return res.affectedRows[len(res.affectedRows)-1], nil +} + +func (res *mysqlResult) AllLastInsertIds() []int64 { + return slices.Clone(res.insertIds) // defensive copy +} + +func (res *mysqlResult) AllRowsAffected() []int64 { + return slices.Clone(res.affectedRows) // defensive copy +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/rows.go b/_LICENSES/github.com/go-sql-driver/mysql/rows.go new file mode 100644 index 0000000000..190e75f9bf --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/rows.go @@ -0,0 +1,225 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql/driver" + "io" + "math" + "reflect" +) + +type resultSet struct { + columns []mysqlField + columnNames []string + done bool +} + +type mysqlRows struct { + mc *mysqlConn + rs resultSet + finish func() +} + +type binaryRows struct { + mysqlRows +} + +type textRows struct { + mysqlRows +} + +func (rows *mysqlRows) Columns() []string { + if rows.rs.columnNames != nil { + return rows.rs.columnNames + } + + columns := make([]string, len(rows.rs.columns)) + if rows.mc != nil && rows.mc.cfg.ColumnsWithAlias { + for i := range columns { + if tableName := rows.rs.columns[i].tableName; len(tableName) > 0 { + columns[i] = tableName + "." + rows.rs.columns[i].name + } else { + columns[i] = rows.rs.columns[i].name + } + } + } else { + for i := range columns { + columns[i] = rows.rs.columns[i].name + } + } + + rows.rs.columnNames = columns + return columns +} + +func (rows *mysqlRows) ColumnTypeDatabaseTypeName(i int) string { + return rows.rs.columns[i].typeDatabaseName() +} + +// func (rows *mysqlRows) ColumnTypeLength(i int) (length int64, ok bool) { +// return int64(rows.rs.columns[i].length), true +// } + +func (rows *mysqlRows) ColumnTypeNullable(i int) (nullable, ok bool) { + return rows.rs.columns[i].flags&flagNotNULL == 0, true +} + +func (rows *mysqlRows) ColumnTypePrecisionScale(i int) (int64, int64, bool) { + column := rows.rs.columns[i] + decimals := int64(column.decimals) + + switch column.fieldType { + case fieldTypeDecimal, fieldTypeNewDecimal: + if decimals > 0 { + return int64(column.length) - 2, decimals, true + } + return int64(column.length) - 1, decimals, true + case fieldTypeTimestamp, fieldTypeDateTime, fieldTypeTime: + return decimals, decimals, true + case fieldTypeFloat, fieldTypeDouble: + if decimals == 0x1f { + return math.MaxInt64, math.MaxInt64, true + } + return math.MaxInt64, decimals, true + } + + return 0, 0, false +} + +func (rows *mysqlRows) ColumnTypeScanType(i int) reflect.Type { + return rows.rs.columns[i].scanType() +} + +func (rows *mysqlRows) Close() (err error) { + if f := rows.finish; f != nil { + f() + rows.finish = nil + } + + mc := rows.mc + if mc == nil { + return nil + } + if err := mc.error(); err != nil { + return err + } + + // Remove unread packets from stream + if !rows.rs.done { + err = mc.skipRows() + } + if err == nil { + handleOk := mc.clearResult() + if err = handleOk.discardResults(); err != nil { + return err + } + } + + rows.mc = nil + return err +} + +func (rows *mysqlRows) HasNextResultSet() (b bool) { + if rows.mc == nil { + return false + } + return rows.mc.status&statusMoreResultsExists != 0 +} + +func (rows *mysqlRows) nextResultSet() (int, error) { + if rows.mc == nil { + return 0, io.EOF + } + if err := rows.mc.error(); err != nil { + return 0, err + } + + // Remove unread packets from stream + if !rows.rs.done { + if err := rows.mc.skipRows(); err != nil { + return 0, err + } + rows.rs.done = true + } + + if !rows.HasNextResultSet() { + rows.mc = nil + return 0, io.EOF + } + rows.rs = resultSet{} + // rows.mc.affectedRows and rows.mc.insertIds accumulate on each call to + // nextResultSet. + resLen, _, err := rows.mc.resultUnchanged().readResultSetHeaderPacket() + if err != nil { + // Clean up about multi-results flag + rows.rs.done = true + rows.mc.status = rows.mc.status & (^statusMoreResultsExists) + } + return resLen, err +} + +func (rows *mysqlRows) nextNotEmptyResultSet() (int, error) { + for { + resLen, err := rows.nextResultSet() + if err != nil { + return 0, err + } + + if resLen > 0 { + return resLen, nil + } + + rows.rs.done = true + } +} + +func (rows *binaryRows) NextResultSet() error { + resLen, err := rows.nextNotEmptyResultSet() + if err != nil { + return err + } + + rows.rs.columns, err = rows.mc.readColumns(resLen, nil) + return err +} + +func (rows *binaryRows) Next(dest []driver.Value) error { + if mc := rows.mc; mc != nil { + if err := mc.error(); err != nil { + return err + } + + // Fetch next row from stream + return rows.readRow(dest) + } + return io.EOF +} + +func (rows *textRows) NextResultSet() (err error) { + resLen, err := rows.nextNotEmptyResultSet() + if err != nil { + return err + } + + rows.rs.columns, err = rows.mc.readColumns(resLen, nil) + return err +} + +func (rows *textRows) Next(dest []driver.Value) error { + if mc := rows.mc; mc != nil { + if err := mc.error(); err != nil { + return err + } + + // Fetch next row from stream + return rows.readRow(dest) + } + return io.EOF +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/statement.go b/_LICENSES/github.com/go-sql-driver/mysql/statement.go new file mode 100644 index 0000000000..0261903b9d --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/statement.go @@ -0,0 +1,234 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql/driver" + "encoding/json" + "fmt" + "io" + "reflect" +) + +type mysqlStmt struct { + mc *mysqlConn + id uint32 + paramCount int + columns []mysqlField +} + +func (stmt *mysqlStmt) Close() error { + if stmt.mc == nil || stmt.mc.closed.Load() { + // driver.Stmt.Close could be called more than once, thus this function + // had to be idempotent. See also Issue #450 and golang/go#16019. + // This bug has been fixed in Go 1.8. + // https://github.com/golang/go/commit/90b8a0ca2d0b565c7c7199ffcf77b15ea6b6db3a + // But we keep this function idempotent because it is safer. + return nil + } + + err := stmt.mc.writeCommandPacketUint32(comStmtClose, stmt.id) + stmt.mc = nil + return err +} + +func (stmt *mysqlStmt) NumInput() int { + return stmt.paramCount +} + +func (stmt *mysqlStmt) ColumnConverter(idx int) driver.ValueConverter { + return converter{} +} + +func (stmt *mysqlStmt) CheckNamedValue(nv *driver.NamedValue) (err error) { + nv.Value, err = converter{}.ConvertValue(nv.Value) + return +} + +func (stmt *mysqlStmt) Exec(args []driver.Value) (driver.Result, error) { + if stmt.mc.closed.Load() { + return nil, driver.ErrBadConn + } + // Send command + err := stmt.writeExecutePacket(args) + if err != nil { + return nil, stmt.mc.markBadConn(err) + } + + mc := stmt.mc + handleOk := stmt.mc.clearResult() + + // Read Result + resLen, metadataFollows, err := handleOk.readResultSetHeaderPacket() + if err != nil { + return nil, err + } + + if resLen > 0 { + // Columns + if metadataFollows && stmt.mc.extCapabilities&clientCacheMetadata != 0 { + // we can not skip column metadata because next stmt.Query() may use it. + if stmt.columns, err = mc.readColumns(resLen, stmt.columns); err != nil { + return nil, err + } + } else { + if err = mc.skipColumns(resLen); err != nil { + return nil, err + } + } + + // Rows + if err = mc.skipRows(); err != nil { + return nil, err + } + } + + if err := handleOk.discardResults(); err != nil { + return nil, err + } + + copied := mc.result + return &copied, nil +} + +func (stmt *mysqlStmt) Query(args []driver.Value) (driver.Rows, error) { + return stmt.query(args) +} + +func (stmt *mysqlStmt) query(args []driver.Value) (*binaryRows, error) { + if stmt.mc.closed.Load() { + return nil, driver.ErrBadConn + } + // Send command + err := stmt.writeExecutePacket(args) + if err != nil { + return nil, stmt.mc.markBadConn(err) + } + + mc := stmt.mc + + // Read Result + handleOk := stmt.mc.clearResult() + resLen, metadataFollows, err := handleOk.readResultSetHeaderPacket() + if err != nil { + return nil, err + } + + rows := new(binaryRows) + + if resLen > 0 { + rows.mc = mc + if metadataFollows { + if rows.rs.columns, err = mc.readColumns(resLen, stmt.columns); err != nil { + return nil, err + } + stmt.columns = rows.rs.columns + } else { + if err = mc.skipEof(); err != nil { + return nil, err + } + rows.rs.columns = stmt.columns + } + } else { + rows.rs.done = true + + switch err := rows.NextResultSet(); err { + case nil, io.EOF: + return rows, nil + default: + return nil, err + } + } + + return rows, err +} + +var jsonType = reflect.TypeFor[json.RawMessage]() + +type converter struct{} + +// ConvertValue mirrors the reference/default converter in database/sql/driver +// with _one_ exception. We support uint64 with their high bit and the default +// implementation does not. This function should be kept in sync with +// database/sql/driver defaultConverter.ConvertValue() except for that +// deliberate difference. +func (c converter) ConvertValue(v any) (driver.Value, error) { + if driver.IsValue(v) { + return v, nil + } + + if vr, ok := v.(driver.Valuer); ok { + sv, err := callValuerValue(vr) + if err != nil { + return nil, err + } + if driver.IsValue(sv) { + return sv, nil + } + // A value returned from the Valuer interface can be "a type handled by + // a database driver's NamedValueChecker interface" so we should accept + // uint64 here as well. + if u, ok := sv.(uint64); ok { + return u, nil + } + return nil, fmt.Errorf("non-Value type %T returned from Value", sv) + } + rv := reflect.ValueOf(v) + switch rv.Kind() { + case reflect.Ptr: + // indirect pointers + if rv.IsNil() { + return nil, nil + } else { + return c.ConvertValue(rv.Elem().Interface()) + } + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + return rv.Int(), nil + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + return rv.Uint(), nil + case reflect.Float32, reflect.Float64: + return rv.Float(), nil + case reflect.Bool: + return rv.Bool(), nil + case reflect.Slice: + switch t := rv.Type(); { + case t == jsonType: + return v, nil + case t.Elem().Kind() == reflect.Uint8: + return rv.Bytes(), nil + default: + return nil, fmt.Errorf("unsupported type %T, a slice of %s", v, t.Elem().Kind()) + } + case reflect.String: + return rv.String(), nil + } + return nil, fmt.Errorf("unsupported type %T, a %s", v, rv.Kind()) +} + +var valuerReflectType = reflect.TypeFor[driver.Valuer]() + +// callValuerValue returns vr.Value(), with one exception: +// If vr.Value is an auto-generated method on a pointer type and the +// pointer is nil, it would panic at runtime in the panicwrap +// method. Treat it like nil instead. +// +// This is so people can implement driver.Value on value types and +// still use nil pointers to those types to mean nil/NULL, just like +// string/*string. +// +// This is an exact copy of the same-named unexported function from the +// database/sql package. +func callValuerValue(vr driver.Valuer) (v driver.Value, err error) { + if rv := reflect.ValueOf(vr); rv.Kind() == reflect.Ptr && + rv.IsNil() && + rv.Type().Elem().Implements(valuerReflectType) { + return nil, nil + } + return vr.Value() +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/statement_test.go b/_LICENSES/github.com/go-sql-driver/mysql/statement_test.go new file mode 100644 index 0000000000..15f9d7c335 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/statement_test.go @@ -0,0 +1,151 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2017 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "database/sql/driver" + "encoding/json" + "testing" +) + +func TestConvertDerivedString(t *testing.T) { + type derived string + + output, err := converter{}.ConvertValue(derived("value")) + if err != nil { + t.Fatal("Derived string type not convertible", err) + } + + if output != "value" { + t.Fatalf("Derived string type not converted, got %#v %T", output, output) + } +} + +func TestConvertDerivedByteSlice(t *testing.T) { + type derived []uint8 + + output, err := converter{}.ConvertValue(derived("value")) + if err != nil { + t.Fatal("Byte slice not convertible", err) + } + + if !bytes.Equal(output.([]byte), []byte("value")) { + t.Fatalf("Byte slice not converted, got %#v %T", output, output) + } +} + +func TestConvertDerivedUnsupportedSlice(t *testing.T) { + type derived []int + + _, err := converter{}.ConvertValue(derived{1}) + if err == nil || err.Error() != "unsupported type mysql.derived, a slice of int" { + t.Fatal("Unexpected error", err) + } +} + +func TestConvertDerivedBool(t *testing.T) { + type derived bool + + output, err := converter{}.ConvertValue(derived(true)) + if err != nil { + t.Fatal("Derived bool type not convertible", err) + } + + if output != true { + t.Fatalf("Derived bool type not converted, got %#v %T", output, output) + } +} + +func TestConvertPointer(t *testing.T) { + str := "value" + + output, err := converter{}.ConvertValue(&str) + if err != nil { + t.Fatal("Pointer type not convertible", err) + } + + if output != "value" { + t.Fatalf("Pointer type not converted, got %#v %T", output, output) + } +} + +func TestConvertSignedIntegers(t *testing.T) { + values := []any{ + int8(-42), + int16(-42), + int32(-42), + int64(-42), + int(-42), + } + + for _, value := range values { + output, err := converter{}.ConvertValue(value) + if err != nil { + t.Fatalf("%T type not convertible %s", value, err) + } + + if output != int64(-42) { + t.Fatalf("%T type not converted, got %#v %T", value, output, output) + } + } +} + +type myUint64 struct { + value uint64 +} + +func (u myUint64) Value() (driver.Value, error) { + return u.value, nil +} + +func TestConvertUnsignedIntegers(t *testing.T) { + values := []any{ + uint8(42), + uint16(42), + uint32(42), + uint64(42), + uint(42), + myUint64{uint64(42)}, + } + + for _, value := range values { + output, err := converter{}.ConvertValue(value) + if err != nil { + t.Fatalf("%T type not convertible %s", value, err) + } + + if output != uint64(42) { + t.Fatalf("%T type not converted, got %#v %T", value, output, output) + } + } + + output, err := converter{}.ConvertValue(^uint64(0)) + if err != nil { + t.Fatal("uint64 high-bit not convertible", err) + } + + if output != ^uint64(0) { + t.Fatalf("uint64 high-bit converted, got %#v %T", output, output) + } +} + +func TestConvertJSON(t *testing.T) { + raw := json.RawMessage("{}") + + out, err := converter{}.ConvertValue(raw) + + if err != nil { + t.Fatal("json.RawMessage was failed in convert", err) + } + + if _, ok := out.(json.RawMessage); !ok { + t.Fatalf("json.RawMessage converted, got %#v %T", out, out) + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/transaction.go b/_LICENSES/github.com/go-sql-driver/mysql/transaction.go new file mode 100644 index 0000000000..8c502f49e6 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/transaction.go @@ -0,0 +1,45 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +type mysqlTx struct { + mc *mysqlConn +} + +func (tx *mysqlTx) Commit() (err error) { + if tx.mc == nil { + return ErrInvalidConn + } + if tx.mc.closed.Load() { + err = tx.mc.error() + if err == nil { + err = ErrInvalidConn + } + return + } + err = tx.mc.exec("COMMIT") + tx.mc = nil + return +} + +func (tx *mysqlTx) Rollback() (err error) { + if tx.mc == nil { + return ErrInvalidConn + } + if tx.mc.closed.Load() { + err = tx.mc.error() + if err == nil { + err = ErrInvalidConn + } + return + } + err = tx.mc.exec("ROLLBACK") + tx.mc = nil + return +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/utils.go b/_LICENSES/github.com/go-sql-driver/mysql/utils.go new file mode 100644 index 0000000000..2dccb7d53e --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/utils.go @@ -0,0 +1,805 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "crypto/tls" + "database/sql" + "database/sql/driver" + "encoding/binary" + "errors" + "fmt" + "io" + "strconv" + "strings" + "sync" + "sync/atomic" + "time" +) + +// Registry for custom tls.Configs +var ( + tlsConfigLock sync.RWMutex + tlsConfigRegistry map[string]*tls.Config +) + +// RegisterTLSConfig registers a custom tls.Config to be used with sql.Open. +// Use the key as a value in the DSN where tls=value. +// +// Note: The provided tls.Config is exclusively owned by the driver after +// registering it. +// +// rootCertPool := x509.NewCertPool() +// pem, err := os.ReadFile("/path/ca-cert.pem") +// if err != nil { +// log.Fatal(err) +// } +// if ok := rootCertPool.AppendCertsFromPEM(pem); !ok { +// log.Fatal("Failed to append PEM.") +// } +// clientCert := make([]tls.Certificate, 0, 1) +// certs, err := tls.LoadX509KeyPair("/path/client-cert.pem", "/path/client-key.pem") +// if err != nil { +// log.Fatal(err) +// } +// clientCert = append(clientCert, certs) +// mysql.RegisterTLSConfig("custom", &tls.Config{ +// RootCAs: rootCertPool, +// Certificates: clientCert, +// }) +// db, err := sql.Open("mysql", "user@tcp(localhost:3306)/test?tls=custom") +func RegisterTLSConfig(key string, config *tls.Config) error { + if _, isBool := readBool(key); isBool || strings.ToLower(key) == "skip-verify" || strings.ToLower(key) == "preferred" { + return fmt.Errorf("key '%s' is reserved", key) + } + + tlsConfigLock.Lock() + if tlsConfigRegistry == nil { + tlsConfigRegistry = make(map[string]*tls.Config) + } + + tlsConfigRegistry[key] = config + tlsConfigLock.Unlock() + return nil +} + +// DeregisterTLSConfig removes the tls.Config associated with key. +func DeregisterTLSConfig(key string) { + tlsConfigLock.Lock() + if tlsConfigRegistry != nil { + delete(tlsConfigRegistry, key) + } + tlsConfigLock.Unlock() +} + +func getTLSConfigClone(key string) (config *tls.Config) { + tlsConfigLock.RLock() + if v, ok := tlsConfigRegistry[key]; ok { + config = v.Clone() + } + tlsConfigLock.RUnlock() + return +} + +// Returns the bool value of the input. +// The 2nd return value indicates if the input was a valid bool value +func readBool(input string) (value bool, valid bool) { + switch input { + case "1", "true", "TRUE", "True": + return true, true + case "0", "false", "FALSE", "False": + return false, true + } + + // Not a valid bool value + return +} + +/****************************************************************************** +* Time related utils * +******************************************************************************/ + +func parseDateTime(b []byte, loc *time.Location) (time.Time, error) { + const base = "0000-00-00 00:00:00.000000" + switch len(b) { + case 10, 19, 21, 22, 23, 24, 25, 26: // up to "YYYY-MM-DD HH:MM:SS.MMMMMM" + if string(b) == base[:len(b)] { + return time.Time{}, nil + } + + year, err := parseByteYear(b) + if err != nil { + return time.Time{}, err + } + if b[4] != '-' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[4]) + } + + m, err := parseByte2Digits(b[5], b[6]) + if err != nil { + return time.Time{}, err + } + month := time.Month(m) + + if b[7] != '-' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[7]) + } + + day, err := parseByte2Digits(b[8], b[9]) + if err != nil { + return time.Time{}, err + } + if len(b) == 10 { + return time.Date(year, month, day, 0, 0, 0, 0, loc), nil + } + + if b[10] != ' ' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[10]) + } + + hour, err := parseByte2Digits(b[11], b[12]) + if err != nil { + return time.Time{}, err + } + if b[13] != ':' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[13]) + } + + min, err := parseByte2Digits(b[14], b[15]) + if err != nil { + return time.Time{}, err + } + if b[16] != ':' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[16]) + } + + sec, err := parseByte2Digits(b[17], b[18]) + if err != nil { + return time.Time{}, err + } + if len(b) == 19 { + return time.Date(year, month, day, hour, min, sec, 0, loc), nil + } + + if b[19] != '.' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[19]) + } + nsec, err := parseByteNanoSec(b[20:]) + if err != nil { + return time.Time{}, err + } + return time.Date(year, month, day, hour, min, sec, nsec, loc), nil + default: + return time.Time{}, fmt.Errorf("invalid time bytes: %s", b) + } +} + +func parseByteYear(b []byte) (int, error) { + year, n := 0, 1000 + for i := range 4 { + v, err := bToi(b[i]) + if err != nil { + return 0, err + } + year += v * n + n /= 10 + } + return year, nil +} + +func parseByte2Digits(b1, b2 byte) (int, error) { + d1, err := bToi(b1) + if err != nil { + return 0, err + } + d2, err := bToi(b2) + if err != nil { + return 0, err + } + return d1*10 + d2, nil +} + +func parseByteNanoSec(b []byte) (int, error) { + ns, digit := 0, 100000 // max is 6-digits + for i := range b { + v, err := bToi(b[i]) + if err != nil { + return 0, err + } + ns += v * digit + digit /= 10 + } + // nanoseconds has 10-digits. (needs to scale digits) + // 10 - 6 = 4, so we have to multiple 1000. + return ns * 1000, nil +} + +func bToi(b byte) (int, error) { + if b < '0' || b > '9' { + return 0, errors.New("not [0-9]") + } + return int(b - '0'), nil +} + +func parseBinaryDateTime(num uint64, data []byte, loc *time.Location) (driver.Value, error) { + switch num { + case 0: + return time.Time{}, nil + case 4: + return time.Date( + int(binary.LittleEndian.Uint16(data[:2])), // year + time.Month(data[2]), // month + int(data[3]), // day + 0, 0, 0, 0, + loc, + ), nil + case 7: + return time.Date( + int(binary.LittleEndian.Uint16(data[:2])), // year + time.Month(data[2]), // month + int(data[3]), // day + int(data[4]), // hour + int(data[5]), // minutes + int(data[6]), // seconds + 0, + loc, + ), nil + case 11: + return time.Date( + int(binary.LittleEndian.Uint16(data[:2])), // year + time.Month(data[2]), // month + int(data[3]), // day + int(data[4]), // hour + int(data[5]), // minutes + int(data[6]), // seconds + int(binary.LittleEndian.Uint32(data[7:11]))*1000, // nanoseconds + loc, + ), nil + } + return nil, fmt.Errorf("invalid DATETIME packet length %d", num) +} + +func appendDateTime(buf []byte, t time.Time, timeTruncate time.Duration) ([]byte, error) { + if timeTruncate > 0 { + t = t.Truncate(timeTruncate) + } + + year, month, day := t.Date() + hour, min, sec := t.Clock() + nsec := t.Nanosecond() + + if year < 1 || year > 9999 { + return buf, errors.New("year is not in the range [1, 9999]: " + strconv.Itoa(year)) // use errors.New instead of fmt.Errorf to avoid year escape to heap + } + year100 := year / 100 + year1 := year % 100 + + var localBuf [len("2006-01-02T15:04:05.999999999")]byte // does not escape + localBuf[0], localBuf[1], localBuf[2], localBuf[3] = digits10[year100], digits01[year100], digits10[year1], digits01[year1] + localBuf[4] = '-' + localBuf[5], localBuf[6] = digits10[month], digits01[month] + localBuf[7] = '-' + localBuf[8], localBuf[9] = digits10[day], digits01[day] + + if hour == 0 && min == 0 && sec == 0 && nsec == 0 { + return append(buf, localBuf[:10]...), nil + } + + localBuf[10] = ' ' + localBuf[11], localBuf[12] = digits10[hour], digits01[hour] + localBuf[13] = ':' + localBuf[14], localBuf[15] = digits10[min], digits01[min] + localBuf[16] = ':' + localBuf[17], localBuf[18] = digits10[sec], digits01[sec] + + if nsec == 0 { + return append(buf, localBuf[:19]...), nil + } + nsec100000000 := nsec / 100000000 + nsec1000000 := (nsec / 1000000) % 100 + nsec10000 := (nsec / 10000) % 100 + nsec100 := (nsec / 100) % 100 + nsec1 := nsec % 100 + localBuf[19] = '.' + + // milli second + localBuf[20], localBuf[21], localBuf[22] = + digits01[nsec100000000], digits10[nsec1000000], digits01[nsec1000000] + // micro second + localBuf[23], localBuf[24], localBuf[25] = + digits10[nsec10000], digits01[nsec10000], digits10[nsec100] + // nano second + localBuf[26], localBuf[27], localBuf[28] = + digits01[nsec100], digits10[nsec1], digits01[nsec1] + + // trim trailing zeros + n := len(localBuf) + for n > 0 && localBuf[n-1] == '0' { + n-- + } + + return append(buf, localBuf[:n]...), nil +} + +// zeroDateTime is used in formatBinaryDateTime to avoid an allocation +// if the DATE or DATETIME has the zero value. +// It must never be changed. +// The current behavior depends on database/sql copying the result. +var zeroDateTime = []byte("0000-00-00 00:00:00.000000") + +const digits01 = "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789" +const digits10 = "0000000000111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999" + +func appendMicrosecs(dst, src []byte, decimals int) []byte { + if decimals <= 0 { + return dst + } + if len(src) == 0 { + return append(dst, ".000000"[:decimals+1]...) + } + + microsecs := binary.LittleEndian.Uint32(src[:4]) + p1 := byte(microsecs / 10000) + microsecs -= 10000 * uint32(p1) + p2 := byte(microsecs / 100) + microsecs -= 100 * uint32(p2) + p3 := byte(microsecs) + + switch decimals { + default: + return append(dst, '.', + digits10[p1], digits01[p1], + digits10[p2], digits01[p2], + digits10[p3], digits01[p3], + ) + case 1: + return append(dst, '.', + digits10[p1], + ) + case 2: + return append(dst, '.', + digits10[p1], digits01[p1], + ) + case 3: + return append(dst, '.', + digits10[p1], digits01[p1], + digits10[p2], + ) + case 4: + return append(dst, '.', + digits10[p1], digits01[p1], + digits10[p2], digits01[p2], + ) + case 5: + return append(dst, '.', + digits10[p1], digits01[p1], + digits10[p2], digits01[p2], + digits10[p3], + ) + } +} + +func formatBinaryDateTime(src []byte, length uint8) (driver.Value, error) { + // length expects the deterministic length of the zero value, + // negative time and 100+ hours are automatically added if needed + if len(src) == 0 { + return zeroDateTime[:length], nil + } + var dst []byte // return value + var p1, p2, p3 byte // current digit pair + + switch length { + case 10, 19, 21, 22, 23, 24, 25, 26: + default: + t := "DATE" + if length > 10 { + t += "TIME" + } + return nil, fmt.Errorf("illegal %s length %d", t, length) + } + switch len(src) { + case 4, 7, 11: + default: + t := "DATE" + if length > 10 { + t += "TIME" + } + return nil, fmt.Errorf("illegal %s packet length %d", t, len(src)) + } + dst = make([]byte, 0, length) + // start with the date + year := binary.LittleEndian.Uint16(src[:2]) + pt := year / 100 + p1 = byte(year - 100*uint16(pt)) + p2, p3 = src[2], src[3] + dst = append(dst, + digits10[pt], digits01[pt], + digits10[p1], digits01[p1], '-', + digits10[p2], digits01[p2], '-', + digits10[p3], digits01[p3], + ) + if length == 10 { + return dst, nil + } + if len(src) == 4 { + return append(dst, zeroDateTime[10:length]...), nil + } + dst = append(dst, ' ') + p1 = src[4] // hour + src = src[5:] + + // p1 is 2-digit hour, src is after hour + p2, p3 = src[0], src[1] + dst = append(dst, + digits10[p1], digits01[p1], ':', + digits10[p2], digits01[p2], ':', + digits10[p3], digits01[p3], + ) + return appendMicrosecs(dst, src[2:], int(length)-20), nil +} + +func formatBinaryTime(src []byte, length uint8) (driver.Value, error) { + // length expects the deterministic length of the zero value, + // negative time and 100+ hours are automatically added if needed + if len(src) == 0 { + return zeroDateTime[11 : 11+length], nil + } + var dst []byte // return value + + switch length { + case + 8, // time (can be up to 10 when negative and 100+ hours) + 10, 11, 12, 13, 14, 15: // time with fractional seconds + default: + return nil, fmt.Errorf("illegal TIME length %d", length) + } + switch len(src) { + case 8, 12: + default: + return nil, fmt.Errorf("invalid TIME packet length %d", len(src)) + } + // +2 to enable negative time and 100+ hours + dst = make([]byte, 0, length+2) + if src[0] == 1 { + dst = append(dst, '-') + } + days := binary.LittleEndian.Uint32(src[1:5]) + hours := int64(days)*24 + int64(src[5]) + + if hours >= 100 { + dst = strconv.AppendInt(dst, hours, 10) + } else { + dst = append(dst, digits10[hours], digits01[hours]) + } + + min, sec := src[6], src[7] + dst = append(dst, ':', + digits10[min], digits01[min], ':', + digits10[sec], digits01[sec], + ) + return appendMicrosecs(dst, src[8:], int(length)-9), nil +} + +/****************************************************************************** +* Convert from and to bytes * +******************************************************************************/ + +// 24bit integer: used for packet headers. + +func putUint24(data []byte, n int) { + data[2] = byte(n >> 16) + data[1] = byte(n >> 8) + data[0] = byte(n) +} + +func getUint24(data []byte) int { + return int(data[2])<<16 | int(data[1])<<8 | int(data[0]) +} + +func uint64ToString(n uint64) []byte { + var a [20]byte + i := 20 + + // U+0030 = 0 + // ... + // U+0039 = 9 + + var q uint64 + for n >= 10 { + i-- + q = n / 10 + a[i] = uint8(n-q*10) + 0x30 + n = q + } + + i-- + a[i] = uint8(n) + 0x30 + + return a[i:] +} + +// returns the string read as a bytes slice, whether the value is NULL, +// the number of bytes read and an error, in case the string is longer than +// the input slice +func readLengthEncodedString(b []byte) ([]byte, bool, int, error) { + // Get length + num, isNull, n := readLengthEncodedInteger(b) + if num < 1 { + return b[n:n], isNull, n, nil + } + + n += int(num) + + // Check data length + if len(b) >= n { + return b[n-int(num) : n : n], false, n, nil + } + return nil, false, n, io.EOF +} + +// returns the number of bytes skipped and an error, in case the string is +// longer than the input slice +func skipLengthEncodedString(b []byte) (int, error) { + // Get length + num, _, n := readLengthEncodedInteger(b) + if num < 1 { + return n, nil + } + + n += int(num) + + // Check data length + if len(b) >= n { + return n, nil + } + return n, io.EOF +} + +// returns the number read, whether the value is NULL and the number of bytes read +func readLengthEncodedInteger(b []byte) (uint64, bool, int) { + // See issue #349 + if len(b) == 0 { + return 0, true, 1 + } + + switch b[0] { + // 251: NULL + case 0xfb: + return 0, true, 1 + + // 252: value of following 2 + case 0xfc: + return uint64(binary.LittleEndian.Uint16(b[1:])), false, 3 + + // 253: value of following 3 + case 0xfd: + return uint64(getUint24(b[1:])), false, 4 + + // 254: value of following 8 + case 0xfe: + return uint64(binary.LittleEndian.Uint64(b[1:])), false, 9 + } + + // 0-250: value of first byte + return uint64(b[0]), false, 1 +} + +// encodes a uint64 value and appends it to the given bytes slice +func appendLengthEncodedInteger(b []byte, n uint64) []byte { + switch { + case n <= 250: + return append(b, byte(n)) + + case n <= 0xffff: + b = append(b, 0xfc) + return binary.LittleEndian.AppendUint16(b, uint16(n)) + + case n <= 0xffffff: + return append(b, 0xfd, byte(n), byte(n>>8), byte(n>>16)) + } + b = append(b, 0xfe) + return binary.LittleEndian.AppendUint64(b, n) +} + +func appendLengthEncodedString(b []byte, s string) []byte { + b = appendLengthEncodedInteger(b, uint64(len(s))) + return append(b, s...) +} + +// reserveBuffer checks cap(buf) and expand buffer to len(buf) + appendSize. +// If cap(buf) is not enough, reallocate new buffer. +func reserveBuffer(buf []byte, appendSize int) []byte { + newSize := len(buf) + appendSize + if cap(buf) < newSize { + // Grow buffer exponentially + newBuf := make([]byte, len(buf)*2+appendSize) + copy(newBuf, buf) + buf = newBuf + } + return buf[:newSize] +} + +// Lookup table for backslash escapes (used for both string and bytes) +var backslashEscapeTable [256]byte + +func init() { + backslashEscapeTable['\x00'] = '0' + backslashEscapeTable['\n'] = 'n' + backslashEscapeTable['\r'] = 'r' + backslashEscapeTable['\x1a'] = 'Z' + backslashEscapeTable['\''] = '\'' + backslashEscapeTable['"'] = '"' + backslashEscapeTable['\\'] = '\\' +} + +// escapeStringBackslash is similar to escapeBytesBackslash but for string. +func escapeStringBackslash(buf []byte, v string) []byte { + pos := len(buf) + buf = reserveBuffer(buf, len(v)*2+2) + buf[pos] = '\'' + pos++ + for i := 0; i < len(v); i++ { + c := v[i] + if esc := backslashEscapeTable[c]; esc != 0 { + buf[pos+1] = esc + buf[pos] = '\\' + pos += 2 + } else { + buf[pos] = c + pos++ + } + } + buf[pos] = '\'' + pos++ + return buf[:pos] +} + +// escapeBytesBackslash appends _binary'...' or '...' with backslash escaping for bytes. +func escapeBytesBackslash(buf, v []byte, binary bool) []byte { + pos := len(buf) + if binary { + buf = reserveBuffer(buf, len(v)*2+9) + copy(buf[pos:], []byte("_binary'")) + pos += 8 + } else { + buf = reserveBuffer(buf, len(v)*2+2) + buf[pos] = '\'' + pos++ + } + for _, c := range v { + if esc := backslashEscapeTable[c]; esc != 0 { + buf[pos+1] = esc + buf[pos] = '\\' + pos += 2 + } else { + buf[pos] = c + pos++ + } + } + buf[pos] = '\'' + pos++ + return buf[:pos] +} + +// escapeBytesQuotes appends _binary'...' or '...' with single-quote escaping for bytes. +func escapeBytesQuotes(buf, v []byte, binary bool) []byte { + pos := len(buf) + if binary { + buf = reserveBuffer(buf, len(v)*2+9) + copy(buf[pos:], []byte("_binary'")) + pos += 8 + } else { + buf = reserveBuffer(buf, len(v)*2+2) + buf[pos] = '\'' + pos++ + } + for _, c := range v { + if c == '\'' { + buf[pos+1] = '\'' + buf[pos] = '\'' + pos += 2 + } else { + buf[pos] = c + pos++ + } + } + buf[pos] = '\'' + pos++ + return buf[:pos] +} + +// escapeStringQuotes is similar to escapeBytesQuotes but for string. +func escapeStringQuotes(buf []byte, v string) []byte { + pos := len(buf) + buf = reserveBuffer(buf, len(v)*2+2) + buf[pos] = '\'' + pos++ + for i := range len(v) { + c := v[i] + if c == '\'' { + buf[pos+1] = '\'' + buf[pos] = '\'' + pos += 2 + } else { + buf[pos] = c + pos++ + } + } + buf[pos] = '\'' + pos++ + return buf[:pos] +} + +/****************************************************************************** +* Sync utils * +******************************************************************************/ + +// noCopy may be embedded into structs which must not be copied +// after the first use. +// +// See https://github.com/golang/go/issues/8005#issuecomment-190753527 +// for details. +type noCopy struct{} + +// Lock is a no-op used by -copylocks checker from `go vet`. +func (*noCopy) Lock() {} + +// Unlock is a no-op used by -copylocks checker from `go vet`. +// noCopy should implement sync.Locker from Go 1.11 +// https://github.com/golang/go/commit/c2eba53e7f80df21d51285879d51ab81bcfbf6bc +// https://github.com/golang/go/issues/26165 +func (*noCopy) Unlock() {} + +// atomicError is a wrapper for atomically accessed error values +type atomicError struct { + _ noCopy + value atomic.Value +} + +// Set sets the error value regardless of the previous value. +// The value must not be nil +func (ae *atomicError) Set(value error) { + ae.value.Store(value) +} + +// Value returns the current error value +func (ae *atomicError) Value() error { + if v := ae.value.Load(); v != nil { + // this will panic if the value doesn't implement the error interface + return v.(error) + } + return nil +} + +func namedValueToValue(named []driver.NamedValue) ([]driver.Value, error) { + dargs := make([]driver.Value, len(named)) + for n, param := range named { + if len(param.Name) > 0 { + // TODO: support the use of Named Parameters #561 + return nil, errors.New("mysql: driver does not support the use of Named Parameters") + } + dargs[n] = param.Value + } + return dargs, nil +} + +func mapIsolationLevel(level driver.IsolationLevel) (string, error) { + switch sql.IsolationLevel(level) { + case sql.LevelRepeatableRead: + return "REPEATABLE READ", nil + case sql.LevelReadCommitted: + return "READ COMMITTED", nil + case sql.LevelReadUncommitted: + return "READ UNCOMMITTED", nil + case sql.LevelSerializable: + return "SERIALIZABLE", nil + default: + return "", fmt.Errorf("mysql: unsupported isolation level: %v", level) + } +} diff --git a/_LICENSES/github.com/go-sql-driver/mysql/utils_test.go b/_LICENSES/github.com/go-sql-driver/mysql/utils_test.go new file mode 100644 index 0000000000..4c171f6216 --- /dev/null +++ b/_LICENSES/github.com/go-sql-driver/mysql/utils_test.go @@ -0,0 +1,555 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "database/sql" + "database/sql/driver" + "encoding/binary" + "testing" + "time" +) + +func TestLengthEncodedInteger(t *testing.T) { + var integerTests = []struct { + num uint64 + encoded []byte + }{ + {0x0000000000000000, []byte{0x00}}, + {0x0000000000000012, []byte{0x12}}, + {0x00000000000000fa, []byte{0xfa}}, + {0x0000000000000100, []byte{0xfc, 0x00, 0x01}}, + {0x0000000000001234, []byte{0xfc, 0x34, 0x12}}, + {0x000000000000ffff, []byte{0xfc, 0xff, 0xff}}, + {0x0000000000010000, []byte{0xfd, 0x00, 0x00, 0x01}}, + {0x0000000000123456, []byte{0xfd, 0x56, 0x34, 0x12}}, + {0x0000000000ffffff, []byte{0xfd, 0xff, 0xff, 0xff}}, + {0x0000000001000000, []byte{0xfe, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00}}, + {0x123456789abcdef0, []byte{0xfe, 0xf0, 0xde, 0xbc, 0x9a, 0x78, 0x56, 0x34, 0x12}}, + {0xffffffffffffffff, []byte{0xfe, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}}, + } + + for _, tst := range integerTests { + num, isNull, numLen := readLengthEncodedInteger(tst.encoded) + if isNull { + t.Errorf("%x: expected %d, got NULL", tst.encoded, tst.num) + } + if num != tst.num { + t.Errorf("%x: expected %d, got %d", tst.encoded, tst.num, num) + } + if numLen != len(tst.encoded) { + t.Errorf("%x: expected size %d, got %d", tst.encoded, len(tst.encoded), numLen) + } + encoded := appendLengthEncodedInteger(nil, num) + if !bytes.Equal(encoded, tst.encoded) { + t.Errorf("%v: expected %x, got %x", num, tst.encoded, encoded) + } + } +} + +func TestFormatBinaryDateTime(t *testing.T) { + rawDate := [11]byte{} + binary.LittleEndian.PutUint16(rawDate[:2], 1978) // years + rawDate[2] = 12 // months + rawDate[3] = 30 // days + rawDate[4] = 15 // hours + rawDate[5] = 46 // minutes + rawDate[6] = 23 // seconds + binary.LittleEndian.PutUint32(rawDate[7:], 987654) // microseconds + expect := func(expected string, inlen, outlen uint8) { + actual, _ := formatBinaryDateTime(rawDate[:inlen], outlen) + bytes, ok := actual.([]byte) + if !ok { + t.Errorf("formatBinaryDateTime must return []byte, was %T", actual) + } + if string(bytes) != expected { + t.Errorf( + "expected %q, got %q for length in %d, out %d", + expected, actual, inlen, outlen, + ) + } + } + expect("0000-00-00", 0, 10) + expect("0000-00-00 00:00:00", 0, 19) + expect("1978-12-30", 4, 10) + expect("1978-12-30 15:46:23", 7, 19) + expect("1978-12-30 15:46:23.987654", 11, 26) +} + +func TestFormatBinaryTime(t *testing.T) { + expect := func(expected string, src []byte, outlen uint8) { + actual, _ := formatBinaryTime(src, outlen) + bytes, ok := actual.([]byte) + if !ok { + t.Errorf("formatBinaryDateTime must return []byte, was %T", actual) + } + if string(bytes) != expected { + t.Errorf( + "expected %q, got %q for src=%q and outlen=%d", + expected, actual, src, outlen) + } + } + + // binary format: + // sign (0: positive, 1: negative), days(4), hours, minutes, seconds, micro(4) + + // Zeros + expect("00:00:00", []byte{}, 8) + expect("00:00:00.0", []byte{}, 10) + expect("00:00:00.000000", []byte{}, 15) + + // Without micro(4) + expect("12:34:56", []byte{0, 0, 0, 0, 0, 12, 34, 56}, 8) + expect("-12:34:56", []byte{1, 0, 0, 0, 0, 12, 34, 56}, 8) + expect("12:34:56.00", []byte{0, 0, 0, 0, 0, 12, 34, 56}, 11) + expect("24:34:56", []byte{0, 1, 0, 0, 0, 0, 34, 56}, 8) + expect("-99:34:56", []byte{1, 4, 0, 0, 0, 3, 34, 56}, 8) + expect("103079215103:34:56", []byte{0, 255, 255, 255, 255, 23, 34, 56}, 8) + + // With micro(4) + expect("12:34:56.00", []byte{0, 0, 0, 0, 0, 12, 34, 56, 99, 0, 0, 0}, 11) + expect("12:34:56.000099", []byte{0, 0, 0, 0, 0, 12, 34, 56, 99, 0, 0, 0}, 15) +} + +func TestEscapeBackslash(t *testing.T) { + expect := func(expected, value string) { + actual := string(escapeBytesBackslash([]byte{}, []byte(value), false)) + if actual != expected { + t.Errorf( + "expected %s, got %s", + expected, actual, + ) + } + + actual = string(escapeStringBackslash([]byte{}, value)) + if actual != expected { + t.Errorf( + "expected %s, got %s", + expected, actual, + ) + } + } + + expect("'foo\\0bar'", "foo\x00bar") + expect("'foo\\nbar'", "foo\nbar") + expect("'foo\\rbar'", "foo\rbar") + expect("'foo\\Zbar'", "foo\x1abar") + expect("'foo\\\"bar'", "foo\"bar") + expect("'foo\\\\bar'", "foo\\bar") + expect("'foo\\'bar'", "foo'bar") + + // Test binary flag for escapeBytesBackslash + binExpect := func(expected, value string) { + actual := string(escapeBytesBackslash([]byte{}, []byte(value), true)) + if actual != expected { + t.Errorf( + "expected %s, got %s (binary)", + expected, actual, + ) + } + } + binExpect("_binary'foo\\0bar'", "foo\x00bar") + binExpect("_binary'foo\\nbar'", "foo\nbar") + binExpect("_binary'foo\\rbar'", "foo\rbar") + binExpect("_binary'foo\\Zbar'", "foo\x1abar") + binExpect("_binary'foo\\\"bar'", "foo\"bar") + binExpect("_binary'foo\\\\bar'", "foo\\bar") + binExpect("_binary'foo\\'bar'", "foo'bar") +} + +func TestEscapeQuotes(t *testing.T) { + expect := func(expected, value string) { + actual := string(escapeBytesQuotes([]byte{}, []byte(value), false)) + if actual != expected { + t.Errorf( + "expected %s, got %s", + expected, actual, + ) + } + + actual = string(escapeStringQuotes([]byte{}, value)) + if actual != expected { + t.Errorf( + "expected %s, got %s", + expected, actual, + ) + } + } + + expect("'foo\x00bar'", "foo\x00bar") // not affected + expect("'foo\nbar'", "foo\nbar") // not affected + expect("'foo\rbar'", "foo\rbar") // not affected + expect("'foo\x1abar'", "foo\x1abar") // not affected + expect("'foo''bar'", "foo'bar") // affected + expect("'foo\"bar'", "foo\"bar") // not affected + + // Test binary flag for escapeBytesQuotes + binExpect := func(expected, value string) { + actual := string(escapeBytesQuotes([]byte{}, []byte(value), true)) + if actual != expected { + t.Errorf( + "expected %s, got %s (binary)", + expected, actual, + ) + } + } + binExpect("_binary'foo\x00bar'", "foo\x00bar") + binExpect("_binary'foo\nbar'", "foo\nbar") + binExpect("_binary'foo\rbar'", "foo\rbar") + binExpect("_binary'foo\x1abar'", "foo\x1abar") + binExpect("_binary'foo''bar'", "foo'bar") + binExpect("_binary'foo\"bar'", "foo\"bar") +} + +func TestAtomicError(t *testing.T) { + var ae atomicError + if ae.Value() != nil { + t.Fatal("Expected value to be nil") + } + + ae.Set(ErrMalformPkt) + if v := ae.Value(); v != ErrMalformPkt { + if v == nil { + t.Fatal("Value is still nil") + } + t.Fatal("Error did not match") + } + ae.Set(ErrPktSync) + if ae.Value() == ErrMalformPkt { + t.Fatal("Error still matches old error") + } + if v := ae.Value(); v != ErrPktSync { + t.Fatal("Error did not match") + } +} + +func TestIsolationLevelMapping(t *testing.T) { + data := []struct { + level driver.IsolationLevel + expected string + }{ + { + level: driver.IsolationLevel(sql.LevelReadCommitted), + expected: "READ COMMITTED", + }, + { + level: driver.IsolationLevel(sql.LevelRepeatableRead), + expected: "REPEATABLE READ", + }, + { + level: driver.IsolationLevel(sql.LevelReadUncommitted), + expected: "READ UNCOMMITTED", + }, + { + level: driver.IsolationLevel(sql.LevelSerializable), + expected: "SERIALIZABLE", + }, + } + + for i, td := range data { + if actual, err := mapIsolationLevel(td.level); actual != td.expected || err != nil { + t.Fatal(i, td.expected, actual, err) + } + } + + // check unsupported mapping + expectedErr := "mysql: unsupported isolation level: 7" + actual, err := mapIsolationLevel(driver.IsolationLevel(sql.LevelLinearizable)) + if actual != "" || err == nil { + t.Fatal("Expected error on unsupported isolation level") + } + if err.Error() != expectedErr { + t.Fatalf("Expected error to be %q, got %q", expectedErr, err) + } +} + +func TestAppendDateTime(t *testing.T) { + tests := []struct { + t time.Time + str string + timeTruncate time.Duration + expectedErr bool + }{ + { + t: time.Date(1234, 5, 6, 0, 0, 0, 0, time.UTC), + str: "1234-05-06", + }, + { + t: time.Date(4567, 12, 31, 12, 0, 0, 0, time.UTC), + str: "4567-12-31 12:00:00", + }, + { + t: time.Date(2020, 5, 30, 12, 34, 0, 0, time.UTC), + str: "2020-05-30 12:34:00", + }, + { + t: time.Date(2020, 5, 30, 12, 34, 56, 0, time.UTC), + str: "2020-05-30 12:34:56", + }, + { + t: time.Date(2020, 5, 30, 22, 33, 44, 123000000, time.UTC), + str: "2020-05-30 22:33:44.123", + }, + { + t: time.Date(2020, 5, 30, 22, 33, 44, 123456000, time.UTC), + str: "2020-05-30 22:33:44.123456", + }, + { + t: time.Date(2020, 5, 30, 22, 33, 44, 123456789, time.UTC), + str: "2020-05-30 22:33:44.123456789", + }, + { + t: time.Date(9999, 12, 31, 23, 59, 59, 999999999, time.UTC), + str: "9999-12-31 23:59:59.999999999", + }, + { + t: time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC), + str: "0001-01-01", + }, + // Truncated time + { + t: time.Date(1234, 5, 6, 0, 0, 0, 0, time.UTC), + str: "1234-05-06", + timeTruncate: time.Second, + }, + { + t: time.Date(4567, 12, 31, 12, 0, 0, 0, time.UTC), + str: "4567-12-31 12:00:00", + timeTruncate: time.Minute, + }, + { + t: time.Date(2020, 5, 30, 12, 34, 0, 0, time.UTC), + str: "2020-05-30 12:34:00", + timeTruncate: 0, + }, + { + t: time.Date(2020, 5, 30, 12, 34, 56, 0, time.UTC), + str: "2020-05-30 12:34:56", + timeTruncate: time.Second, + }, + { + t: time.Date(2020, 5, 30, 22, 33, 44, 123000000, time.UTC), + str: "2020-05-30 22:33:44", + timeTruncate: time.Second, + }, + { + t: time.Date(2020, 5, 30, 22, 33, 44, 123456000, time.UTC), + str: "2020-05-30 22:33:44.123", + timeTruncate: time.Millisecond, + }, + { + t: time.Date(2020, 5, 30, 22, 33, 44, 123456789, time.UTC), + str: "2020-05-30 22:33:44", + timeTruncate: time.Second, + }, + { + t: time.Date(9999, 12, 31, 23, 59, 59, 999999999, time.UTC), + str: "9999-12-31 23:59:59.999999999", + timeTruncate: 0, + }, + { + t: time.Date(1, 1, 1, 1, 1, 1, 1, time.UTC), + str: "0001-01-01", + timeTruncate: 365 * 24 * time.Hour, + }, + // year out of range + { + t: time.Date(0, 1, 1, 0, 0, 0, 0, time.UTC), + expectedErr: true, + }, + { + t: time.Date(10000, 1, 1, 0, 0, 0, 0, time.UTC), + expectedErr: true, + }, + } + for _, v := range tests { + buf := make([]byte, 0, 32) + buf, err := appendDateTime(buf, v.t, v.timeTruncate) + if err != nil { + if !v.expectedErr { + t.Errorf("appendDateTime(%v) returned an error: %v", v.t, err) + } + continue + } + if str := string(buf); str != v.str { + t.Errorf("appendDateTime(%v), have: %s, want: %s", v.t, str, v.str) + } + } +} + +func TestParseDateTime(t *testing.T) { + cases := []struct { + name string + str string + }{ + { + name: "parse date", + str: "2020-05-13", + }, + { + name: "parse null date", + str: sDate0, + }, + { + name: "parse datetime", + str: "2020-05-13 21:30:45", + }, + { + name: "parse null datetime", + str: sDateTime0, + }, + { + name: "parse datetime nanosec 1-digit", + str: "2020-05-25 23:22:01.1", + }, + { + name: "parse datetime nanosec 2-digits", + str: "2020-05-25 23:22:01.15", + }, + { + name: "parse datetime nanosec 3-digits", + str: "2020-05-25 23:22:01.159", + }, + { + name: "parse datetime nanosec 4-digits", + str: "2020-05-25 23:22:01.1594", + }, + { + name: "parse datetime nanosec 5-digits", + str: "2020-05-25 23:22:01.15949", + }, + { + name: "parse datetime nanosec 6-digits", + str: "2020-05-25 23:22:01.159491", + }, + } + + for _, loc := range []*time.Location{ + time.UTC, + time.FixedZone("test", 8*60*60), + } { + for _, cc := range cases { + t.Run(cc.name+"-"+loc.String(), func(t *testing.T) { + var want time.Time + if cc.str != sDate0 && cc.str != sDateTime0 { + var err error + want, err = time.ParseInLocation(timeFormat[:len(cc.str)], cc.str, loc) + if err != nil { + t.Fatal(err) + } + } + got, err := parseDateTime([]byte(cc.str), loc) + if err != nil { + t.Fatal(err) + } + + if !want.Equal(got) { + t.Fatalf("want: %v, but got %v", want, got) + } + }) + } + } +} + +func TestInvalidDateTime(t *testing.T) { + cases := []struct { + name string + str string + want time.Time + }{ + { + name: "parse datetime without day", + str: "0000-00-00 21:30:45", + want: time.Date(0, 0, 0, 21, 30, 45, 0, time.UTC), + }, + } + + for _, cc := range cases { + t.Run(cc.name, func(t *testing.T) { + got, err := parseDateTime([]byte(cc.str), time.UTC) + if err != nil { + t.Fatal(err) + } + + if !cc.want.Equal(got) { + t.Fatalf("want: %v, but got %v", cc.want, got) + } + }) + } +} + +func TestParseDateTimeFail(t *testing.T) { + cases := []struct { + name string + str string + wantErr string + }{ + { + name: "parse invalid time", + str: "hello", + wantErr: "invalid time bytes: hello", + }, + { + name: "parse year", + str: "000!-00-00 00:00:00.000000", + wantErr: "not [0-9]", + }, + { + name: "parse month", + str: "0000-!0-00 00:00:00.000000", + wantErr: "not [0-9]", + }, + { + name: `parse "-" after parsed year`, + str: "0000:00-00 00:00:00.000000", + wantErr: "bad value for field: `:`", + }, + { + name: `parse "-" after parsed month`, + str: "0000-00:00 00:00:00.000000", + wantErr: "bad value for field: `:`", + }, + { + name: `parse " " after parsed date`, + str: "0000-00-00+00:00:00.000000", + wantErr: "bad value for field: `+`", + }, + { + name: `parse ":" after parsed date`, + str: "0000-00-00 00-00:00.000000", + wantErr: "bad value for field: `-`", + }, + { + name: `parse ":" after parsed hour`, + str: "0000-00-00 00:00-00.000000", + wantErr: "bad value for field: `-`", + }, + { + name: `parse "." after parsed sec`, + str: "0000-00-00 00:00:00?000000", + wantErr: "bad value for field: `?`", + }, + } + + for _, cc := range cases { + t.Run(cc.name, func(t *testing.T) { + got, err := parseDateTime([]byte(cc.str), time.UTC) + if err == nil { + t.Fatal("want error") + } + if cc.wantErr != err.Error() { + t.Fatalf("want `%s`, but got `%s`", cc.wantErr, err) + } + if !got.IsZero() { + t.Fatal("want zero time") + } + }) + } +} diff --git a/_LICENSES/github.com/testcontainers/testcontainers-go/modules/mysql/LICENSE b/_LICENSES/github.com/testcontainers/testcontainers-go/modules/mysql/LICENSE new file mode 100644 index 0000000000..607a9c3c46 --- /dev/null +++ b/_LICENSES/github.com/testcontainers/testcontainers-go/modules/mysql/LICENSE @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2017-2019 Gianluca Arbezzano + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/cmd/ate-setup/cli-diff.md b/cmd/ate-setup/cli-diff.md index 69eeffe46d..69910970c9 100644 --- a/cmd/ate-setup/cli-diff.md +++ b/cmd/ate-setup/cli-diff.md @@ -23,7 +23,8 @@ Deliberately, and covered by `shim_test.go`: - Every environment variable, including `SETUP_CSI`, `STORAGE_CLASS`, `NO_DEV_ENV`, `ATE_INSTALL_KIND`, `KUBECTL_CONTEXT`, `PROJECT_ID`, and the `ATE_API_POSTGRES_*` set. `ate-setup` reads them directly, so they need no - translation. + translation. The same holds for the `ATE_API_STORE_BACKEND` and + `ATE_API_MYSQL_*` variables. - `--help` / `-h` anywhere on the line, no arguments at all (usage, exit 1), and an unrecognized flag (`Error: unknown option: …`, usage, exit 1) — including after earlier actions on the same line have already run. diff --git a/cmd/ate-setup/commands.md b/cmd/ate-setup/commands.md index 160dc5d6b1..7ad4700475 100644 --- a/cmd/ate-setup/commands.md +++ b/cmd/ate-setup/commands.md @@ -99,6 +99,12 @@ apiserver, the controller, atenet, and atelet. It creates every `create` resource below on the way, so those subcommands are only needed to redo one on a running cluster. +The store is PostgreSQL by default. `ATE_API_STORE_BACKEND=mysql` with +`ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING` selects an existing external MySQL +instead, skips the bundled StatefulSet, and makes `deploy postgres` refuse to +run. See [`differences.md`](differences.md#known-differences-worth-flagging) +for the MySQL variables. + ## Publish | `ate-setup` | `hack/install-ate.sh` | @@ -132,6 +138,9 @@ Individual secrets and config that `deploy ate-system` creates automatically. | `create api-server-env-vars` | `--create-api-server-env-vars` | | `create api-authentication-config` | `--create-api-authentication-config` | +`create api-server-env-vars` writes the store backend and the connection +settings of that backend only. + ## Setup | `ate-setup` | `hack/install-ate.sh` | diff --git a/cmd/ate-setup/differences.md b/cmd/ate-setup/differences.md index 07104b6d6d..14c47746ac 100644 --- a/cmd/ate-setup/differences.md +++ b/cmd/ate-setup/differences.md @@ -311,12 +311,32 @@ never exported the variables leaves a working proxy alone. Exporting it empty removes the sidecar and the annotation and falls back to the bundled StatefulSet. +**MySQL is new in `ate-setup`.** The shell installer only knew PostgreSQL. +`ATE_API_STORE_BACKEND=mysql` points ate-api-server at an existing MySQL 8.0+ +or PlanetScale database through `ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING` +(required) and `ATE_API_MYSQL_OWNER_CONNECTION_STRING`. There is no bundled +MySQL, so the PostgreSQL StatefulSet is skipped. + +Each backend rejects the other's non-empty variables; exported empty ones are +allowed. A MySQL install never adopts the cluster's Cloud SQL record, so it +removes a proxy sidecar left by an earlier PostgreSQL install. + +`ATE_API_MYSQL_SERVER_CA_FILE` is published as the `mysql-server-ca` Secret and +turns on TLS verified against that CA. `ATE_API_MYSQL_TLS_CA_FILE`, +`ATE_API_MYSQL_TLS_CERT_FILE`, and `ATE_API_MYSQL_TLS_KEY_FILE` pass paths +inside the pod through unchanged, as a PostgreSQL DSN names its certificate +files. + +The backend is recorded in `ate-api-server-secret-envvars`. A redeploy with +`ATE_API_STORE_BACKEND` unset fails on a cluster that records `mysql`, rather +than moving it onto an empty bundled PostgreSQL. + ## Testing The shell installer had no tests. `cmd/ate-setup` has unit tests for template rendering, overlay selection, config resolution, the authentication config, the -apiserver environment ConfigMap and Secret, Cloud SQL resolution, the OTLP -endpoint override, delegated script arguments, manifest deletion, per-demo -rendering, image reference rewriting, kustomize composition, the control plane -pinning on every apply path, and the size10 PostgreSQL resize against the real -manifests. +apiserver environment ConfigMap and Secret for each store backend, Cloud SQL +resolution, the OTLP endpoint override, delegated script arguments, manifest +deletion, per-demo rendering, image reference rewriting, kustomize composition, +the control plane pinning on every apply path, and the size10 PostgreSQL resize +against the real manifests. diff --git a/cmd/ate-setup/internal/cmd/deploy.go b/cmd/ate-setup/internal/cmd/deploy.go index 579f91e229..6980247fb3 100644 --- a/cmd/ate-setup/internal/cmd/deploy.go +++ b/cmd/ate-setup/internal/cmd/deploy.go @@ -39,7 +39,8 @@ the atelet DaemonSet, then waits for each to roll out. The bundled PostgreSQL StatefulSet is skipped when ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING or the ATE_API_POSTGRES_CLOUDSQL_* variables -select an external database. +select an external database, or when ATE_API_STORE_BACKEND=mysql selects an +external MySQL. Shape the install with the global --atenet-dataplane, --cluster-size, and --cordon-control-plane flags.`, @@ -123,7 +124,7 @@ var deployPostgresCmd = &cobra.Command{ "deploy ate-system" already brings PostgreSQL up, unless ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING or the ATE_API_POSTGRES_CLOUDSQL_* variables select an external database; this subcommand is for bringing the StatefulSet up -by itself.`, +by itself. It refuses to run with ATE_API_STORE_BACKEND=mysql.`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { return env.DeployPostgres(cmd.Context()) diff --git a/cmd/ate-setup/internal/config/config.go b/cmd/ate-setup/internal/config/config.go index 96b6af4130..f4ba392bf0 100644 --- a/cmd/ate-setup/internal/config/config.go +++ b/cmd/ate-setup/internal/config/config.go @@ -72,6 +72,12 @@ const ( DefaultPostgresOwnerRole = "substrate_owner" ) +// Store backends, the values ATE_API_STORE_BACKEND accepts. +const ( + StoreBackendPostgres = "postgres" + StoreBackendMySQL = "mysql" +) + // Cloud SQL Auth Proxy IP types, the values ATE_API_POSTGRES_CLOUDSQL_IP_TYPE // accepts. const ( @@ -136,6 +142,16 @@ type Config struct { // Router selects the atenet router dataplane. Router string + // StoreBackend is the database ateapi keeps its state in + // (ATE_API_STORE_BACKEND): StoreBackendPostgres or StoreBackendMySQL. + StoreBackend string + // StoreBackendSet records whether ATE_API_STORE_BACKEND was given rather + // than defaulted, so that a redeploy that does not name a backend cannot + // silently move a MySQL install onto the bundled PostgreSQL. + StoreBackendSet bool + // StorePoolMaxConns sizes the apiserver's read/write pool for either + // backend (ATE_API_STORE_POOL_MAX_CONNS). Empty leaves the backend default. + StorePoolMaxConns string // The read/write and owner connections can use different login identities. // With one configured connection, both pools use it. Both empty selects // bundled PostgreSQL. @@ -150,9 +166,6 @@ type Config struct { // PostgresSchema is the PostgreSQL schema for the Substrate tables // (ATE_API_POSTGRES_SCHEMA). Empty means DefaultPostgresSchema. PostgresSchema string - // PostgresPoolMaxConns sizes the apiserver's read/write pool - // (ATE_API_POSTGRES_POOL_MAX_CONNS). Empty leaves the DSN or pgxpool default. - PostgresPoolMaxConns string // PostgresServerCAFile is a local PEM file holding the server CA of an // external PostgreSQL (ATE_API_POSTGRES_SERVER_CA_FILE). Its contents are // published as the postgres-server-ca Secret, which ate-api-server mounts @@ -162,6 +175,25 @@ type Config struct { // Proxy sidecar instead of a directly reachable PostgreSQL. CloudSQL CloudSQLConfig + // MySQL settings, used only when StoreBackend is StoreBackendMySQL. The + // database is always external and must already exist. The owner DSN + // defaults to the read/write one. + MySQLReadWriteConnectionString string + MySQLOwnerConnectionString string + // MySQLServerCAFile is a local PEM file holding the MySQL server CA + // (ATE_API_MYSQL_SERVER_CA_FILE). Its contents are published as the + // mysql-server-ca Secret, which ate-api-server mounts at + // /run/mysql-server-ca/server-ca.pem. + MySQLServerCAFile string + // MySQL TLS files already inside the ate-api-server pod, such as the pod + // identity credential bundle (ATE_API_MYSQL_TLS_CA_FILE, + // ATE_API_MYSQL_TLS_CERT_FILE, ATE_API_MYSQL_TLS_KEY_FILE). They play the + // part of the sslrootcert, sslcert and sslkey paths a PostgreSQL DSN names, + // which a MySQL DSN cannot carry. + MySQLTLSCAFile string + MySQLTLSCertFile string + MySQLTLSKeyFile string + // RolloutTimeout is the timeout duration for rollout status checks. RolloutTimeout time.Duration // rolloutTimeoutSet records whether RolloutTimeout was asked for rather @@ -334,6 +366,7 @@ func Load(opts Options) (*Config, error) { kubeconfig, kubeconfigEnv := loadKubeconfig(opts.Kubeconfig, env["KUBECONFIG"]) readWriteConnectionString := env["ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING"] ownerConnectionString := firstNonEmpty(env["ATE_API_POSTGRES_OWNER_CONNECTION_STRING"], readWriteConnectionString) + mysqlReadWriteConnectionString := env["ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING"] cfg := &Config{ Root: root, @@ -349,6 +382,9 @@ func Load(opts Options) (*Config, error) { KODockerRepo: env["KO_DOCKER_REPO"], KODefaultPlatforms: env["KO_DEFAULTPLATFORMS"], Images: loadImageSource(opts, env), + StoreBackend: firstNonEmpty(env["ATE_API_STORE_BACKEND"], StoreBackendPostgres), + StoreBackendSet: env["ATE_API_STORE_BACKEND"] != "", + StorePoolMaxConns: env["ATE_API_STORE_POOL_MAX_CONNS"], PostgresReadWriteConnectionString: readWriteConnectionString, PostgresOwnerConnectionString: ownerConnectionString, PostgresReadWriteRole: firstNonEmpty(env["ATE_API_POSTGRES_READ_WRITE_ROLE"], DefaultPostgresReadWriteRole), @@ -356,7 +392,6 @@ func Load(opts Options) (*Config, error) { PostgresReadWriteRoleSet: env["ATE_API_POSTGRES_READ_WRITE_ROLE"] != "", PostgresOwnerRoleSet: env["ATE_API_POSTGRES_OWNER_ROLE"] != "", PostgresSchema: env["ATE_API_POSTGRES_SCHEMA"], - PostgresPoolMaxConns: env["ATE_API_POSTGRES_POOL_MAX_CONNS"], PostgresServerCAFile: env["ATE_API_POSTGRES_SERVER_CA_FILE"], CloudSQL: CloudSQLConfig{ Instance: cloudsqlInstance, @@ -365,6 +400,12 @@ func Load(opts Options) (*Config, error) { IAMAuth: env["ATE_API_POSTGRES_CLOUDSQL_IAM_AUTH"], IPType: env["ATE_API_POSTGRES_CLOUDSQL_IP_TYPE"], }, + MySQLReadWriteConnectionString: mysqlReadWriteConnectionString, + MySQLOwnerConnectionString: firstNonEmpty(env["ATE_API_MYSQL_OWNER_CONNECTION_STRING"], mysqlReadWriteConnectionString), + MySQLServerCAFile: env["ATE_API_MYSQL_SERVER_CA_FILE"], + MySQLTLSCAFile: env["ATE_API_MYSQL_TLS_CA_FILE"], + MySQLTLSCertFile: env["ATE_API_MYSQL_TLS_CERT_FILE"], + MySQLTLSKeyFile: env["ATE_API_MYSQL_TLS_KEY_FILE"], RolloutTimeout: rolloutTimeout, rolloutTimeoutSet: timeoutStr != "", PodcertWorkersPerSigner: podcertWorkers, @@ -388,9 +429,58 @@ func Load(opts Options) (*Config, error) { if err := validate(cfg); err != nil { return nil, err } + if err := validateStoreBackend(cfg, env); err != nil { + return nil, err + } + if cfg.StoreBackend == StoreBackendMySQL { + // An explicitly empty instance rather than an unset one: a MySQL + // install never adopts the cluster's recorded Cloud SQL instance, so + // a proxy sidecar left by an earlier PostgreSQL install is removed. + cfg.CloudSQL = CloudSQLConfig{InstanceSet: true} + } return cfg, nil } +// validateStoreBackend checks the backend choice and refuses settings for the +// other backend, which would otherwise be ignored without a word. The prefix +// check covers the Cloud SQL variables too: on MySQL they would keep a proxy +// sidecar for a database the apiserver no longer uses. An exported but empty +// ATE_API_POSTGRES_CLOUDSQL_INSTANCE is allowed, since it asks for removal. +func validateStoreBackend(cfg *Config, env map[string]string) error { + var conflictPrefix string + switch cfg.StoreBackend { + case StoreBackendPostgres: + conflictPrefix = "ATE_API_MYSQL_" + case StoreBackendMySQL: + conflictPrefix = "ATE_API_POSTGRES_" + if cfg.MySQLReadWriteConnectionString == "" { + return fmt.Errorf("ATE_API_STORE_BACKEND=%s requires ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING; "+ + "there is no bundled MySQL", StoreBackendMySQL) + } + if cfg.MySQLServerCAFile != "" && cfg.MySQLTLSCAFile != "" { + return fmt.Errorf("set ATE_API_MYSQL_SERVER_CA_FILE (a local file to upload) or ATE_API_MYSQL_TLS_CA_FILE (a path in the pod), not both") + } + if (cfg.MySQLTLSCertFile == "") != (cfg.MySQLTLSKeyFile == "") { + return fmt.Errorf("ATE_API_MYSQL_TLS_CERT_FILE and ATE_API_MYSQL_TLS_KEY_FILE must be set together") + } + default: + return fmt.Errorf("ATE_API_STORE_BACKEND must be %s or %s, got %q", + StoreBackendPostgres, StoreBackendMySQL, cfg.StoreBackend) + } + var conflicts []string + for name, value := range env { + if strings.HasPrefix(name, conflictPrefix) && value != "" { + conflicts = append(conflicts, name) + } + } + if len(conflicts) > 0 { + sort.Strings(conflicts) + return fmt.Errorf("ATE_API_STORE_BACKEND=%s conflicts with %s; unset them or select the other backend", + cfg.StoreBackend, strings.Join(conflicts, ", ")) + } + return nil +} + // loadKubeconfig splits the kubeconfig setting into the path handed to // client-go and the value exported to the shell scripts. // @@ -603,6 +693,11 @@ func (c *Config) PostgresSchemaName() string { return DefaultPostgresSchema } +// MySQL reports whether ateapi keeps its state in MySQL. +func (c *Config) MySQL() bool { + return c.StoreBackend == StoreBackendMySQL +} + // WaitTimeout returns how long to wait for a workload whose historical timeout // was historical. // diff --git a/cmd/ate-setup/internal/config/config_test.go b/cmd/ate-setup/internal/config/config_test.go index f70be4ce1a..fca5da7bee 100644 --- a/cmd/ate-setup/internal/config/config_test.go +++ b/cmd/ate-setup/internal/config/config_test.go @@ -15,6 +15,7 @@ package config import ( + "maps" "os" "path/filepath" "runtime" @@ -45,11 +46,18 @@ func loadEnv(t *testing.T) { "ATE_API_POSTGRES_CLOUDSQL_IP_TYPE", "ATE_API_POSTGRES_OWNER_CONNECTION_STRING", "ATE_API_POSTGRES_OWNER_ROLE", - "ATE_API_POSTGRES_POOL_MAX_CONNS", "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING", "ATE_API_POSTGRES_READ_WRITE_ROLE", "ATE_API_POSTGRES_SCHEMA", "ATE_API_POSTGRES_SERVER_CA_FILE", + "ATE_API_MYSQL_OWNER_CONNECTION_STRING", + "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING", + "ATE_API_MYSQL_SERVER_CA_FILE", + "ATE_API_MYSQL_TLS_CA_FILE", + "ATE_API_MYSQL_TLS_CERT_FILE", + "ATE_API_MYSQL_TLS_KEY_FILE", + "ATE_API_STORE_BACKEND", + "ATE_API_STORE_POOL_MAX_CONNS", "ATE_ATENET_DATAPLANE", "ATE_CREDENTIAL_PROVIDER", "ATE_IMAGE_REPO", @@ -115,6 +123,9 @@ func TestLoadDefaults(t *testing.T) { if cfg.CordonControlPlane { t.Error("CordonControlPlane = true, want false") } + if cfg.StoreBackend != StoreBackendPostgres || cfg.StoreBackendSet || cfg.MySQL() { + t.Errorf("StoreBackend = %q (set %v), want a defaulted %q", cfg.StoreBackend, cfg.StoreBackendSet, StoreBackendPostgres) + } } func TestLoadClusterSize(t *testing.T) { @@ -287,15 +298,15 @@ func TestLoadPostgresSchema(t *testing.T) { // mounted file respectively, neither of which the shell installer synthesizes. func TestLoadPostgresTuning(t *testing.T) { loadEnv(t) - t.Setenv("ATE_API_POSTGRES_POOL_MAX_CONNS", "50") + t.Setenv("ATE_API_STORE_POOL_MAX_CONNS", "50") t.Setenv("ATE_API_POSTGRES_SERVER_CA_FILE", "/etc/ssl/server-ca.pem") cfg, err := Load(Options{}) if err != nil { t.Fatalf("Load() error = %v", err) } - if cfg.PostgresPoolMaxConns != "50" { - t.Errorf("PostgresPoolMaxConns = %q, want 50", cfg.PostgresPoolMaxConns) + if cfg.StorePoolMaxConns != "50" { + t.Errorf("StorePoolMaxConns = %q, want 50", cfg.StorePoolMaxConns) } if want := "/etc/ssl/server-ca.pem"; cfg.PostgresServerCAFile != want { t.Errorf("PostgresServerCAFile = %q, want %q", cfg.PostgresServerCAFile, want) @@ -363,6 +374,79 @@ func TestLoadCloudSQL(t *testing.T) { }) } +func TestLoadMySQL(t *testing.T) { + const dsn = "runtime:pw@tcp(db:3306)/substrate?tls=true" + t.Run("one login", func(t *testing.T) { + loadEnv(t) + t.Setenv("ATE_API_STORE_BACKEND", StoreBackendMySQL) + t.Setenv("ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING", dsn) + + cfg, err := Load(Options{}) + if err != nil { + t.Fatalf("Load() error = %v", err) + } + if !cfg.MySQL() || !cfg.StoreBackendSet { + t.Errorf("StoreBackend = %q (set %v), want an explicit %q", cfg.StoreBackend, cfg.StoreBackendSet, StoreBackendMySQL) + } + if cfg.MySQLReadWriteConnectionString != dsn || cfg.MySQLOwnerConnectionString != dsn { + t.Errorf("MySQL connections = %q, %q, want %q for both", cfg.MySQLReadWriteConnectionString, cfg.MySQLOwnerConnectionString, dsn) + } + // Explicitly empty, so the cluster's recorded instance is not adopted + // and a leftover proxy sidecar is removed. + if want := (CloudSQLConfig{InstanceSet: true}); cfg.CloudSQL != want { + t.Errorf("CloudSQL = %+v, want %+v", cfg.CloudSQL, want) + } + }) + + t.Run("exported but empty Cloud SQL instance is allowed", func(t *testing.T) { + loadEnv(t) + t.Setenv("ATE_API_STORE_BACKEND", StoreBackendMySQL) + t.Setenv("ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING", dsn) + t.Setenv("ATE_API_POSTGRES_CLOUDSQL_INSTANCE", "") + if _, err := Load(Options{}); err != nil { + t.Fatalf("Load() error = %v", err) + } + }) +} + +func TestLoadRejectsInvalidStoreBackend(t *testing.T) { + const dsn = "runtime:pw@tcp(db:3306)/substrate" + mysql := map[string]string{ + "ATE_API_STORE_BACKEND": StoreBackendMySQL, + "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING": dsn, + } + with := func(base map[string]string, name, value string) map[string]string { + env := maps.Clone(base) + env[name] = value + return env + } + for _, tc := range []struct { + name string + env map[string]string + wantErr string + }{ + {"unknown backend", map[string]string{"ATE_API_STORE_BACKEND": "sqlite"}, "ATE_API_STORE_BACKEND must be"}, + {"MySQL without a DSN", map[string]string{"ATE_API_STORE_BACKEND": StoreBackendMySQL}, "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING"}, + {"MySQL with a Cloud SQL instance", with(mysql, "ATE_API_POSTGRES_CLOUDSQL_INSTANCE", "p:r:i"), "ATE_API_POSTGRES_CLOUDSQL_INSTANCE"}, + {"MySQL with both CA settings", with(with(mysql, "ATE_API_MYSQL_SERVER_CA_FILE", "/ca.pem"), "ATE_API_MYSQL_TLS_CA_FILE", "/run/ca.pem"), "not both"}, + {"MySQL client certificate without a key", with(mysql, "ATE_API_MYSQL_TLS_CERT_FILE", "/run/cert.pem"), "set together"}, + // Without the backend switch the DSN would be ignored and the bundled + // PostgreSQL deployed in its place. + {"PostgreSQL with a MySQL DSN", map[string]string{"ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING": dsn}, "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING"}, + } { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + for name, value := range tc.env { + t.Setenv(name, value) + } + _, err := Load(Options{}) + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("Load() error = %v, want it to contain %q", err, tc.wantErr) + } + }) + } +} + // EXPECTED_JWT_ISSUER overrides the issuer derived from the GKE coordinates, // which is how a cluster authenticating against something other than its own // OIDC discovery document is installed. diff --git a/cmd/ate-setup/internal/steps/apienv.go b/cmd/ate-setup/internal/steps/apienv.go index 895fb5a74c..177bebb66a 100644 --- a/cmd/ate-setup/internal/steps/apienv.go +++ b/cmd/ate-setup/internal/steps/apienv.go @@ -32,36 +32,90 @@ import ( // is what turns a new DSN into a restart. const envHashAnnotation = "ate.dev/env-hash" -// CreateAPIServerEnvVars reconciles how ate-api-server reaches its PostgreSQL -// store: both DSNs and the schema into the ate-api-server-secret-envvars Secret, -// stable roles and Cloud SQL settings into the ConfigMap, -// and an external server CA into postgres-server-ca. +// mysqlServerCAPath is where ate-api-server.yaml mounts the mysql-server-ca +// Secret's server-ca.pem. +const mysqlServerCAPath = "/run/mysql-server-ca/server-ca.pem" + +// CreateAPIServerEnvVars reconciles how ate-api-server reaches its store. The +// connection strings, the backend selector, and the PostgreSQL schema go into +// the ate-api-server-secret-envvars Secret, other settings into the ConfigMap, +// and an external server CA into its own Secret. // -// ate-api-server.yaml pulls both in through optional envFrom sources and -// resolves the PostgreSQL connection, role, and schema flags from -// the result. It lists the secretRef last, so the Secret wins over a DSN a -// previous installer left in the ConfigMap. +// ate-api-server.yaml pulls both in through optional envFrom sources and lists +// the secretRef last, so the Secret wins over a DSN a previous installer left +// in the ConfigMap. Each backend writes only its own keys. func (e *Env) CreateAPIServerEnvVars(ctx context.Context) error { log.Step("create_api_server_env_vars") if err := e.Kube.EnsureNamespace(ctx, e.Namespace()); err != nil { return err } + if err := e.checkRecordedStoreBackend(ctx); err != nil { + return err + } + + var configVars, secretVars map[string]string + if e.Cfg.MySQL() { + configVars, secretVars = mysqlAPIServerEnvVars(e.Cfg) + } else { + var err error + if configVars, secretVars, err = e.postgresAPIServerEnvVars(ctx); err != nil { + return err + } + } + if err := e.Kube.ApplyConfigMap(ctx, e.Namespace(), ConfigMapAPIEnvVars, configVars); err != nil { + return err + } + if err := e.Kube.ApplySecret(ctx, e.Namespace(), SecretAPIEnvVars, secretVars); err != nil { + return err + } + if err := e.applyServerCA(ctx, e.Cfg.PostgresServerCAFile, "ATE_API_POSTGRES_SERVER_CA_FILE", SecretPostgresServerCA); err != nil { + return err + } + if err := e.applyServerCA(ctx, e.Cfg.MySQLServerCAFile, "ATE_API_MYSQL_SERVER_CA_FILE", SecretMySQLServerCA); err != nil { + return err + } + return e.annotateAPIServerEnvHash(ctx) +} +// checkRecordedStoreBackend refuses to move a cluster off the backend its +// Secret records when ATE_API_STORE_BACKEND was left at its default. Without +// it, a redeploy of a MySQL install from a shell that never exported the +// variable would point the apiserver at a new, empty bundled PostgreSQL. +func (e *Env) checkRecordedStoreBackend(ctx context.Context) error { + if e.Cfg.StoreBackendSet { + return nil + } + secret, err := e.Kube.GetSecret(ctx, e.Namespace(), SecretAPIEnvVars) + if err != nil || secret == nil { + return err + } + recorded := string(secret.Data[envStoreBackend]) + if recorded == "" || recorded == e.Cfg.StoreBackend { + return nil + } + return fmt.Errorf("ate-api-server uses the %s store backend and ATE_API_STORE_BACKEND is unset; "+ + "set ATE_API_STORE_BACKEND=%s with its connection settings to keep it, or ATE_API_STORE_BACKEND=%s to switch", + recorded, recorded, e.Cfg.StoreBackend) +} + +// postgresAPIServerEnvVars resolves the PostgreSQL ConfigMap and Secret +// payloads, adopting what the cluster records for Cloud SQL. +func (e *Env) postgresAPIServerEnvVars(ctx context.Context) (configVars, secretVars map[string]string, err error) { readWriteDSN := e.Cfg.PostgresReadWriteConnectionString ownerDSN := e.Cfg.PostgresOwnerConnectionString readWriteRole := e.Cfg.PostgresReadWriteRole ownerRole := e.Cfg.PostgresOwnerRole - poolMaxConns := e.Cfg.PostgresPoolMaxConns + poolMaxConns := e.Cfg.StorePoolMaxConns cloudsql, err := e.resolveCloudSQL(ctx) if err != nil { - return err + return nil, nil, err } if readWriteDSN == "" && cloudsql.Adopted { // Fill missing credentials from the adopted Cloud SQL configuration. recordedReadWriteDSN, recordedOwnerDSN, err := e.recordedConnectionStrings(ctx) if err != nil { - return err + return nil, nil, err } readWriteDSN = recordedReadWriteDSN if ownerDSN == "" { @@ -71,7 +125,7 @@ func (e *Env) CreateAPIServerEnvVars(ctx context.Context) error { if readWriteDSN == "" { if cloudsql.Instance != "" { if readWriteDSN, err = cloudSQLDSN(cloudsql); err != nil { - return err + return nil, nil, err } } else { readWriteDSN = config.DefaultPostgresConnectionString @@ -95,7 +149,7 @@ func (e *Env) CreateAPIServerEnvVars(ctx context.Context) error { if cloudsql.Adopted { recorded, err := e.recordedAPIServerEnvVars(ctx) if err != nil { - return err + return nil, nil, err } if !e.Cfg.PostgresReadWriteRoleSet && recorded["ATE_API_POSTGRES_READ_WRITE_ROLE"] != "" { readWriteRole = recorded["ATE_API_POSTGRES_READ_WRITE_ROLE"] @@ -104,52 +158,76 @@ func (e *Env) CreateAPIServerEnvVars(ctx context.Context) error { ownerRole = recorded["ATE_API_POSTGRES_OWNER_ROLE"] } if poolMaxConns == "" { - poolMaxConns = recorded["ATE_API_POSTGRES_POOL_MAX_CONNS"] + poolMaxConns = recorded["ATE_API_STORE_POOL_MAX_CONNS"] } if e.Cfg.PostgresSchema == "" { secret, err := e.Kube.GetSecret(ctx, e.Namespace(), SecretAPIEnvVars) if err != nil { - return err + return nil, nil, err } if secret != nil && len(secret.Data["ATE_API_POSTGRES_SCHEMA"]) != 0 { schema = string(secret.Data["ATE_API_POSTGRES_SCHEMA"]) } } } - configVars := cloudSQLEnvVars(cloudsql) + configVars = cloudSQLEnvVars(cloudsql) configVars["ATE_API_POSTGRES_READ_WRITE_ROLE"] = readWriteRole configVars["ATE_API_POSTGRES_OWNER_ROLE"] = ownerRole if poolMaxConns != "" { - configVars["ATE_API_POSTGRES_POOL_MAX_CONNS"] = poolMaxConns - } - if err := e.Kube.ApplyConfigMap(ctx, e.Namespace(), ConfigMapAPIEnvVars, configVars); err != nil { - return err - } - if err := e.Kube.ApplySecret(ctx, e.Namespace(), SecretAPIEnvVars, - buildAPIServerEnvVars(readWriteDSN, ownerDSN, schema)); err != nil { - return err + configVars["ATE_API_STORE_POOL_MAX_CONNS"] = poolMaxConns } - if err := e.applyPostgresServerCA(ctx); err != nil { - return err - } - return e.annotateAPIServerEnvHash(ctx) + return configVars, buildAPIServerEnvVars(readWriteDSN, ownerDSN, schema), nil } -// buildAPIServerEnvVars is the Secret payload. ate-api-server takes both -// connection strings and the schema from it, and exits on an empty schema; an -// unrecognized key here reaches the container as a stray environment variable. +// envStoreBackend selects ateapi's backend. It sits in the Secret with the +// DSNs, so the record of which backend the DSNs belong to moves with them. +const envStoreBackend = "ATE_API_STORE_BACKEND" + +// buildAPIServerEnvVars is the PostgreSQL Secret payload. ate-api-server takes +// both connection strings and the schema from it, and exits on an empty +// schema; an unrecognized key here reaches the container as a stray +// environment variable. // // The DSN can carry a password, for an external database without IAM // authentication, which is why this is a Secret and not the ConfigMap // alongside it. func buildAPIServerEnvVars(readWriteDSN, ownerDSN, schema string) map[string]string { return map[string]string{ + envStoreBackend: config.StoreBackendPostgres, "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING": readWriteDSN, "ATE_API_POSTGRES_OWNER_CONNECTION_STRING": ownerDSN, "ATE_API_POSTGRES_SCHEMA": schema, } } +// mysqlAPIServerEnvVars is the MySQL ConfigMap and Secret payload. The DSNs +// carry passwords, so they go in the Secret. The ConfigMap holds no Cloud SQL +// keys, which prunes any an earlier PostgreSQL install left behind. +func mysqlAPIServerEnvVars(cfg *config.Config) (configVars, secretVars map[string]string) { + configVars = map[string]string{} + if cfg.StorePoolMaxConns != "" { + configVars["ATE_API_STORE_POOL_MAX_CONNS"] = cfg.StorePoolMaxConns + } + for name, value := range map[string]string{ + "ATE_API_MYSQL_TLS_CA_FILE": cfg.MySQLTLSCAFile, + "ATE_API_MYSQL_TLS_CERT_FILE": cfg.MySQLTLSCertFile, + "ATE_API_MYSQL_TLS_KEY_FILE": cfg.MySQLTLSKeyFile, + } { + if value != "" { + configVars[name] = value + } + } + if cfg.MySQLServerCAFile != "" { + configVars["ATE_API_MYSQL_TLS_CA_FILE"] = mysqlServerCAPath + } + secretVars = map[string]string{ + envStoreBackend: config.StoreBackendMySQL, + "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING": cfg.MySQLReadWriteConnectionString, + "ATE_API_MYSQL_OWNER_CONNECTION_STRING": cfg.MySQLOwnerConnectionString, + } + return configVars, secretVars +} + // recordedConnectionStrings reads the credentials paired with an adopted Cloud // SQL instance. func (e *Env) recordedConnectionStrings(ctx context.Context) (string, string, error) { @@ -165,21 +243,20 @@ func (e *Env) recordedConnectionStrings(ctx context.Context) (string, string, er return readWriteDSN, ownerDSN, nil } -// applyPostgresServerCA publishes the server CA of an external PostgreSQL, -// which ate-api-server mounts at /run/postgres-server-ca/server-ca.pem for -// sslmode=verify-ca DSNs. For Cloud SQL: +// applyServerCA publishes the server CA of an external database, read from +// the file the env variable names, into secret. ate-api-server mounts it at +// /run//server-ca.pem. For Cloud SQL: // // gcloud sql ssl server-ca-certs list --instance= --format="value(cert)" -func (e *Env) applyPostgresServerCA(ctx context.Context) error { - path := e.Cfg.PostgresServerCAFile +func (e *Env) applyServerCA(ctx context.Context, path, env, secret string) error { if path == "" { return nil } pem, err := os.ReadFile(path) if err != nil { - return fmt.Errorf("reading ATE_API_POSTGRES_SERVER_CA_FILE: %w", err) + return fmt.Errorf("reading %s: %w", env, err) } - return e.Kube.ApplySecret(ctx, e.Namespace(), SecretPostgresServerCA, map[string]string{ + return e.Kube.ApplySecret(ctx, e.Namespace(), secret, map[string]string{ "server-ca.pem": string(pem), }) } diff --git a/cmd/ate-setup/internal/steps/apienv_test.go b/cmd/ate-setup/internal/steps/apienv_test.go index eb288a3818..05737255c0 100644 --- a/cmd/ate-setup/internal/steps/apienv_test.go +++ b/cmd/ate-setup/internal/steps/apienv_test.go @@ -15,14 +15,21 @@ package steps import ( + "maps" + "os" + "path" + "path/filepath" + "slices" "strings" "testing" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" ) // The digest exists to turn an envFrom change into a rollout, so what matters @@ -147,7 +154,7 @@ func TestCreateAPIServerEnvVarsPostgresIdentities(t *testing.T) { func TestCreateAPIServerEnvVarsPoolSize(t *testing.T) { cfg := config.Config{ PostgresReadWriteConnectionString: "postgres://runtime@postgres/atepg", - PostgresPoolMaxConns: "20", + StorePoolMaxConns: "20", } e := &Env{Cfg: &cfg, Kube: fakeKube(t, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}}, @@ -165,9 +172,9 @@ func TestCreateAPIServerEnvVarsPoolSize(t *testing.T) { if err != nil { t.Fatal(err) } - if cm.Data["ATE_API_POSTGRES_POOL_MAX_CONNS"] != "20" || + if cm.Data["ATE_API_STORE_POOL_MAX_CONNS"] != "20" || secret.StringData["ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING"] != cfg.PostgresReadWriteConnectionString { - t.Fatalf("pool size %q, connection %q", cm.Data["ATE_API_POSTGRES_POOL_MAX_CONNS"], + t.Fatalf("pool size %q, connection %q", cm.Data["ATE_API_STORE_POOL_MAX_CONNS"], secret.StringData["ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING"]) } } @@ -205,7 +212,7 @@ func TestCreateAPIServerEnvVarsAdoptsPostgresIdentity(t *testing.T) { PostgresOwnerRoleSet: true, PostgresOwnerConnectionString: explicitOwnerDSN, PostgresSchema: "other_schema", - PostgresPoolMaxConns: "30", + StorePoolMaxConns: "30", }, wantReadWrite: config.DefaultPostgresReadWriteRole, wantOwner: config.DefaultPostgresOwnerRole, @@ -223,7 +230,7 @@ func TestCreateAPIServerEnvVarsAdoptsPostgresIdentity(t *testing.T) { "ATE_API_POSTGRES_CLOUDSQL_INSTANCE": "p:r:i", "ATE_API_POSTGRES_READ_WRITE_ROLE": "tenant_readwrite", "ATE_API_POSTGRES_OWNER_ROLE": "tenant_owner", - "ATE_API_POSTGRES_POOL_MAX_CONNS": "20", + "ATE_API_STORE_POOL_MAX_CONNS": "20", }, }, &corev1.Secret{ @@ -255,13 +262,13 @@ func TestCreateAPIServerEnvVarsAdoptsPostgresIdentity(t *testing.T) { } if cm.Data["ATE_API_POSTGRES_READ_WRITE_ROLE"] != tc.wantReadWrite || cm.Data["ATE_API_POSTGRES_OWNER_ROLE"] != tc.wantOwner || - cm.Data["ATE_API_POSTGRES_POOL_MAX_CONNS"] != tc.wantPoolSize || + cm.Data["ATE_API_STORE_POOL_MAX_CONNS"] != tc.wantPoolSize || secret.StringData["ATE_API_POSTGRES_SCHEMA"] != tc.wantSchema || secret.StringData["ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING"] != recordedDSN || secret.StringData["ATE_API_POSTGRES_OWNER_CONNECTION_STRING"] != tc.wantOwnerDSN { t.Fatalf("identity after redeploy: roles %q/%q, schema %q, pool size %q, connections %q/%q", cm.Data["ATE_API_POSTGRES_READ_WRITE_ROLE"], cm.Data["ATE_API_POSTGRES_OWNER_ROLE"], - secret.StringData["ATE_API_POSTGRES_SCHEMA"], cm.Data["ATE_API_POSTGRES_POOL_MAX_CONNS"], + secret.StringData["ATE_API_POSTGRES_SCHEMA"], cm.Data["ATE_API_STORE_POOL_MAX_CONNS"], secret.StringData["ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING"], secret.StringData["ATE_API_POSTGRES_OWNER_CONNECTION_STRING"]) } @@ -385,3 +392,162 @@ func TestAnnotateAPIServerEnvHash(t *testing.T) { } }) } + +// Switching a Cloud SQL PostgreSQL install to MySQL writes the MySQL contract +// and does not adopt the recorded instance. The fake clientset merges an apply +// over keys it did not write, so the stale keys seeded here survive; on a +// cluster, server-side apply prunes the ones ate-setup wrote earlier. +func TestCreateAPIServerEnvVarsMySQL(t *testing.T) { + const readWriteDSN = "runtime:pw@tcp(db:3306)/substrate?tls=true" + caFile := filepath.Join(t.TempDir(), "ca.pem") + if err := os.WriteFile(caFile, []byte("-----BEGIN CERTIFICATE-----\n"), 0o600); err != nil { + t.Fatal(err) + } + cfg := config.Config{ + StoreBackend: config.StoreBackendMySQL, + StoreBackendSet: true, + MySQLReadWriteConnectionString: readWriteDSN, + MySQLOwnerConnectionString: readWriteDSN, + MySQLServerCAFile: caFile, + CloudSQL: config.CloudSQLConfig{InstanceSet: true}, + } + e := &Env{Cfg: &cfg, Kube: fakeKube(t, + &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespaceAteSystem}}, + apiServerEnvVarsConfigMap(map[string]string{ + envCloudSQLInstance: "p:r:i", + "ATE_API_POSTGRES_READ_WRITE_ROLE": "tenant_readwrite", + }), + &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: SecretAPIEnvVars, Namespace: NamespaceAteSystem}, + Data: map[string][]byte{ + "ATE_API_STORE_BACKEND": []byte(config.StoreBackendPostgres), + "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING": []byte("user=svc@p.iam host=127.0.0.1"), + }, + }, + // The fake clientset applies only over an existing object. + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: SecretMySQLServerCA, Namespace: NamespaceAteSystem}}, + )} + if err := e.CreateAPIServerEnvVars(t.Context()); err != nil { + t.Fatal(err) + } + + cm, err := e.Kube.GetConfigMap(t.Context(), NamespaceAteSystem, ConfigMapAPIEnvVars) + if err != nil { + t.Fatal(err) + } + if _, ok := cm.Data["CSQL_PROXY_PORT"]; ok { + t.Errorf("ConfigMap data = %v, want no Cloud SQL proxy settings on MySQL", cm.Data) + } + secret, err := e.Kube.GetSecret(t.Context(), NamespaceAteSystem, SecretAPIEnvVars) + if err != nil { + t.Fatal(err) + } + wantSecret := map[string]string{ + "ATE_API_STORE_BACKEND": config.StoreBackendMySQL, + "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING": readWriteDSN, + "ATE_API_MYSQL_OWNER_CONNECTION_STRING": readWriteDSN, + } + if !maps.Equal(secret.StringData, wantSecret) { + t.Errorf("Secret data = %v, want %v", secret.StringData, wantSecret) + } + ca, err := e.Kube.GetSecret(t.Context(), NamespaceAteSystem, SecretMySQLServerCA) + if err != nil { + t.Fatal(err) + } + if ca == nil || ca.StringData["server-ca.pem"] != "-----BEGIN CERTIFICATE-----\n" { + t.Errorf("%s = %+v, want the CA file's contents under server-ca.pem", SecretMySQLServerCA, ca) + } +} + +// A redeploy that leaves ATE_API_STORE_BACKEND unset must not move a MySQL +// install onto an empty bundled PostgreSQL. +func TestCheckRecordedStoreBackend(t *testing.T) { + defaulted := config.Config{StoreBackend: config.StoreBackendPostgres} + for _, tc := range []struct { + name string + cfg config.Config + noSecret bool + recorded string + wantErr bool + }{ + {name: "fresh install", cfg: defaulted, noSecret: true}, + {name: "default matches the record", cfg: defaulted, recorded: config.StoreBackendPostgres}, + {name: "record predates the backend key", cfg: defaulted}, + {name: "default would switch away from MySQL", cfg: defaulted, recorded: config.StoreBackendMySQL, wantErr: true}, + { + name: "explicit switch to PostgreSQL", + cfg: config.Config{StoreBackend: config.StoreBackendPostgres, StoreBackendSet: true}, + recorded: config.StoreBackendMySQL, + }, + } { + t.Run(tc.name, func(t *testing.T) { + var objects []runtime.Object + if !tc.noSecret { + secret := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: SecretAPIEnvVars, Namespace: NamespaceAteSystem}} + if tc.recorded != "" { + secret.Data = map[string][]byte{envStoreBackend: []byte(tc.recorded)} + } + objects = append(objects, secret) + } + e := &Env{Cfg: &tc.cfg, Kube: fakeKube(t, objects...)} + err := e.checkRecordedStoreBackend(t.Context()) + if gotErr := err != nil; gotErr != tc.wantErr { + t.Fatalf("checkRecordedStoreBackend() error = %v, wantErr %v", err, tc.wantErr) + } + if err != nil && !strings.Contains(err.Error(), "ATE_API_STORE_BACKEND=mysql") { + t.Errorf("checkRecordedStoreBackend() error = %q, want it to name the setting that keeps MySQL", err) + } + }) + } +} + +// ate-api-server.yaml has to read every MySQL setting the installer writes and +// mount the Secret at the path ATE_API_MYSQL_TLS_CA_FILE names. +func TestAPIServerManifestReadsMySQLSettings(t *testing.T) { + data, err := os.ReadFile(filepath.Join(repoRoot(t), "manifests", "ate-install", "ate-api-server.yaml")) + if err != nil { + t.Fatal(err) + } + objs, err := kube.DecodeManifestBytes(data) + if err != nil { + t.Fatal(err) + } + obj := findObject(objs, "Deployment", "ate-api-server") + if obj == nil { + t.Fatal("ate-api-server.yaml has no deployment/ate-api-server") + } + var dep appsv1.Deployment + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(obj.Object, &dep); err != nil { + t.Fatal(err) + } + container := dep.Spec.Template.Spec.Containers[0] + for _, arg := range []string{ + "--mysql-read-write-connection-string=@env", + "--mysql-owner-connection-string=@env", + "--mysql-tls-ca-file=@env", + "--mysql-tls-cert-file=@env", + "--mysql-tls-key-file=@env", + } { + if !slices.Contains(container.Args, arg) { + t.Errorf("ate-api-server args lack %s", arg) + } + } + var mounted bool + for _, m := range container.VolumeMounts { + if m.Name == SecretMySQLServerCA { + mounted = m.ReadOnly && m.MountPath == path.Dir(mysqlServerCAPath) + } + } + if !mounted { + t.Errorf("ate-api-server does not mount %s read-only at %s", SecretMySQLServerCA, path.Dir(mysqlServerCAPath)) + } + var optional bool + for _, v := range dep.Spec.Template.Spec.Volumes { + if v.Name == SecretMySQLServerCA && v.Secret != nil && v.Secret.SecretName == SecretMySQLServerCA { + optional = v.Secret.Optional != nil && *v.Secret.Optional + } + } + if !optional { + t.Errorf("volume %s is not the optional %s Secret; PostgreSQL installs never create it", SecretMySQLServerCA, SecretMySQLServerCA) + } +} diff --git a/cmd/ate-setup/internal/steps/create.go b/cmd/ate-setup/internal/steps/create.go index 4b69cfe191..95401ff18f 100644 --- a/cmd/ate-setup/internal/steps/create.go +++ b/cmd/ate-setup/internal/steps/create.go @@ -38,6 +38,7 @@ const ( SecretEgressMITMCAPool = "egress-mitm-ca-pool" SecretAPIEnvVars = "ate-api-server-secret-envvars" SecretPostgresServerCA = "postgres-server-ca" + SecretMySQLServerCA = "mysql-server-ca" ConfigMapAPIEnvVars = "ate-api-server-envvars" ConfigMapAPIAuthn = "ate-api-authentication" // poolKeyID is the identifier given to the first CA and JWT key in a new diff --git a/cmd/ate-setup/internal/steps/create_test.go b/cmd/ate-setup/internal/steps/create_test.go index ffab153750..af563353f4 100644 --- a/cmd/ate-setup/internal/steps/create_test.go +++ b/cmd/ate-setup/internal/steps/create_test.go @@ -24,18 +24,19 @@ import ( "github.com/google/go-cmp/cmp" corev1 "k8s.io/api/core/v1" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" "github.com/agent-substrate/substrate/internal/localca" ) // ate-api-server requires both connection strings and its schema in the -// credential-bearing Secret. +// credential-bearing Secret, along with the backend they belong to. func TestBuildAPIServerEnvVars(t *testing.T) { const readWriteDSN = "postgresql://readwrite@postgres/atepg" const ownerDSN = "postgresql://owner@postgres/atepg" got := buildAPIServerEnvVars(readWriteDSN, ownerDSN, "public") - want := []string{"ATE_API_POSTGRES_OWNER_CONNECTION_STRING", "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING", "ATE_API_POSTGRES_SCHEMA"} + want := []string{"ATE_API_POSTGRES_OWNER_CONNECTION_STRING", "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING", "ATE_API_POSTGRES_SCHEMA", "ATE_API_STORE_BACKEND"} if keys := slices.Sorted(maps.Keys(got)); !slices.Equal(keys, want) { t.Errorf("keys = %v, want %v", keys, want) } @@ -45,6 +46,41 @@ func TestBuildAPIServerEnvVars(t *testing.T) { if got["ATE_API_POSTGRES_SCHEMA"] != "public" { t.Errorf("ATE_API_POSTGRES_SCHEMA = %q, want %q", got["ATE_API_POSTGRES_SCHEMA"], "public") } + if got["ATE_API_STORE_BACKEND"] != config.StoreBackendPostgres { + t.Errorf("ATE_API_STORE_BACKEND = %q, want %q", got["ATE_API_STORE_BACKEND"], config.StoreBackendPostgres) + } +} + +// The MySQL contract: DSNs and the backend in the Secret, tuning and the CA +// path in the ConfigMap, and no PostgreSQL or Cloud SQL key in either. +func TestMySQLAPIServerEnvVars(t *testing.T) { + const readWriteDSN = "runtime:pw@tcp(db:3306)/substrate?tls=true" + const ownerDSN = "owner:pw@tcp(db:3306)/substrate?tls=true" + configVars, secretVars := mysqlAPIServerEnvVars(&config.Config{ + MySQLReadWriteConnectionString: readWriteDSN, + MySQLOwnerConnectionString: ownerDSN, + StorePoolMaxConns: "40", + MySQLServerCAFile: "/local/ca.pem", + MySQLTLSCertFile: "/run/podidentity.podcert.ate.dev/credential-bundle.pem", + MySQLTLSKeyFile: "/run/podidentity.podcert.ate.dev/credential-bundle.pem", + }) + wantConfig := map[string]string{ + "ATE_API_STORE_POOL_MAX_CONNS": "40", + "ATE_API_MYSQL_TLS_CA_FILE": "/run/mysql-server-ca/server-ca.pem", + "ATE_API_MYSQL_TLS_CERT_FILE": "/run/podidentity.podcert.ate.dev/credential-bundle.pem", + "ATE_API_MYSQL_TLS_KEY_FILE": "/run/podidentity.podcert.ate.dev/credential-bundle.pem", + } + if diff := cmp.Diff(wantConfig, configVars); diff != "" { + t.Errorf("ConfigMap data differs (-want +got):\n%s", diff) + } + wantSecret := map[string]string{ + "ATE_API_STORE_BACKEND": config.StoreBackendMySQL, + "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING": readWriteDSN, + "ATE_API_MYSQL_OWNER_CONNECTION_STRING": ownerDSN, + } + if diff := cmp.Diff(wantSecret, secretVars); diff != "" { + t.Errorf("Secret data differs (-want +got):\n%s", diff) + } } // The expected strings here are what the shell installer's diff --git a/cmd/ate-setup/internal/steps/postgres.go b/cmd/ate-setup/internal/steps/postgres.go index 6657799332..42bcb5f5ef 100644 --- a/cmd/ate-setup/internal/steps/postgres.go +++ b/cmd/ate-setup/internal/steps/postgres.go @@ -23,6 +23,7 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "sigs.k8s.io/yaml" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/log" ) @@ -61,8 +62,11 @@ type postgresPlan struct { // planPostgres decides between the bundled database and an external one, // configured either as an explicit DSN or as a Cloud SQL instance — the -// latter possibly adopted from the cluster. +// latter possibly adopted from the cluster. A MySQL store is always external. func (e *Env) planPostgres(ctx context.Context) (postgresPlan, error) { + if e.Cfg.MySQL() { + return postgresPlan{external: "ATE_API_STORE_BACKEND=" + config.StoreBackendMySQL}, nil + } if e.Cfg.PostgresReadWriteConnectionString != "" { return postgresPlan{external: "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING"}, nil } @@ -228,6 +232,10 @@ func applyPostgresSize10Overrides(objs []*unstructured.Unstructured, confPatch [ // StatefulSet on its own. func (e *Env) DeployPostgres(ctx context.Context) error { log.Step("deploy_postgres") + if e.Cfg.MySQL() { + return fmt.Errorf("ATE_API_STORE_BACKEND=%s keeps ateapi's state in an external MySQL; "+ + "the bundled PostgreSQL would go unused", config.StoreBackendMySQL) + } if err := e.EnsureAteSystemNamespace(ctx); err != nil { return err diff --git a/cmd/ate-setup/internal/steps/postgres_test.go b/cmd/ate-setup/internal/steps/postgres_test.go index 52714b95a0..f51dfecfb9 100644 --- a/cmd/ate-setup/internal/steps/postgres_test.go +++ b/cmd/ate-setup/internal/steps/postgres_test.go @@ -152,6 +152,7 @@ func TestApplyPostgresSize10OverridesRejectsMissingObjects(t *testing.T) { func TestPlanPostgres(t *testing.T) { for _, tc := range []struct { name string + backend string connString string cloudSQL config.CloudSQLConfig recorded map[string]string @@ -184,10 +185,20 @@ func TestPlanPostgres(t *testing.T) { recorded: map[string]string{envCloudSQLInstance: "p:r:i"}, want: postgresPlan{bundled: true}, }, + { + // There is no bundled MySQL, and a recorded Cloud SQL instance + // from an earlier PostgreSQL install does not matter. + name: "MySQL", + backend: config.StoreBackendMySQL, + cloudSQL: config.CloudSQLConfig{InstanceSet: true}, + recorded: map[string]string{envCloudSQLInstance: "p:r:i"}, + want: postgresPlan{external: "ATE_API_STORE_BACKEND=mysql"}, + }, } { t.Run(tc.name, func(t *testing.T) { e := &Env{ Cfg: &config.Config{ + StoreBackend: tc.backend, PostgresReadWriteConnectionString: tc.connString, CloudSQL: tc.cloudSQL, }, @@ -271,3 +282,14 @@ func TestApplyPostgresRequiresPostgresPool(t *testing.T) { t.Fatalf("applyPostgres() error = %v, want the missing %s pool", err, postgresPoolSelector) } } + +func TestDeployPostgresRefusesMySQL(t *testing.T) { + e := &Env{ + Cfg: &config.Config{Root: repoRoot(t), StoreBackend: config.StoreBackendMySQL}, + Kube: fakeKube(t), + } + err := e.DeployPostgres(t.Context()) + if err == nil || !strings.Contains(err.Error(), "ATE_API_STORE_BACKEND=mysql") { + t.Fatalf("DeployPostgres() error = %v, want a refusal naming the MySQL backend", err) + } +} diff --git a/cmd/ate-setup/internal/steps/prereqs.go b/cmd/ate-setup/internal/steps/prereqs.go index 9f5149e699..b1df8e65e0 100644 --- a/cmd/ate-setup/internal/steps/prereqs.go +++ b/cmd/ate-setup/internal/steps/prereqs.go @@ -47,8 +47,8 @@ func (e *Env) EnsureAPIServerPrerequisites(ctx context.Context) error { if err := e.ensureSecret(ctx, NamespacePodCert, SecretServiceDNSCA, e.CreatePodCertificateControllerCAs); err != nil { return err } - // Always reconcile the PostgreSQL connection settings, so that a changed - // ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING reaches an existing install. + // Always reconcile the store settings, so that a changed connection string + // reaches an existing install. if err := e.CreateAPIServerEnvVars(ctx); err != nil { return err } diff --git a/cmd/ateapi/internal/authz/authz_test.go b/cmd/ateapi/internal/authz/authz_test.go index 8a8b33c3c9..5a8a8f5268 100644 --- a/cmd/ateapi/internal/authz/authz_test.go +++ b/cmd/ateapi/internal/authz/authz_test.go @@ -16,14 +16,17 @@ package authz import ( "context" + "database/sql" "errors" + "fmt" "os" - "os/exec" - "runtime" "strings" + "sync" + "sync/atomic" "testing" "time" + gomysql "github.com/go-sql-driver/mysql" "github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/stdlib" openfgav1 "github.com/openfga/api/proto/openfga/v1" @@ -31,102 +34,272 @@ import ( serverErrors "github.com/openfga/openfga/pkg/server/errors" "github.com/pressly/goose/v3" "github.com/testcontainers/testcontainers-go" - "github.com/testcontainers/testcontainers-go/modules/postgres" + tcmysql "github.com/testcontainers/testcontainers-go/modules/mysql" + tcpostgres "github.com/testcontainers/testcontainers-go/modules/postgres" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/dockerenv" "github.com/agent-substrate/substrate/internal/principal" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) -func configureDockerEnv(ctx context.Context) error { - if os.Getenv("DOCKER_HOST") != "" { - return nil +// One container per backend serves every test in this package. Each test gets +// a fresh database in it, so OpenFGA stores and tuples never leak between tests. +var ( + databaseSeq atomic.Int64 + + postgresOnce sync.Once + postgresContainer *tcpostgres.PostgresContainer + postgresAdmin *pgxpool.Pool + postgresErr error + + mysqlOnce sync.Once + mysqlContainer *tcmysql.MySQLContainer + mysqlAdmin *sql.DB + mysqlAdminDSN string + mysqlErr error +) + +func TestMain(m *testing.M) { + code := m.Run() + if postgresAdmin != nil { + postgresAdmin.Close() } - output, err := exec.CommandContext(ctx, "docker", "context", "inspect", "--format", "{{.Endpoints.docker.Host}}").Output() - if err != nil { - return err + if mysqlAdmin != nil { + _ = mysqlAdmin.Close() + } + var containers []testcontainers.Container + if postgresContainer != nil { + containers = append(containers, postgresContainer) } - host := strings.TrimSpace(string(output)) - if host == "" { - return nil + if mysqlContainer != nil { + containers = append(containers, mysqlContainer) } - _ = os.Setenv("DOCKER_HOST", host) - if os.Getenv("TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE") == "" { - socket := host - if runtime.GOOS == "darwin" { - socket = "/var/run/docker.sock" + for _, ctr := range containers { + if err := testcontainers.TerminateContainer(ctr); err != nil { + fmt.Fprintf(os.Stderr, "terminating testcontainer: %v\n", err) + if code == 0 { + code = 1 + } } - _ = os.Setenv("TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE", socket) } - return nil + os.Exit(code) } -func startPostgres(t *testing.T) *pgxpool.Pool { +func requireContainer(t *testing.T, name string, err error) { t.Helper() - ctx := context.Background() - if err := configureDockerEnv(ctx); err != nil { - t.Skipf("skipping test; docker is unavailable: %v", err) + if err == nil { + return } - - pgContainer, err := postgres.Run(ctx, - "postgres:18-alpine", - postgres.WithDatabase("testdb"), - postgres.WithUsername("postgres"), - postgres.WithPassword("postgres"), - postgres.BasicWaitStrategies(), - ) - if err != nil { - t.Skipf("skipping test; failed to start postgres container: %v", err) + if dockerenv.Required() { + t.Fatalf("%s testcontainer unavailable and required (CI or REQUIRE_DOCKER is set): %v", name, err) } - t.Cleanup(func() { - _ = testcontainers.TerminateContainer(pgContainer) + t.Skipf("%s testcontainer unavailable (requires Docker): %v", name, err) +} + +// testDB is one migrated store database and the backend-specific operations +// the tests need on it. +type testDB struct { + name string + backend Backend + // sqlDB runs queries on the backend's pool. + sqlDB *sql.DB + // singleConnBackend returns a second backend on the same database whose + // pool holds at most one connection. + singleConnBackend func(t *testing.T) Backend +} + +type testTx struct { + tx Tx + exec func(ctx context.Context, query string) error + commit func(ctx context.Context) error + rollback func(ctx context.Context) error +} + +// forEachBackend runs fn against a fresh PostgreSQL and a fresh MySQL store +// database. +func forEachBackend(t *testing.T, fn func(t *testing.T, db testDB)) { + t.Run("postgres", func(t *testing.T) { fn(t, startPostgres(t)) }) + t.Run("mysql", func(t *testing.T) { fn(t, startMySQL(t)) }) +} + +func startPostgres(t *testing.T) testDB { + t.Helper() + ctx := context.Background() + postgresOnce.Do(func() { + if err := dockerenv.Configure(ctx); err != nil { + postgresErr = err + return + } + ctr, err := tcpostgres.Run(ctx, + "postgres:18-alpine", + tcpostgres.WithDatabase("testdb"), + tcpostgres.WithUsername("postgres"), + tcpostgres.WithPassword("postgres"), + tcpostgres.BasicWaitStrategies(), + ) + if ctr != nil { + postgresContainer = ctr + } + if err != nil { + postgresErr = err + return + } + connStr, err := ctr.ConnectionString(ctx, "sslmode=disable") + if err != nil { + postgresErr = err + return + } + pool, err := pgxpool.New(ctx, connStr) + if err != nil { + postgresErr = err + return + } + postgresAdmin = pool + postgresErr = pingWithRetries(ctx, pool.Ping) }) + requireContainer(t, "PostgreSQL", postgresErr) - connStr, err := pgContainer.ConnectionString(ctx, "sslmode=disable") - if err != nil { - t.Fatalf("getting connection string: %v", err) + dbName := fmt.Sprintf("authz_%d", databaseSeq.Add(1)) + if _, err := postgresAdmin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil { + t.Fatalf("creating database %s: %v", dbName, err) } - - pool, err := pgxpool.New(ctx, connStr) + cfg := postgresAdmin.Config().Copy() + cfg.ConnConfig.Database = dbName + pool, err := pgxpool.NewWithConfig(ctx, cfg) if err != nil { t.Fatalf("creating pgxpool: %v", err) } - t.Cleanup(func() { - pool.Close() + t.Cleanup(pool.Close) + + sqlDB := stdlib.OpenDBFromPool(pool) + t.Cleanup(func() { _ = sqlDB.Close() }) + applyMigrations(t, ctx, goose.DialectPostgres, sqlDB, "../store/atepg/migrations") + + return testDB{ + name: "postgres", + backend: PostgresBackend(pool), + sqlDB: sqlDB, + singleConnBackend: func(t *testing.T) Backend { + t.Helper() + singleCfg := pool.Config() + singleCfg.MaxConns = 1 + singleCfg.MinConns = 0 + single, err := pgxpool.NewWithConfig(context.Background(), singleCfg) + if err != nil { + t.Fatalf("creating single-conn pool: %v", err) + } + t.Cleanup(single.Close) + return PostgresBackend(single) + }, + } +} + +func startMySQL(t *testing.T) testDB { + t.Helper() + ctx := context.Background() + mysqlOnce.Do(func() { + if err := dockerenv.Configure(ctx); err != nil { + mysqlErr = err + return + } + ctr, err := tcmysql.Run(ctx, + "mysql:8.4", + tcmysql.WithDatabase("authz"), + tcmysql.WithUsername("root"), + tcmysql.WithPassword("root"), + ) + if ctr != nil { + mysqlContainer = ctr + } + if err != nil { + mysqlErr = err + return + } + // The session settings atemy.Open applies; importing atemy here + // would be an import cycle. + dsn, err := ctr.ConnectionString(ctx, "parseTime=true", "loc=UTC", "clientFoundRows=true", "interpolateParams=true", "transaction_isolation=%27READ-COMMITTED%27") + if err != nil { + mysqlErr = err + return + } + db, err := sql.Open("mysql", dsn) + if err != nil { + mysqlErr = err + return + } + mysqlAdmin = db + mysqlAdminDSN = dsn + mysqlErr = pingWithRetries(ctx, db.PingContext) }) + requireContainer(t, "MySQL", mysqlErr) + + dbName := fmt.Sprintf("authz_%d", databaseSeq.Add(1)) + if _, err := mysqlAdmin.ExecContext(ctx, "CREATE DATABASE "+dbName); err != nil { + t.Fatalf("creating database %s: %v", dbName, err) + } + cfg, err := gomysql.ParseDSN(mysqlAdminDSN) + if err != nil { + t.Fatalf("parsing MySQL DSN: %v", err) + } + cfg.DBName = dbName + dsn := cfg.FormatDSN() + openDB := func(t *testing.T) *sql.DB { + t.Helper() + db, err := sql.Open("mysql", dsn) + if err != nil { + t.Fatalf("opening MySQL database: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + return db + } + db := openDB(t) + applyMigrations(t, ctx, goose.DialectMySQL, db, "../store/atemy/migrations") + + return testDB{ + name: "mysql", + backend: MySQLBackend(db), + sqlDB: db, + singleConnBackend: func(t *testing.T) Backend { + t.Helper() + single := openDB(t) + single.SetMaxOpenConns(1) + return MySQLBackend(single) + }, + } +} - var pingErr error +func pingWithRetries(ctx context.Context, ping func(context.Context) error) error { + var err error for i := 0; i < 30; i++ { - pingErr = pool.Ping(ctx) - if pingErr == nil { - break + if err = ping(ctx); err == nil { + return nil } time.Sleep(500 * time.Millisecond) } - if pingErr != nil { - t.Fatalf("timed out waiting for postgres ping: %v", pingErr) - } + return fmt.Errorf("pinging database after retries: %w", err) +} - db := stdlib.OpenDBFromPool(pool) - defer db.Close() - provider, err := goose.NewProvider(goose.DialectPostgres, db, os.DirFS("../store/atepg/migrations")) +func applyMigrations(t *testing.T, ctx context.Context, dialect goose.Dialect, db *sql.DB, dir string) { + t.Helper() + provider, err := goose.NewProvider(dialect, db, os.DirFS(dir)) if err != nil { t.Fatalf("create goose provider: %v", err) } if _, err := provider.Up(ctx); err != nil { t.Fatalf("run Substrate and OpenFGA migrations: %v", err) } - return pool } -func newTestAuthz(t *testing.T, ctx context.Context, pool *pgxpool.Pool) (*server.Server, string, string) { +func newTestAuthz(t *testing.T, ctx context.Context, backend Backend) (*server.Server, string, string) { t.Helper() - fgaSrv, err := NewOpenFGAServer(pool) + fgaSrv, err := NewOpenFGAServer(backend) if err != nil { t.Fatalf("NewOpenFGAServer failed: %v", err) } t.Cleanup(fgaSrv.Close) - storeID, modelID, err := EnsureStoreAndModel(ctx, pool, fgaSrv) + storeID, modelID, err := EnsureStoreAndModel(ctx, backend, fgaSrv) if err != nil { t.Fatalf("EnsureStoreAndModel failed: %v", err) } @@ -134,19 +307,22 @@ func newTestAuthz(t *testing.T, ctx context.Context, pool *pgxpool.Pool) (*serve } func TestEnsureStoreAndModel_NilArgs(t *testing.T) { - if _, err := NewOpenFGAServer(nil); err == nil { - t.Fatal("expected error when pool is nil in NewOpenFGAServer") + if _, err := NewOpenFGAServer(Backend{}); err == nil { + t.Fatal("expected error when the backend is empty in NewOpenFGAServer") } - if _, _, err := EnsureStoreAndModel(context.Background(), nil, nil); err == nil { - t.Fatal("expected error when pool is nil in EnsureStoreAndModel") + if _, _, err := EnsureStoreAndModel(context.Background(), Backend{}, nil); err == nil { + t.Fatal("expected error when the backend is empty in EnsureStoreAndModel") } } func TestEnsureStoreAndModel_InitializeAndCheck(t *testing.T) { - pool := startPostgres(t) + forEachBackend(t, testEnsureStoreAndModelInitializeAndCheck) +} + +func testEnsureStoreAndModelInitializeAndCheck(t *testing.T, db testDB) { ctx := context.Background() - fgaSrv, storeID, modelID := newTestAuthz(t, ctx, pool) + fgaSrv, storeID, modelID := newTestAuthz(t, ctx, db.backend) if storeID == "" { t.Fatal("expected non-empty storeID") @@ -156,11 +332,8 @@ func TestEnsureStoreAndModel_InitializeAndCheck(t *testing.T) { } // Write relationship tuples inside a transaction and verify authorization checks against the model. - tx, err := pool.Begin(ctx) - if err != nil { - t.Fatalf("pool.Begin failed: %v", err) - } - _, err = fgaSrv.Write(ContextWithTx(ctx, tx), &openfgav1.WriteRequest{ + tx := db.begin(t, ctx) + _, err := fgaSrv.Write(ContextWithTx(ctx, tx.tx), &openfgav1.WriteRequest{ StoreId: storeID, AuthorizationModelId: modelID, Writes: &openfgav1.WriteRequestWrites{ @@ -181,7 +354,7 @@ func TestEnsureStoreAndModel_InitializeAndCheck(t *testing.T) { if err != nil { t.Fatalf("Write tuples failed: %v", err) } - if err := tx.Commit(ctx); err != nil { + if err := tx.commit(ctx); err != nil { t.Fatalf("tx.Commit failed: %v", err) } @@ -218,7 +391,7 @@ func TestEnsureStoreAndModel_InitializeAndCheck(t *testing.T) { } // Verify idempotent re-initialization on the same shared pool reuses the existing store and model. - _, storeID2, modelID2 := newTestAuthz(t, ctx, pool) + _, storeID2, modelID2 := newTestAuthz(t, ctx, db.backend) if storeID2 != storeID { t.Errorf("expected same storeID %q on re-init, got %q", storeID, storeID2) @@ -228,13 +401,39 @@ func TestEnsureStoreAndModel_InitializeAndCheck(t *testing.T) { } } +func TestAcquireInitLock_CanceledWhileHeld(t *testing.T) { + forEachBackend(t, func(t *testing.T, db testDB) { + unlock, err := acquireInitLock(t.Context(), db.backend) + if err != nil { + t.Fatalf("acquireInitLock failed: %v", err) + } + ctx, cancel := context.WithTimeout(t.Context(), 500*time.Millisecond) + defer cancel() + _, err = acquireInitLock(ctx, db.backend) + unlock() + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("acquireInitLock while held = %v, want context.DeadlineExceeded", err) + } + ctx, cancel = context.WithTimeout(t.Context(), 5*time.Second) + defer cancel() + unlock, err = acquireInitLock(ctx, db.backend) + if err != nil { + t.Fatalf("acquireInitLock after unlock failed: %v", err) + } + unlock() + }) +} + func TestTransactionalDatastore_RollbackAndCommit(t *testing.T) { + forEachBackend(t, testTransactionalDatastoreRollbackAndCommit) +} + +func testTransactionalDatastoreRollbackAndCommit(t *testing.T, db testDB) { ctx := context.Background() - pool := startPostgres(t) - fgaSrv, storeID, modelID := newTestAuthz(t, ctx, pool) + fgaSrv, storeID, modelID := newTestAuthz(t, ctx, db.backend) - // Calling Write or Read (ReadPage) without an active pgx.Tx in ctx must fail loudly. + // Calling Write or Read (ReadPage) without an active transaction in ctx must fail loudly. if _, err := fgaSrv.Write(ctx, &openfgav1.WriteRequest{ StoreId: storeID, AuthorizationModelId: modelID, @@ -293,23 +492,22 @@ func TestTransactionalDatastore_RollbackAndCommit(t *testing.T) { atespaceExists := func() bool { t.Helper() var exists bool - if err := pool.QueryRow(ctx, "SELECT EXISTS(SELECT 1 FROM atespaces WHERE name = 'team-tx')").Scan(&exists); err != nil { + if err := db.sqlDB.QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM atespaces WHERE name = 'team-tx')").Scan(&exists); err != nil { t.Fatalf("checking atespaces row failed: %v", err) } return exists } + const insertAtespace = "INSERT INTO atespaces (name, uid, version, proto) VALUES ('team-tx', 'uid-1', 1, '')" + // Write both a Substrate atespaces row and an OpenFGA tuple inside a transaction // that rolls back -> neither the atespaces row nor the tuple may persist. - txRollback, err := pool.Begin(ctx) - if err != nil { - t.Fatalf("pool.Begin failed: %v", err) - } - if _, err := txRollback.Exec(ctx, "INSERT INTO atespaces (name, uid, version, proto) VALUES ('team-tx', 'uid-1', 1, $1)", []byte{}); err != nil { + txRollback := db.begin(t, ctx) + if err := txRollback.exec(ctx, insertAtespace); err != nil { t.Fatalf("txRollback insert atespaces failed: %v", err) } - writeTuple(ContextWithTx(ctx, txRollback)) - if err := txRollback.Rollback(ctx); err != nil { + writeTuple(ContextWithTx(ctx, txRollback.tx)) + if err := txRollback.rollback(ctx); err != nil { t.Fatalf("Rollback failed: %v", err) } if atespaceExists() { @@ -321,15 +519,12 @@ func TestTransactionalDatastore_RollbackAndCommit(t *testing.T) { // Write both the Substrate atespaces row and the OpenFGA tuple in a committed // transaction -> both persist atomically. - txCommit, err := pool.Begin(ctx) - if err != nil { - t.Fatalf("pool.Begin failed: %v", err) - } - if _, err := txCommit.Exec(ctx, "INSERT INTO atespaces (name, uid, version, proto) VALUES ('team-tx', 'uid-1', 1, $1)", []byte{}); err != nil { + txCommit := db.begin(t, ctx) + if err := txCommit.exec(ctx, insertAtespace); err != nil { t.Fatalf("txCommit insert atespaces failed: %v", err) } - writeTuple(ContextWithTx(ctx, txCommit)) - if err := txCommit.Commit(ctx); err != nil { + writeTuple(ContextWithTx(ctx, txCommit.tx)) + if err := txCommit.commit(ctx); err != nil { t.Fatalf("Commit failed: %v", err) } if !atespaceExists() { @@ -340,39 +535,33 @@ func TestTransactionalDatastore_RollbackAndCommit(t *testing.T) { } // Closing fgaServer must not close the shared pool. - fgaSrv2, err := NewOpenFGAServer(pool) + fgaSrv2, err := NewOpenFGAServer(db.backend) if err != nil { t.Fatalf("NewOpenFGAServer failed: %v", err) } fgaSrv2.Close() - if err := pool.Ping(ctx); err != nil { + if err := db.sqlDB.PingContext(ctx); err != nil { t.Fatalf("expected shared pool to remain open after fgaServer.Close(), got %v", err) } // Verify fgaServer.Read and fgaServer.Write inside ContextWithTx do not check out - // a second connection from a pool with MaxConns=1 (preventing pool starvation deadlock). - singleConnCfg := pool.Config() - singleConnCfg.MaxConns = 1 - singleConnCfg.MinConns = 0 - singlePool, err := pgxpool.NewWithConfig(ctx, singleConnCfg) - if err != nil { - t.Fatalf("creating single-conn pool: %v", err) - } - t.Cleanup(singlePool.Close) - - singleFGASrv, err := NewOpenFGAServer(singlePool) + // a second connection from a single-connection pool (preventing pool starvation deadlock). + singleBackend := db.singleConnBackend(t) + singleFGASrv, err := NewOpenFGAServer(singleBackend) if err != nil { - t.Fatalf("NewOpenFGAServer(singlePool) failed: %v", err) + t.Fatalf("NewOpenFGAServer(singleBackend) failed: %v", err) } t.Cleanup(singleFGASrv.Close) + if singleBackend.db != nil { + if got := singleBackend.db.Stats().MaxOpenConnections; got != 1 { + t.Fatalf("MaxOpenConnections after NewOpenFGAServer = %d, want 1", got) + } + } txCtxTimeout, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() - singleTx, err := singlePool.Begin(txCtxTimeout) - if err != nil { - t.Fatalf("singlePool.Begin failed: %v", err) - } - txCtx := ContextWithTx(txCtxTimeout, singleTx) + singleTx := beginOn(t, txCtxTimeout, singleBackend) + txCtx := ContextWithTx(txCtxTimeout, singleTx.tx) readResp, err := singleFGASrv.Read(txCtx, &openfgav1.ReadRequest{ StoreId: storeID, @@ -394,7 +583,7 @@ func TestTransactionalDatastore_RollbackAndCommit(t *testing.T) { if err != nil { t.Fatalf("singleFGASrv.Write delete on single-connection pool failed: %v", err) } - if err := singleTx.Commit(txCtxTimeout); err != nil { + if err := singleTx.commit(txCtxTimeout); err != nil { t.Fatalf("singleTx.Commit failed: %v", err) } if checkAllowed() { @@ -402,14 +591,51 @@ func TestTransactionalDatastore_RollbackAndCommit(t *testing.T) { } } -func writeTestTuple(t *testing.T, ctx context.Context, pool *pgxpool.Pool, pm *PolicyManager, user, relation, object string) { +func (db testDB) begin(t *testing.T, ctx context.Context) testTx { + t.Helper() + return beginOn(t, ctx, db.backend) +} + +// beginOn opens a transaction on backend's own pool, which may differ from +// the testDB pool (such as a single-connection pool). +func beginOn(t *testing.T, ctx context.Context, backend Backend) testTx { t.Helper() - tx, err := pool.Begin(ctx) + if backend.pool != nil { + tx, err := backend.pool.Begin(ctx) + if err != nil { + t.Fatalf("pool.Begin failed: %v", err) + } + t.Cleanup(func() { _ = tx.Rollback(context.Background()) }) + return testTx{ + tx: PgxTx(tx), + exec: func(ctx context.Context, query string) error { + _, err := tx.Exec(ctx, query) + return err + }, + commit: tx.Commit, + rollback: tx.Rollback, + } + } + tx, err := backend.db.BeginTx(ctx, nil) if err != nil { - t.Fatalf("pool.Begin failed: %v", err) + t.Fatalf("db.BeginTx failed: %v", err) + } + t.Cleanup(func() { _ = tx.Rollback() }) + return testTx{ + tx: SQLTx(tx), + exec: func(ctx context.Context, query string) error { + _, err := tx.ExecContext(ctx, query) + return err + }, + commit: func(context.Context) error { return tx.Commit() }, + rollback: func(context.Context) error { return tx.Rollback() }, } - defer func() { _ = tx.Rollback(ctx) }() - if _, err := pm.fgaServer.Write(ContextWithTx(ctx, tx), &openfgav1.WriteRequest{ +} + +func writeTestTuple(t *testing.T, ctx context.Context, db testDB, pm *PolicyManager, user, relation, object string) { + t.Helper() + tx := db.begin(t, ctx) + if _, err := pm.fgaServer.Write(ContextWithTx(ctx, tx.tx), &openfgav1.WriteRequest{ StoreId: pm.storeID, AuthorizationModelId: pm.modelID, Writes: &openfgav1.WriteRequestWrites{ @@ -425,29 +651,32 @@ func writeTestTuple(t *testing.T, ctx context.Context, pool *pgxpool.Pool, pm *P }); err != nil { t.Fatalf("writeTestTuple(%s, %s, %s) failed: %v", user, relation, object, err) } - if err := tx.Commit(ctx); err != nil { + if err := tx.commit(ctx); err != nil { t.Fatalf("tx.Commit failed: %v", err) } } func TestAuthorizerAndPolicyManager_RuntimeChecks(t *testing.T) { - pool := startPostgres(t) + forEachBackend(t, testAuthorizerAndPolicyManagerRuntimeChecks) +} + +func testAuthorizerAndPolicyManagerRuntimeChecks(t *testing.T, db testDB) { ctx := context.Background() - fgaSrv, err := NewOpenFGAServer(pool) + fgaSrv, err := NewOpenFGAServer(db.backend) if err != nil { t.Fatalf("NewOpenFGAServer failed: %v", err) } t.Cleanup(fgaSrv.Close) - authorizer, policyManager, err := New(ctx, pool, fgaSrv, nil) + authorizer, policyManager, err := New(ctx, db.backend, fgaSrv, nil) if err != nil { t.Fatalf("authz.New failed: %v", err) } // 1. Seed global owners (including a Kubernetes ServiceAccount ID with colons). - writeTestTuple(t, ctx, pool, policyManager, "alice", "owner", GlobalRootObject) - writeTestTuple(t, ctx, pool, policyManager, "system:serviceaccount:default:default", "owner", GlobalRootObject) + writeTestTuple(t, ctx, db, policyManager, "alice", "owner", GlobalRootObject) + writeTestTuple(t, ctx, db, policyManager, "system:serviceaccount:default:default", "owner", GlobalRootObject) // 1b. A nil Authorizer fails closed with codes.Internal unless explicitly bypassed. var nilAuthorizer *Authorizer @@ -499,24 +728,21 @@ func TestAuthorizerAndPolicyManager_RuntimeChecks(t *testing.T) { } // 5. Grant bob direct editor access on atespace:team-x. - writeTestTuple(t, ctx, pool, policyManager, "bob", "editor", AtespaceObject("team-x")) + writeTestTuple(t, ctx, db, policyManager, "bob", "editor", AtespaceObject("team-x")) if err := authorizer.Check(bobCtx, RelationCanGet, AtespaceObject("team-x")); err != nil { t.Errorf("expected bob allowed can_get on team-x, got %v", err) } // 6. PolicyManager.DeleteAtespacePolicies requires a transaction, and // removes all tuples on team-x when committed. - if err := policyManager.DeleteAtespacePolicies(ctx, nil, "team-x"); !errors.Is(err, ErrNilTransaction) { + if err := policyManager.DeleteAtespacePolicies(ctx, Tx{}, "team-x"); !errors.Is(err, ErrNilTransaction) { t.Fatalf("DeleteAtespacePolicies with nil tx = %v, want ErrNilTransaction", err) } - txDel, err := pool.Begin(ctx) - if err != nil { - t.Fatalf("pool.Begin failed: %v", err) - } - if err := policyManager.DeleteAtespacePolicies(ctx, txDel, "team-x"); err != nil { + txDel := db.begin(t, ctx) + if err := policyManager.DeleteAtespacePolicies(ctx, txDel.tx, "team-x"); err != nil { t.Fatalf("DeleteAtespacePolicies failed: %v", err) } - if err := txDel.Commit(ctx); err != nil { + if err := txDel.commit(ctx); err != nil { t.Fatalf("txDel.Commit failed: %v", err) } if err := authorizer.Check(bobCtx, RelationCanGet, AtespaceObject("team-x")); status.Code(err) != codes.PermissionDenied { @@ -571,3 +797,44 @@ func TestFormatUser_NoCollision(t *testing.T) { t.Fatalf("formatUser(\"alice\") and formatUser(\"user:alice\") collided on %q", u1) } } + +// TestPolicyManager_MemberLengthPerBackend pins the one difference between +// the backends: OpenFGA's MySQL schema holds a tuple user in 256 characters, +// so only the MySQL store refuses longer members, as InvalidArgument. +func TestPolicyManager_MemberLengthPerBackend(t *testing.T) { + forEachBackend(t, func(t *testing.T, db testDB) { + ctx := t.Context() + fgaSrv, err := NewOpenFGAServer(db.backend) + if err != nil { + t.Fatalf("NewOpenFGAServer failed: %v", err) + } + t.Cleanup(fgaSrv.Close) + _, policyManager, err := New(ctx, db.backend, fgaSrv, nil) + if err != nil { + t.Fatalf("New failed: %v", err) + } + reconcile := func(member string) error { + t.Helper() + tx := db.begin(t, ctx) + defer tx.rollback(ctx) //nolint:errcheck // the test only inspects the reconcile error + return policyManager.ReconcileGlobalBindings(ctx, tx.tx, []*ateapipb.Binding{{Role: "owner", Members: []string{member}}}) + } + + if err := reconcile("user:" + strings.Repeat("a", 251)); err != nil { + t.Errorf("a 256-character member failed: %v", err) + } + err = reconcile("user:" + strings.Repeat("a", 252)) + if db.name == "postgres" { + if err != nil { + t.Errorf("a 257-character member failed on PostgreSQL: %v", err) + } + return + } + if status.Code(err) != codes.InvalidArgument { + t.Errorf("a 257-character member on MySQL = %v, want InvalidArgument", err) + } + if err := reconcile("user:" + strings.Repeat(":", 84)); status.Code(err) != codes.InvalidArgument { + t.Errorf("a member past the limit once encoded on MySQL = %v, want InvalidArgument", err) + } + }) +} diff --git a/cmd/ateapi/internal/authz/authz_util.go b/cmd/ateapi/internal/authz/authz_util.go index 064269bb17..f1d26906e4 100644 --- a/cmd/ateapi/internal/authz/authz_util.go +++ b/cmd/ateapi/internal/authz/authz_util.go @@ -16,6 +16,7 @@ package authz import ( "context" + "database/sql" _ "embed" "fmt" "log/slog" @@ -26,6 +27,7 @@ import ( openfgav1 "github.com/openfga/api/proto/openfga/v1" "github.com/openfga/language/pkg/go/transformer" "github.com/openfga/openfga/pkg/server" + "github.com/openfga/openfga/pkg/storage" "github.com/openfga/openfga/pkg/storage/postgres" "github.com/openfga/openfga/pkg/storage/sqlcommon" "google.golang.org/protobuf/proto" @@ -110,19 +112,43 @@ func FormatMember(member string) (string, error) { //go:embed model.fga var modelDSL string -// NewOpenFGAServer creates an embedded OpenFGA server backed by a transaction-aware -// PostgreSQL datastore on pool. Calling Close on the returned server stops -// OpenFGA's background workers without closing the caller-owned pool. -func NewOpenFGAServer(pool *pgxpool.Pool) (*server.Server, error) { - if pool == nil { - return nil, fmt.Errorf("postgres pool must not be nil") - } - rawDatastore, err := postgres.NewWithDB(pool, nil, sqlcommon.NewConfig()) - if err != nil { - return nil, fmt.Errorf("creating OpenFGA postgres adapter: %w", err) +// Backend is the store database that holds the OpenFGA tables. Build it with +// PostgresBackend or MySQLBackend; the zero value names no database. +type Backend struct { + pool *pgxpool.Pool + db *sql.DB +} + +// PostgresBackend names the PostgreSQL store's connection pool. +func PostgresBackend(pool *pgxpool.Pool) Backend { return Backend{pool: pool} } + +// MySQLBackend names the MySQL store's connection pool. +func MySQLBackend(db *sql.DB) Backend { return Backend{db: db} } + +// NewOpenFGAServer creates an embedded OpenFGA server backed by a +// transaction-aware datastore on the backend's pool. Calling Close on the +// returned server stops OpenFGA's background workers without closing the +// caller-owned pool. +func NewOpenFGAServer(backend Backend) (*server.Server, error) { + var rawDatastore storage.OpenFGADatastore + switch { + case backend.pool != nil: + ds, err := postgres.NewWithDB(backend.pool, nil, sqlcommon.NewConfig()) + if err != nil { + return nil, fmt.Errorf("creating OpenFGA postgres adapter: %w", err) + } + rawDatastore = ds + case backend.db != nil: + ds, err := newMySQLDatastore(backend.db) + if err != nil { + return nil, err + } + rawDatastore = ds + default: + return nil, fmt.Errorf("store backend must not be empty") } fgaServer, err := server.NewServerWithOpts( - server.WithDatastore(newTransactionalDatastore(rawDatastore)), + server.WithDatastore(&transactionalDatastore{OpenFGADatastore: rawDatastore}), ) if err != nil { return nil, fmt.Errorf("creating OpenFGA server: %w", err) @@ -131,17 +157,17 @@ func NewOpenFGAServer(pool *pgxpool.Pool) (*server.Server, error) { } // EnsureStoreAndModel ensures the default OpenFGA store and checked-in authorization model -// are provisioned on fgaServer (serialized across replicas via a PostgreSQL -// advisory lock on pool) and returns the provisioned (storeID, modelID). -func EnsureStoreAndModel(ctx context.Context, pool *pgxpool.Pool, fgaServer *server.Server) (string, string, error) { - if pool == nil { - return "", "", fmt.Errorf("postgres pool must not be nil") +// are provisioned on fgaServer (serialized across replicas via a database lock +// on the backend) and returns the provisioned (storeID, modelID). +func EnsureStoreAndModel(ctx context.Context, backend Backend, fgaServer *server.Server) (string, string, error) { + if backend.pool == nil && backend.db == nil { + return "", "", fmt.Errorf("store backend must not be empty") } if fgaServer == nil { return "", "", fmt.Errorf("fgaServer must not be nil") } - unlock, err := acquireInitLock(ctx, pool) + unlock, err := acquireInitLock(ctx, backend) if err != nil { return "", "", err } @@ -165,12 +191,12 @@ func EnsureStoreAndModel(ctx context.Context, pool *pgxpool.Pool, fgaServer *ser // PolicyManager. bootstrapOwners are principal IDs (with or without the // "user:" prefix) that the Authorizer treats as owners of global:root on every // check, independent of any stored AccessPolicy. -func New(ctx context.Context, pool *pgxpool.Pool, fgaServer *server.Server, bootstrapOwners []string) (*Authorizer, *PolicyManager, error) { +func New(ctx context.Context, backend Backend, fgaServer *server.Server, bootstrapOwners []string) (*Authorizer, *PolicyManager, error) { owners, err := parseBootstrapOwners(bootstrapOwners) if err != nil { return nil, nil, err } - storeID, modelID, err := EnsureStoreAndModel(ctx, pool, fgaServer) + storeID, modelID, err := EnsureStoreAndModel(ctx, backend, fgaServer) if err != nil { return nil, nil, err } @@ -185,6 +211,9 @@ func New(ctx context.Context, pool *pgxpool.Pool, fgaServer *server.Server, boot storeID: storeID, modelID: modelID, } + if backend.db != nil { + policyManager.maxUserLength = mysqlMaxTupleUserLength + } return authorizer, policyManager, nil } @@ -206,8 +235,11 @@ func parseBootstrapOwners(ids []string) (map[string]struct{}, error) { // OpenFGA store provisioning across replicas. const ateFGAInitLockID = int64(0x6174656667610000) // "atefga\0\0" -func acquireInitLock(ctx context.Context, pool *pgxpool.Pool) (func(), error) { - conn, err := pool.Acquire(ctx) +func acquireInitLock(ctx context.Context, backend Backend) (func(), error) { + if backend.db != nil { + return acquireMySQLInitLock(ctx, backend.db) + } + conn, err := backend.pool.Acquire(ctx) if err != nil { return nil, fmt.Errorf("acquiring connection for OpenFGA init lock: %w", err) } diff --git a/cmd/ateapi/internal/authz/datastore.go b/cmd/ateapi/internal/authz/datastore.go index c9ef0c5aef..8fae05fe76 100644 --- a/cmd/ateapi/internal/authz/datastore.go +++ b/cmd/ateapi/internal/authz/datastore.go @@ -29,69 +29,70 @@ import ( ) // ErrNoTransactionInContext is returned by ReadPage and Write when called -// without an active pgx.Tx injected via ContextWithTx. -var ErrNoTransactionInContext = errors.New("authz datastore: active pgx.Tx required in context") +// without an active store transaction injected via ContextWithTx. +var ErrNoTransactionInContext = errors.New("authz datastore: active store transaction required in context") // NOTE: The SQL query and tuple write/delete/changelog execution helpers in this // file are 1:1 adaptations of unexported methods and package-private types in // github.com/openfga/openfga/pkg/storage/postgres/postgres.go (pinned to // PostgreSQL migration version 6 via TestOpenFGAMigrationVersionGuard in -// cmd/ateapi/internal/store/atepg/migrations_test.go). +// cmd/ateapi/internal/store/atepg/migrations_test.go) and of sqlcommon.Write in +// github.com/openfga/openfga/pkg/storage/sqlcommon/sqlcommon.go, which the +// upstream MySQL datastore uses. Both upstream paths run the same statements; +// they differ only in placeholder format, driver, and error mapping, which +// txStatements carries. // // Why this wrapper is necessary: -// 1. Upstream (*postgres.Datastore).Write (postgres.go:584-649) always opens -// its own transaction via s.primaryDB.BeginTx(ctx, ...) and commits it -// before returning, making it impossible to atomically commit or roll back -// OpenFGA tuple mutations alongside Substrate table mutations. -// 2. Upstream (*postgres.Datastore).ReadPage / ReadAuthorizationModel -// always query s.getPgxPool(...) instead of an active pgx.Tx, which both -// escapes the caller's transaction snapshot (e.g. RepeatableRead / Serializable) -// and can deadlock a pool when all connections are held by active write transactions. -// 3. Upstream (*postgres.Datastore).Close (postgres.go:307-314) closes the -// underlying *pgxpool.Pool, which would close Substrate's shared pool when +// 1. Upstream (*postgres.Datastore).Write (postgres.go:584-649) and +// sqlcommon.Write (sqlcommon.go:987-1139) always open their own +// transaction and commit it before returning, making it impossible to +// atomically commit or roll back OpenFGA tuple mutations alongside +// Substrate table mutations. +// 2. Upstream ReadPage / ReadAuthorizationModel always query the pool +// instead of an active transaction, which both escapes the caller's +// transaction snapshot (e.g. RepeatableRead / Serializable) and can +// deadlock a pool when all connections are held by active write transactions. +// 3. Upstream Close (postgres.go:307-314, mysql.go:129-134) closes the +// underlying pool, which would close Substrate's shared pool when // fgaServer.Close() is called. // // Transaction enforcement rules: // - Write and ReadPage (called by fgaServer.Write and fgaServer.Read) require -// an active pgx.Tx in ctx via ContextWithTx and fail fast with +// an active store transaction in ctx via ContextWithTx and fail fast with // ErrNoTransactionInContext if absent. -// - ReadAuthorizationModel uses the active pgx.Tx when present in ctx (e.g. -// when fgaServer.Write validates tuples against the model), and falls back -// to the connection pool when called outside a transaction (e.g. during -// fgaServer.Check or EnsureStoreAndModel). +// - ReadAuthorizationModel uses the active transaction when present in ctx +// (e.g. when fgaServer.Write validates tuples against the model), and falls +// back to the connection pool when called outside a transaction (e.g. +// during fgaServer.Check or EnsureStoreAndModel). // - All other datastore methods (e.g. Read, ReadUserTuple, ReadUsersetTuples) -// are inherited from the embedded *postgres.Datastore and run on the pool. +// are inherited from the embedded upstream datastore and run on the pool. -// transactionalDatastore wraps OpenFGA's postgres.Datastore to allow ReadPage and Write -// operations to participate in an existing external transaction passed via ContextWithTx. +// transactionalDatastore wraps an upstream OpenFGA datastore so ReadPage and +// Write join the store transaction passed via ContextWithTx. type transactionalDatastore struct { - *postgres.Datastore + storage.OpenFGADatastore } -// newTransactionalDatastore wraps ds with transaction-awareness. -func newTransactionalDatastore(ds *postgres.Datastore) *transactionalDatastore { - return &transactionalDatastore{Datastore: ds} -} - -// Close is a no-op because the underlying *pgxpool.Pool is owned and closed by -// the caller (cmd/ateapi/main.go), and OpenFGA's server.Close() calls datastore.Close(). -// Upstream equivalent: (*postgres.Datastore).Close (postgres.go:307-314). +// Close is a no-op because the caller (cmd/ateapi/main.go) owns the pool, and +// OpenFGA's server.Close() calls datastore.Close(). func (d *transactionalDatastore) Close() {} -// ReadAuthorizationModel checks for an active pgx.Tx on ctx. If present (e.g. -// during fgaServer.Write tuple validation), it queries the authorization_model -// table on that transaction so Write never needs to check out a second connection -// from the pool. When no pgx.Tx is in ctx (e.g. during fgaServer.Check or -// EnsureStoreAndModel), it delegates to the underlying pool datastore. +// ReadAuthorizationModel checks for an active store transaction on ctx. If +// present (e.g. during fgaServer.Write tuple validation), it queries the +// authorization_model table on that transaction so Write never needs to check +// out a second connection from the pool. When no transaction is in ctx (e.g. +// during fgaServer.Check or EnsureStoreAndModel), it delegates to the +// underlying pool datastore. // -// 1:1 with (*postgres.Datastore).ReadAuthorizationModel (postgres.go:831-858), -// except rows are queried via tx.Query instead of db.Query when tx is present. +// 1:1 with (*postgres.Datastore).ReadAuthorizationModel (postgres.go:831-858) +// and sqlcommon.ReadAuthorizationModel (sqlcommon.go:1267-1290), except rows +// are queried on the transaction instead of the pool when it is present. func (d *transactionalDatastore) ReadAuthorizationModel(ctx context.Context, store string, modelID string) (*openfgav1.AuthorizationModel, error) { tx, ok := TxFromContext(ctx) if !ok { - return d.Datastore.ReadAuthorizationModel(ctx, store, modelID) + return d.OpenFGADatastore.ReadAuthorizationModel(ctx, store, modelID) } - stmt, args, err := sq.StatementBuilder.PlaceholderFormat(sq.Dollar). + stmt, args, err := tx.q.stbl. Select("authorization_model_id", "schema_version", "type", "type_definition", "serialized_protobuf"). From("authorization_model"). Where(sq.Eq{ @@ -99,25 +100,31 @@ func (d *transactionalDatastore) ReadAuthorizationModel(ctx context.Context, sto "authorization_model_id": modelID, }).ToSql() if err != nil { - return nil, postgres.HandleSQLError(err) + return nil, tx.q.handleError(err) } - rows, err := tx.Query(ctx, stmt, args...) + conn, err := tx.q.connector.Connect(ctx) if err != nil { - return nil, postgres.HandleSQLError(err) + return nil, tx.q.handleError(err) + } + defer conn.Close() + rows, err := conn.Query(ctx, stmt, args...) + if err != nil { + return nil, tx.q.handleError(err) } defer rows.Close() - ret, err := sqlcommon.ConstructAuthorizationModelFromSQLRows(&pgxRowsWrapper{rows: rows}) + ret, err := sqlcommon.ConstructAuthorizationModelFromSQLRows(rows) if err != nil { - return nil, postgres.HandleSQLError(err) + return nil, tx.q.handleError(err) } return ret, nil } -// ReadPage requires an active pgx.Tx on ctx via ContextWithTx and executes the -// paginated tuple query on that transaction. If no transaction is present in ctx, -// it fails fast with ErrNoTransactionInContext. +// ReadPage requires an active store transaction on ctx via ContextWithTx and +// executes the paginated tuple query on that transaction. If no transaction is +// present in ctx, it fails fast with ErrNoTransactionInContext. // -// 1:1 with (*postgres.Datastore).ReadPage (postgres.go:349-361). +// 1:1 with (*postgres.Datastore).ReadPage (postgres.go:349-361) and +// (*mysql.Datastore).ReadPage (mysql.go:150-161). func (d *transactionalDatastore) ReadPage( ctx context.Context, store string, @@ -128,7 +135,7 @@ func (d *transactionalDatastore) ReadPage( if !ok { return nil, "", ErrNoTransactionInContext } - iter, err := readOnTx(tx, store, filter, options) + iter, err := readOnTx(tx.q, store, filter, options) if err != nil { return nil, "", err } @@ -136,16 +143,16 @@ func (d *transactionalDatastore) ReadPage( return iter.ToArray(ctx, options.Pagination) } -// readOnTx is a 1:1 copy of (*postgres.Datastore).read (postgres.go:363-418), -// replacing (*pgxPoolConnector)(db) with &pgxTxConnector{tx: tx}. +// readOnTx is a 1:1 copy of (*postgres.Datastore).read (postgres.go:363-418) +// and (*mysql.Datastore).read (mysql.go:163-218), replacing the pool +// connector with q's transaction connector. func readOnTx( - tx pgx.Tx, + q *txStatements, store string, filter storage.ReadFilter, pageOpts storage.ReadPageOptions, ) (*sqlcommon.SQLTupleIterator, error) { - stbl := sq.StatementBuilder.PlaceholderFormat(sq.Dollar) - sb := stbl. + sb := q.stbl. Select(sqlcommon.SQLIteratorColumns()...). From("tuple"). Where(sq.Eq{"store": store}). @@ -177,16 +184,16 @@ func readOnTx( if pageOpts.Pagination.PageSize != 0 { sb = sb.Limit(uint64(pageOpts.Pagination.PageSize + 1)) } - rowGetter, err := sqlcommon.NewRowGetter(&pgxTxConnector{tx: tx}, sb) + rowGetter, err := sqlcommon.NewRowGetter(q.connector, sb) if err != nil { - return nil, postgres.HandleSQLError(err) + return nil, q.handleError(err) } - return sqlcommon.NewSQLTupleIterator(rowGetter, postgres.HandleSQLError), nil + return sqlcommon.NewSQLTupleIterator(rowGetter, q.handleError), nil } -// Write requires an active pgx.Tx on ctx via ContextWithTx and executes the -// write on that transaction without calling BeginTx or Commit. If no transaction -// is present in ctx, it fails fast with ErrNoTransactionInContext. +// Write requires an active store transaction on ctx via ContextWithTx and +// executes the write on that transaction without calling BeginTx or Commit. If +// no transaction is present in ctx, it fails fast with ErrNoTransactionInContext. // // 1:1 with (*postgres.Datastore).Write (postgres.go:421-431). func (d *transactionalDatastore) Write( @@ -200,15 +207,15 @@ func (d *transactionalDatastore) Write( if !ok { return ErrNoTransactionInContext } - return d.writeOnTx(ctx, tx, store, deletes, writes, storage.NewTupleWriteOptions(opts...), time.Now().UTC()) + return d.writeOnTx(ctx, tx.q, store, deletes, writes, storage.NewTupleWriteOptions(opts...), time.Now().UTC()) } // writeOnTx is a 1:1 copy of (*postgres.Datastore).write (postgres.go:584-649) -// and selectAllExistingRowsForUpdate (postgres.go:435-455), except that it uses -// the caller-supplied tx instead of calling s.primaryDB.BeginTx / txn.Rollback / txn.Commit. +// and sqlcommon.Write (sqlcommon.go:987-1139), except that it uses the +// caller-supplied transaction instead of calling BeginTx / Rollback / Commit. func (d *transactionalDatastore) writeOnTx( ctx context.Context, - tx pgx.Tx, + q *txStatements, store string, deletes storage.Deletes, writes storage.Writes, @@ -220,8 +227,6 @@ func (d *transactionalDatastore) writeOnTx( return nil } - connector := &pgxTxConnector{tx: tx} - stbl := sq.StatementBuilder.PlaceholderFormat(sq.Dollar) existing := make(map[string]*openfgav1.Tuple, len(lockKeys)) for start := 0; start < len(lockKeys); start += storage.DefaultMaxTuplesPerWrite { @@ -229,7 +234,7 @@ func (d *transactionalDatastore) writeOnTx( if end > len(lockKeys) { end = len(lockKeys) } - if err := selectExistingRowsForWrite(ctx, stbl, connector, store, lockKeys[start:end], existing); err != nil { + if err := selectExistingRowsForWrite(ctx, q, store, lockKeys[start:end], existing); err != nil { return err } } @@ -245,42 +250,41 @@ func (d *transactionalDatastore) writeOnTx( return err } - if err := executeDeleteTuples(ctx, tx, store, deleteConditions); err != nil { + if err := executeDeleteTuples(ctx, q, store, deleteConditions); err != nil { return err } - if err := executeWriteTuples(ctx, tx, writeItems); err != nil { + if err := executeWriteTuples(ctx, q, writeItems); err != nil { return err } - if err := executeInsertChanges(ctx, tx, changeLogItems); err != nil { + if err := executeInsertChanges(ctx, q, changeLogItems); err != nil { return err } return nil } // selectExistingRowsForWrite is a 1:1 copy of upstream selectExistingRowsForWrite -// in github.com/openfga/openfga/pkg/storage/postgres/postgres.go. +// in postgres.go:1401 and sqlcommon.go:819-844. func selectExistingRowsForWrite( ctx context.Context, - stbl sq.StatementBuilderType, - connector sqlcommon.Connector, + q *txStatements, store string, keys []sqlcommon.TupleLockKey, existing map[string]*openfgav1.Tuple, ) error { inExpr, args := sqlcommon.BuildRowConstructorIN(keys) - sb := stbl. + sb := q.stbl. Select(sqlcommon.SQLIteratorColumns()...). From("tuple"). Where(sq.Eq{"store": store}). Where(sq.Expr("(object_type, object_id, relation, _user, user_type) IN "+inExpr, args...)). Suffix("FOR UPDATE") - poolGetRows, err := sqlcommon.NewRowGetter(connector, sb) + poolGetRows, err := sqlcommon.NewRowGetter(q.connector, sb) if err != nil { - return postgres.HandleSQLError(err) + return q.handleError(err) } - iter := sqlcommon.NewSQLTupleIterator(poolGetRows, postgres.HandleSQLError) + iter := sqlcommon.NewSQLTupleIterator(poolGetRows, q.handleError) defer iter.Stop() items, _, err := iter.ToArray(ctx, storage.PaginationOptions{PageSize: len(keys)}) @@ -293,9 +297,10 @@ func selectExistingRowsForWrite( return nil } -// executeDeleteTuples is a 1:1 copy of upstream executeDeleteTuples (postgres.go:458-488). -func executeDeleteTuples(ctx context.Context, tx pgx.Tx, store string, deleteConditions sq.Or) error { - stbl := sq.StatementBuilder.PlaceholderFormat(sq.Dollar) +// executeDeleteTuples is a 1:1 copy of upstream executeDeleteTuples +// (postgres.go:458-488) and the delete loop in sqlcommon.Write +// (sqlcommon.go:1032-1057). +func executeDeleteTuples(ctx context.Context, q *txStatements, store string, deleteConditions sq.Or) error { for start, totalDeletes := 0, len(deleteConditions); start < totalDeletes; start += storage.DefaultMaxTuplesPerWrite { end := start + storage.DefaultMaxTuplesPerWrite if end > totalDeletes { @@ -303,28 +308,29 @@ func executeDeleteTuples(ctx context.Context, tx pgx.Tx, store string, deleteCon } deleteConditionsBatch := deleteConditions[start:end] - stmt, args, err := stbl.Delete("tuple"). + stmt, args, err := q.stbl.Delete("tuple"). Where(sq.Eq{"store": store}). Where(deleteConditionsBatch). ToSql() if err != nil { - return postgres.HandleSQLError(err) + return q.handleError(err) } - res, err := tx.Exec(ctx, stmt, args...) + rowsAffected, err := q.exec(ctx, stmt, args...) if err != nil { - return postgres.HandleSQLError(err) + return q.handleError(err) } - if res.RowsAffected() != int64(len(deleteConditionsBatch)) { + if rowsAffected != int64(len(deleteConditionsBatch)) { return storage.ErrWriteConflictOnDelete } } return nil } -// executeWriteTuples is a 1:1 copy of upstream executeWriteTuples (postgres.go:491-538). -func executeWriteTuples(ctx context.Context, tx pgx.Tx, writeItems [][]interface{}) error { - stbl := sq.StatementBuilder.PlaceholderFormat(sq.Dollar) +// executeWriteTuples is a 1:1 copy of upstream executeWriteTuples +// (postgres.go:491-538) and the insert loop in sqlcommon.Write +// (sqlcommon.go:1059-1097). +func executeWriteTuples(ctx context.Context, q *txStatements, writeItems [][]interface{}) error { for start, totalWrites := 0, len(writeItems); start < totalWrites; start += storage.DefaultMaxTuplesPerWrite { end := start + storage.DefaultMaxTuplesPerWrite if end > totalWrites { @@ -332,7 +338,7 @@ func executeWriteTuples(ctx context.Context, tx pgx.Tx, writeItems [][]interface } writesBatch := writeItems[start:end] - insertBuilder := stbl. + insertBuilder := q.stbl. Insert("tuple"). Columns( "store", @@ -353,12 +359,12 @@ func executeWriteTuples(ctx context.Context, tx pgx.Tx, writeItems [][]interface stmt, args, err := insertBuilder.ToSql() if err != nil { - return postgres.HandleSQLError(err) + return q.handleError(err) } - _, err = tx.Exec(ctx, stmt, args...) + _, err = q.exec(ctx, stmt, args...) if err != nil { - dberr := postgres.HandleSQLError(err) + dberr := q.handleError(err) if errors.Is(dberr, storage.ErrCollision) { return storage.ErrWriteConflictOnInsert } @@ -368,9 +374,10 @@ func executeWriteTuples(ctx context.Context, tx pgx.Tx, writeItems [][]interface return nil } -// executeInsertChanges is a 1:1 copy of upstream executeInsertChanges (postgres.go:540-582). -func executeInsertChanges(ctx context.Context, tx pgx.Tx, changeLogItems [][]interface{}) error { - stbl := sq.StatementBuilder.PlaceholderFormat(sq.Dollar) +// executeInsertChanges is a 1:1 copy of upstream executeInsertChanges +// (postgres.go:540-582) and the changelog loop in sqlcommon.Write +// (sqlcommon.go:1100-1131). +func executeInsertChanges(ctx context.Context, q *txStatements, changeLogItems [][]interface{}) error { for start, totalItems := 0, len(changeLogItems); start < totalItems; start += storage.DefaultMaxTuplesPerWrite { end := start + storage.DefaultMaxTuplesPerWrite if end > totalItems { @@ -378,7 +385,7 @@ func executeInsertChanges(ctx context.Context, tx pgx.Tx, changeLogItems [][]int } changeLogBatch := changeLogItems[start:end] - changelogBuilder := stbl. + changelogBuilder := q.stbl. Insert("changelog"). Columns( "store", @@ -399,17 +406,41 @@ func executeInsertChanges(ctx context.Context, tx pgx.Tx, changeLogItems [][]int stmt, args, err := changelogBuilder.ToSql() if err != nil { - return postgres.HandleSQLError(err) + return q.handleError(err) } - _, err = tx.Exec(ctx, stmt, args...) + _, err = q.exec(ctx, stmt, args...) if err != nil { - return postgres.HandleSQLError(err) + return q.handleError(err) } } return nil } +// txStatements runs OpenFGA statements on one caller-owned transaction. It +// holds what differs between the upstream PostgreSQL and MySQL datastores. +type txStatements struct { + stbl sq.StatementBuilderType + connector sqlcommon.Connector + exec func(ctx context.Context, stmt string, args ...any) (rowsAffected int64, err error) + handleError func(err error, args ...any) error +} + +func pgxTxStatements(tx pgx.Tx) *txStatements { + return &txStatements{ + stbl: sq.StatementBuilder.PlaceholderFormat(sq.Dollar), + connector: &pgxTxConnector{tx: tx}, + exec: func(ctx context.Context, stmt string, args ...any) (int64, error) { + tag, err := tx.Exec(ctx, stmt, args...) + if err != nil { + return 0, err + } + return tag.RowsAffected(), nil + }, + handleError: postgres.HandleSQLError, + } +} + // pgxTxConnector, pgxTxConnection, and pgxRowsWrapper are 1:1 copies of the // unexported adapter types in github.com/openfga/openfga/pkg/storage/postgres/postgres.go:101-190. type pgxTxConnector struct { diff --git a/cmd/ateapi/internal/authz/datastore_mysql.go b/cmd/ateapi/internal/authz/datastore_mysql.go new file mode 100644 index 0000000000..0a85324197 --- /dev/null +++ b/cmd/ateapi/internal/authz/datastore_mysql.go @@ -0,0 +1,108 @@ +// Copyright 2026 Google LLC and The OpenFGA Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package authz + +import ( + "context" + "crypto/sha256" + "database/sql" + "database/sql/driver" + "encoding/hex" + "fmt" + + sq "github.com/Masterminds/squirrel" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/openfga/openfga/pkg/storage/mysql" + "github.com/openfga/openfga/pkg/storage/sqlcommon" +) + +// newMySQLDatastore builds the upstream OpenFGA MySQL datastore on db, whose +// schema atemy migration 000002 pins. mysql.NewWithDB (mysql.go:78-86) calls +// db.SetMaxOpenConns, SetConnMaxIdleTime, and SetConnMaxLifetime +// unconditionally, so the config carries db's open-connection limit forward +// and restates the lifetimes atemy.Connect sets. database/sql exposes no +// getter for the lifetimes. +func newMySQLDatastore(db *sql.DB) (*mysql.Datastore, error) { + cfg := sqlcommon.NewConfig() + cfg.MaxOpenConns = db.Stats().MaxOpenConnections + cfg.ConnMaxLifetime = storesql.ConnMaxLifetime + cfg.ConnMaxIdleTime = storesql.ConnMaxIdleTime + ds, err := mysql.NewWithDB(db, cfg) + if err != nil { + return nil, fmt.Errorf("creating OpenFGA mysql adapter: %w", err) + } + return ds, nil +} + +func sqlTxStatements(tx *sql.Tx) *txStatements { + return &txStatements{ + stbl: sq.StatementBuilder.PlaceholderFormat(sq.Question), + connector: sqlcommon.NewTxConnector(tx), + exec: func(ctx context.Context, stmt string, args ...any) (int64, error) { + res, err := tx.ExecContext(ctx, stmt, args...) + if err != nil { + return 0, err + } + return res.RowsAffected() + }, + handleError: mysql.HandleSQLError, + } +} + +// mysqlInitLockName names the user-level lock that serializes OpenFGA store +// provisioning across replicas. GET_LOCK names are server-wide, so the name +// carries the database, as a PostgreSQL advisory lock is scoped to one. MySQL +// caps names at 64 characters, so the database name is hashed. +func mysqlInitLockName(database string) string { + sum := sha256.Sum256([]byte(database)) + return "atefga-init:" + hex.EncodeToString(sum[:16]) +} + +// acquireMySQLInitLock is the MySQL counterpart of acquireInitLock, and like +// it waits until the lock is free or ctx ends. GET_LOCK belongs to the +// session, so it pins one connection until unlock. It uses SELECT GET_LOCK +// without FROM, the only form PlanetScale Vitess accepts. +func acquireMySQLInitLock(ctx context.Context, db *sql.DB) (func(), error) { + conn, err := db.Conn(ctx) + if err != nil { + return nil, fmt.Errorf("acquiring connection for OpenFGA init lock: %w", err) + } + fail := func(err error) (func(), error) { + _ = conn.Close() + return nil, fmt.Errorf("acquiring OpenFGA init lock: %w", err) + } + var database string + if err := conn.QueryRowContext(ctx, `SELECT DATABASE()`).Scan(&database); err != nil { + return fail(err) + } + name := mysqlInitLockName(database) + // A negative timeout waits until the lock is free. Canceling ctx closes the + // connection, and the server then abandons the wait. + var acquired sql.NullInt64 + if err := conn.QueryRowContext(ctx, `SELECT GET_LOCK(?, -1)`, name).Scan(&acquired); err != nil { + return fail(err) + } + if !acquired.Valid || acquired.Int64 != 1 { + return fail(fmt.Errorf("GET_LOCK did not grant %q", name)) + } + return func() { + _, _ = conn.ExecContext(context.Background(), `SELECT RELEASE_LOCK(?)`, name) + // Discard the session rather than pool it. That frees the lock even if + // RELEASE_LOCK failed, and Vitess keeps a session that took a lock on + // a reserved connection until the client disconnects. + _ = conn.Raw(func(any) error { return driver.ErrBadConn }) + _ = conn.Close() + }, nil +} diff --git a/cmd/ateapi/internal/authz/interceptor_test.go b/cmd/ateapi/internal/authz/interceptor_test.go index 8ff243d70c..3abafb988f 100644 --- a/cmd/ateapi/internal/authz/interceptor_test.go +++ b/cmd/ateapi/internal/authz/interceptor_test.go @@ -29,19 +29,19 @@ import ( func setupTestAuthorizer(t *testing.T) *Authorizer { t.Helper() ctx := context.Background() - pool := startPostgres(t) + db := startPostgres(t) - fgaServer, err := NewOpenFGAServer(pool) + fgaServer, err := NewOpenFGAServer(db.backend) if err != nil { t.Fatalf("NewOpenFGAServer failed: %v", err) } t.Cleanup(fgaServer.Close) - authorizer, policyManager, err := New(ctx, pool, fgaServer, nil) + authorizer, policyManager, err := New(ctx, db.backend, fgaServer, nil) if err != nil { t.Fatalf("New failed: %v", err) } - writeTestTuple(t, ctx, pool, policyManager, "alice@example.com", "owner", GlobalRootObject) + writeTestTuple(t, ctx, db, policyManager, "alice@example.com", "owner", GlobalRootObject) return authorizer } diff --git a/cmd/ateapi/internal/authz/policy_manager.go b/cmd/ateapi/internal/authz/policy_manager.go index f6fc0d2d9f..fc9af7fb33 100644 --- a/cmd/ateapi/internal/authz/policy_manager.go +++ b/cmd/ateapi/internal/authz/policy_manager.go @@ -18,41 +18,50 @@ import ( "context" "errors" "fmt" + "unicode/utf8" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" - "github.com/jackc/pgx/v5" openfgav1 "github.com/openfga/api/proto/openfga/v1" "github.com/openfga/openfga/pkg/server" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" "google.golang.org/protobuf/types/known/wrapperspb" ) // ErrNilTransaction is returned by PolicyManager tuple writes called without a -// PostgreSQL transaction. -var ErrNilTransaction = errors.New("authz: policy tuple writes require a non-nil pgx.Tx") +// store transaction. +var ErrNilTransaction = errors.New("authz: policy tuple writes require a store transaction") // PolicyManager writes OpenFGA tuples for access policy and atespace -// mutations within the caller's PostgreSQL transaction. +// mutations within the caller's store transaction. type PolicyManager struct { fgaServer *server.Server storeID string modelID string + // maxUserLength caps a tuple's encoded user in characters; zero means no + // cap beyond the API's own validation. + maxUserLength int } +// mysqlMaxTupleUserLength is the width of the tuple._user column in OpenFGA's +// MySQL schema. PostgreSQL stores the user as TEXT. +const mysqlMaxTupleUserLength = 256 + // ReconcileGlobalBindings reconciles the OpenFGA role bindings on global:root // within tx. The tuple changes commit or roll back with tx. -func (m *PolicyManager) ReconcileGlobalBindings(ctx context.Context, tx pgx.Tx, bindings []*ateapipb.Binding) error { +func (m *PolicyManager) ReconcileGlobalBindings(ctx context.Context, tx Tx, bindings []*ateapipb.Binding) error { return m.reconcileBindings(ctx, tx, GlobalRootObject, bindings) } // ReconcileAtespaceBindings reconciles the OpenFGA role bindings on an atespace // within tx. The tuple changes commit or roll back with tx. -func (m *PolicyManager) ReconcileAtespaceBindings(ctx context.Context, tx pgx.Tx, name string, bindings []*ateapipb.Binding) error { +func (m *PolicyManager) ReconcileAtespaceBindings(ctx context.Context, tx Tx, name string, bindings []*ateapipb.Binding) error { return m.reconcileBindings(ctx, tx, AtespaceObject(name), bindings) } // DeleteAtespacePolicies removes all OpenFGA tuples associated with an atespace // within tx. The tuple deletions commit or roll back with tx. -func (m *PolicyManager) DeleteAtespacePolicies(ctx context.Context, tx pgx.Tx, name string) error { +func (m *PolicyManager) DeleteAtespacePolicies(ctx context.Context, tx Tx, name string) error { return m.reconcileBindings(ctx, tx, AtespaceObject(name), nil) } @@ -115,8 +124,8 @@ func (m *PolicyManager) readObjectTuples(ctx context.Context, obj string) ([]*op // reconcileBindings diffs the existing OpenFGA tuples on obj against // desiredBindings and writes only the net additions and deletions in batches // within tx, leaving unchanged (relation, user) tuples untouched. -func (m *PolicyManager) reconcileBindings(ctx context.Context, tx pgx.Tx, obj string, desiredBindings []*ateapipb.Binding) error { - if tx == nil { +func (m *PolicyManager) reconcileBindings(ctx context.Context, tx Tx, obj string, desiredBindings []*ateapipb.Binding) error { + if tx.isZero() { return ErrNilTransaction } // The OpenFGA server API only carries ctx, so the transactional datastore @@ -146,6 +155,9 @@ func (m *PolicyManager) reconcileBindings(ctx context.Context, tx pgx.Tx, obj st if err != nil { return err } + if n := utf8.RuneCountInString(fgaUser); m.maxUserLength > 0 && n > m.maxUserLength { + return status.Errorf(codes.InvalidArgument, "member %q is %d characters once encoded; this store holds at most %d", rawMember, n, m.maxUserLength) + } ru := relUser{relation: role, user: fgaUser} if _, alreadyDesired := desiredSet[ru]; alreadyDesired { continue diff --git a/cmd/ateapi/internal/authz/tx_context.go b/cmd/ateapi/internal/authz/tx_context.go index 15a1fd2d3e..d234dc84c0 100644 --- a/cmd/ateapi/internal/authz/tx_context.go +++ b/cmd/ateapi/internal/authz/tx_context.go @@ -16,22 +16,48 @@ package authz import ( "context" + "database/sql" "github.com/jackc/pgx/v5" ) -type txContextKey struct{} +// Tx is the store transaction that OpenFGA tuple reads and writes join. Build +// it with PgxTx for the PostgreSQL store or SQLTx for the MySQL store; the +// zero value carries no transaction. +type Tx struct { + q *txStatements +} + +// PgxTx wraps a PostgreSQL store transaction. +func PgxTx(tx pgx.Tx) Tx { + if tx == nil { + return Tx{} + } + return Tx{q: pgxTxStatements(tx)} +} -// ContextWithTx injects an active pgx.Tx into ctx. -func ContextWithTx(ctx context.Context, tx pgx.Tx) context.Context { +// SQLTx wraps a MySQL store transaction. +func SQLTx(tx *sql.Tx) Tx { if tx == nil { + return Tx{} + } + return Tx{q: sqlTxStatements(tx)} +} + +func (t Tx) isZero() bool { return t.q == nil } + +type txContextKey struct{} + +// ContextWithTx injects an active store transaction into ctx. +func ContextWithTx(ctx context.Context, tx Tx) context.Context { + if tx.isZero() { return ctx } return context.WithValue(ctx, txContextKey{}, tx) } -// TxFromContext retrieves the active pgx.Tx from ctx, if present. -func TxFromContext(ctx context.Context) (pgx.Tx, bool) { - tx, ok := ctx.Value(txContextKey{}).(pgx.Tx) - return tx, ok && tx != nil +// TxFromContext retrieves the active store transaction from ctx, if present. +func TxFromContext(ctx context.Context) (Tx, bool) { + tx, ok := ctx.Value(txContextKey{}).(Tx) + return tx, ok && !tx.isZero() } diff --git a/cmd/ateapi/internal/controlapi/access_policy_test.go b/cmd/ateapi/internal/controlapi/access_policy_test.go index 554db49a67..15fa8f36c3 100644 --- a/cmd/ateapi/internal/controlapi/access_policy_test.go +++ b/cmd/ateapi/internal/controlapi/access_policy_test.go @@ -29,11 +29,10 @@ import ( ) func TestAccessPolicy_GlobalAndAtespaceGovernance(t *testing.T) { - persistence := storetest.SetupPostgresPersistence(t) + persistence, authzBackend := storetest.SetupAuthzTestStore(t) ctx := context.Background() - pool := persistence.Pool() - fgaServer, err := authz.NewOpenFGAServer(pool) + fgaServer, err := authz.NewOpenFGAServer(authzBackend) if err != nil { t.Fatalf("NewOpenFGAServer failed: %v", err) } @@ -41,7 +40,7 @@ func TestAccessPolicy_GlobalAndAtespaceGovernance(t *testing.T) { // alice@example.com is a bootstrap owner: a global owner through server // configuration, never through a stored AccessPolicy. - authorizer, policyManager, err := authz.New(ctx, pool, fgaServer, []string{"alice@example.com"}) + authorizer, policyManager, err := authz.New(ctx, authzBackend, fgaServer, []string{"alice@example.com"}) if err != nil { t.Fatalf("authz.New failed: %v", err) } @@ -195,7 +194,7 @@ func TestAccessPolicy_GlobalAndAtespaceGovernance(t *testing.T) { // 6. Removing Alice from the bootstrap owners (a restart with a different // configuration) leaves only her stored grant; once that is gone too she // is denied, while the new bootstrap owner is allowed. - restarted, _, err := authz.New(ctx, pool, fgaServer, []string{"charlie@example.com"}) + restarted, _, err := authz.New(ctx, authzBackend, fgaServer, []string{"charlie@example.com"}) if err != nil { t.Fatalf("authz.New after restart failed: %v", err) } diff --git a/cmd/ateapi/internal/controlapi/atespace_test.go b/cmd/ateapi/internal/controlapi/atespace_test.go index 64a7ddf19a..cff18acbf8 100644 --- a/cmd/ateapi/internal/controlapi/atespace_test.go +++ b/cmd/ateapi/internal/controlapi/atespace_test.go @@ -52,13 +52,13 @@ func TestAtespace_EndToEndOpenFGAScenarios(t *testing.T) { persistence := storetest.SetupPostgresPersistence(t) pool := persistence.Pool() - fgaServer, err := authz.NewOpenFGAServer(pool) + fgaServer, err := authz.NewOpenFGAServer(authz.PostgresBackend(pool)) if err != nil { t.Fatalf("authz.NewOpenFGAServer failed: %v", err) } t.Cleanup(fgaServer.Close) - authorizer, policyManager, err := authz.New(ctx, pool, fgaServer, nil) + authorizer, policyManager, err := authz.New(ctx, authz.PostgresBackend(pool), fgaServer, nil) if err != nil { t.Fatalf("authz.New failed: %v", err) } @@ -125,7 +125,7 @@ func TestAtespace_EndToEndOpenFGAScenarios(t *testing.T) { t.Fatalf("pool.Begin failed: %v", err) } defer tx.Rollback(ctx) //nolint:errcheck - _, err = fgaServer.Write(authz.ContextWithTx(ctx, tx), &openfgav1.WriteRequest{ + _, err = fgaServer.Write(authz.ContextWithTx(ctx, authz.PgxTx(tx)), &openfgav1.WriteRequest{ StoreId: storeID, Writes: &openfgav1.WriteRequestWrites{ TupleKeys: []*openfgav1.TupleKey{ diff --git a/cmd/ateapi/internal/store/atemy/access_policy.go b/cmd/ateapi/internal/store/atemy/access_policy.go new file mode 100644 index 0000000000..046af56c04 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/access_policy.go @@ -0,0 +1,235 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +const globalAccessPolicyID = 1 + +func (p *Persistence) CreateGlobalAccessPolicy(ctx context.Context, policy *ateapipb.AccessPolicy) (*ateapipb.AccessPolicy, error) { + dbPolicy := proto.Clone(policy).(*ateapipb.AccessPolicy) + dbPolicy.Metadata = &ateapipb.ResourceMetadata{Name: "default"} + storesql.SetCreateMetadata(dbPolicy.Metadata) + protoBytes, err := proto.Marshal(dbPolicy) + if err != nil { + return nil, fmt.Errorf("marshaling global access policy: %w", err) + } + err = inTx(ctx, p.db, func(tx *sql.Tx) error { + if _, err := tx.ExecContext(ctx, ` + INSERT INTO global_access_policy (id, uid, version, proto) + VALUES (?, ?, ?, ?)`, globalAccessPolicyID, dbPolicy.GetMetadata().GetUid(), dbPolicy.GetMetadata().GetVersion(), protoBytes); err != nil { + if isUniqueViolation(err) { + return store.ErrAlreadyExists + } + return fmt.Errorf("inserting global access policy: %w", err) + } + if err := p.policyManager.ReconcileGlobalBindings(ctx, authz.SQLTx(tx), dbPolicy.GetBindings()); err != nil { + return fmt.Errorf("reconciling global access policy bindings: %w", err) + } + return nil + }) + if err != nil { + return nil, err + } + return dbPolicy, nil +} + +func (p *Persistence) GetGlobalAccessPolicy(ctx context.Context) (*ateapipb.AccessPolicy, error) { + return getAccessPolicyRow(ctx, p.db, ` + SELECT uid, version, proto FROM global_access_policy + WHERE id = ?`, globalAccessPolicyID) +} + +func (p *Persistence) UpdateGlobalAccessPolicy(ctx context.Context, precondition store.Precondition, mutate func(*ateapipb.AccessPolicy) error) (*ateapipb.AccessPolicy, error) { + if err := precondition.Validate(); err != nil { + return nil, err + } + var dbPolicy *ateapipb.AccessPolicy + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + var err error + dbPolicy, err = getAccessPolicyRow(ctx, tx, ` + SELECT uid, version, proto FROM global_access_policy + WHERE id = ? FOR UPDATE`, globalAccessPolicyID) + if err != nil { + return err + } + if err := mutateAccessPolicy(dbPolicy, precondition, mutate); err != nil { + return err + } + protoBytes, err := proto.Marshal(dbPolicy) + if err != nil { + return fmt.Errorf("marshaling updated global access policy: %w", err) + } + if _, err := tx.ExecContext(ctx, ` + UPDATE global_access_policy SET version = ?, proto = ? + WHERE id = ?`, dbPolicy.GetMetadata().GetVersion(), protoBytes, globalAccessPolicyID); err != nil { + return fmt.Errorf("updating global access policy: %w", err) + } + if err := p.policyManager.ReconcileGlobalBindings(ctx, authz.SQLTx(tx), dbPolicy.GetBindings()); err != nil { + return fmt.Errorf("reconciling global access policy bindings: %w", err) + } + return nil + }) + if err != nil { + return nil, err + } + return dbPolicy, nil +} + +// CreateAtespaceAccessPolicy returns ErrFailedPrecondition if the atespace +// does not exist. +func (p *Persistence) CreateAtespaceAccessPolicy(ctx context.Context, name string, policy *ateapipb.AccessPolicy) (*ateapipb.AccessPolicy, error) { + dbPolicy := proto.Clone(policy).(*ateapipb.AccessPolicy) + dbPolicy.Metadata = &ateapipb.ResourceMetadata{Name: "default"} + storesql.SetCreateMetadata(dbPolicy.Metadata) + protoBytes, err := proto.Marshal(dbPolicy) + if err != nil { + return nil, fmt.Errorf("marshaling access policy: %w", err) + } + err = inTx(ctx, p.db, func(tx *sql.Tx) error { + if err := lockAtespace(ctx, tx, name); err != nil { + return err + } + if _, err := tx.ExecContext(ctx, ` + INSERT INTO atespace_access_policies (atespace_name, uid, version, proto) + VALUES (?, ?, ?, ?)`, name, dbPolicy.GetMetadata().GetUid(), dbPolicy.GetMetadata().GetVersion(), protoBytes); err != nil { + if isUniqueViolation(err) { + return store.ErrAlreadyExists + } + return fmt.Errorf("inserting access policy for %s: %w", name, err) + } + if err := p.policyManager.ReconcileAtespaceBindings(ctx, authz.SQLTx(tx), name, dbPolicy.GetBindings()); err != nil { + return fmt.Errorf("reconciling access policy bindings for %s: %w", name, err) + } + return nil + }) + if err != nil { + return nil, err + } + return dbPolicy, nil +} + +func (p *Persistence) GetAtespaceAccessPolicy(ctx context.Context, name string) (*ateapipb.AccessPolicy, error) { + return getAccessPolicyRow(ctx, p.db, ` + SELECT uid, version, proto FROM atespace_access_policies + WHERE atespace_name = ?`, name) +} + +func (p *Persistence) UpdateAtespaceAccessPolicy(ctx context.Context, name string, precondition store.Precondition, mutate func(*ateapipb.AccessPolicy) error) (*ateapipb.AccessPolicy, error) { + if err := precondition.Validate(); err != nil { + return nil, err + } + var dbPolicy *ateapipb.AccessPolicy + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + var err error + dbPolicy, err = getAccessPolicyRow(ctx, tx, ` + SELECT uid, version, proto FROM atespace_access_policies + WHERE atespace_name = ? FOR UPDATE`, name) + if err != nil { + return err + } + if err := mutateAccessPolicy(dbPolicy, precondition, mutate); err != nil { + return err + } + protoBytes, err := proto.Marshal(dbPolicy) + if err != nil { + return fmt.Errorf("marshaling updated access policy: %w", err) + } + if _, err := tx.ExecContext(ctx, ` + UPDATE atespace_access_policies SET version = ?, proto = ? + WHERE atespace_name = ?`, dbPolicy.GetMetadata().GetVersion(), protoBytes, name); err != nil { + return fmt.Errorf("updating access policy for %s: %w", name, err) + } + if err := p.policyManager.ReconcileAtespaceBindings(ctx, authz.SQLTx(tx), name, dbPolicy.GetBindings()); err != nil { + return fmt.Errorf("reconciling access policy bindings for %s: %w", name, err) + } + return nil + }) + if err != nil { + return nil, err + } + return dbPolicy, nil +} + +func (p *Persistence) DeleteAtespaceAccessPolicy(ctx context.Context, name string, precondition store.DeletePreconditions) (*ateapipb.AccessPolicy, error) { + var deleted *ateapipb.AccessPolicy + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + uid, version, protoBytes, err := deleteLockedRow(ctx, tx, "atespace_access_policies", "atespace_name = ?", precondition, name) + if err != nil { + return err + } + if deleted, err = unmarshalAccessPolicy(uid, version, protoBytes); err != nil { + return err + } + if err := p.policyManager.DeleteAtespacePolicies(ctx, authz.SQLTx(tx), name); err != nil { + return fmt.Errorf("deleting authorization tuples for %s: %w", name, err) + } + return nil + }) + if err != nil { + return nil, err + } + return deleted, nil +} + +// mutateAccessPolicy checks precondition against the locked policy, applies +// mutate, and advances the stored metadata, discarding any metadata edits. +func mutateAccessPolicy(dbPolicy *ateapipb.AccessPolicy, precondition store.Precondition, mutate func(*ateapipb.AccessPolicy) error) error { + if err := precondition.Check(dbPolicy.GetMetadata()); err != nil { + return err + } + oldMeta := proto.CloneOf(dbPolicy.Metadata) + if err := mutate(dbPolicy); err != nil { + return err + } + dbPolicy.Metadata = oldMeta + storesql.SetUpdateMetadata(dbPolicy.Metadata, oldMeta) + return nil +} + +func getAccessPolicyRow(ctx context.Context, q querier, query string, args ...any) (*ateapipb.AccessPolicy, error) { + var uid string + var version int64 + var protoBytes []byte + if err := q.QueryRowContext(ctx, query, args...).Scan(&uid, &version, &protoBytes); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting access policy: %w", err) + } + return unmarshalAccessPolicy(uid, version, protoBytes) +} + +func unmarshalAccessPolicy(uid string, version int64, protoBytes []byte) (*ateapipb.AccessPolicy, error) { + policy := &ateapipb.AccessPolicy{} + if err := storesql.UnmarshalStored(protoBytes, policy); err != nil { + return nil, fmt.Errorf("unmarshaling access policy: %w", err) + } + if err := storesql.ValidateProtoMetadataMatchesColumns("access policy", policy.GetMetadata(), uid, version); err != nil { + return nil, err + } + return policy, nil +} diff --git a/cmd/ateapi/internal/store/atemy/access_policy_test.go b/cmd/ateapi/internal/store/atemy/access_policy_test.go new file mode 100644 index 0000000000..3e57b1b3f0 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/access_policy_test.go @@ -0,0 +1,111 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "errors" + "sync" + "testing" + + "github.com/google/go-cmp/cmp" + "google.golang.org/protobuf/testing/protocmp" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// TestGlobalAccessPolicy_UpdateIsAtomic checks that the policy row and its +// OpenFGA tuples commit or roll back together. +func TestGlobalAccessPolicy_UpdateIsAtomic(t *testing.T) { + p := setupMySQLPersistence(t) + ctx := t.Context() + + created, err := p.CreateGlobalAccessPolicy(ctx, &ateapipb.AccessPolicy{ + Metadata: &ateapipb.ResourceMetadata{Name: "default"}, + Bindings: []*ateapipb.Binding{{Role: authz.RoleOwner, Members: []string{"user:alice"}}}, + }) + if err != nil { + t.Fatalf("CreateGlobalAccessPolicy failed: %v", err) + } + wantTuples := []string{"owner user:alice"} + if diff := cmp.Diff(wantTuples, tuplesOn(t, p, "global", "root")); diff != "" { + t.Fatalf("tuples after create (-want +got):\n%s", diff) + } + pre := store.Precondition{UID: created.GetMetadata().GetUid(), Version: created.GetMetadata().GetVersion()} + + t.Run("failed tuple write rolls back the row and earlier tuple deletes", func(t *testing.T) { + // Reconciling deletes alice's tuple and then fails writing a relation + // the model does not define, after the row UPDATE has run. A failed + // statement does not abort a MySQL transaction, so only the rollback + // undoes the statements before it. + _, err := p.UpdateGlobalAccessPolicy(ctx, pre, func(ap *ateapipb.AccessPolicy) error { + ap.Bindings = []*ateapipb.Binding{{Role: "not-a-relation", Members: []string{"user:bob"}}} + return nil + }) + if err == nil { + t.Fatal("UpdateGlobalAccessPolicy with an undefined relation succeeded, want an error") + } + got, err := p.GetGlobalAccessPolicy(ctx) + if err != nil { + t.Fatalf("GetGlobalAccessPolicy failed: %v", err) + } + if diff := cmp.Diff(created, got, protocmp.Transform()); diff != "" { + t.Errorf("policy row changed by failed update (-want +got):\n%s", diff) + } + if diff := cmp.Diff(wantTuples, tuplesOn(t, p, "global", "root")); diff != "" { + t.Errorf("tuples changed by failed update (-want +got):\n%s", diff) + } + }) + + t.Run("concurrent updates leave tuples matching the winning row", func(t *testing.T) { + members := []string{"user:bob", "user:carol"} + errs := make([]error, len(members)) + var wg sync.WaitGroup + for i, member := range members { + wg.Go(func() { + _, errs[i] = p.UpdateGlobalAccessPolicy(ctx, pre, func(ap *ateapipb.AccessPolicy) error { + ap.Bindings = []*ateapipb.Binding{{Role: authz.RoleOwner, Members: []string{member}}} + return nil + }) + }) + } + wg.Wait() + + winners := 0 + for i, err := range errs { + switch { + case err == nil: + winners++ + case !errors.Is(err, store.ErrVersionConflict): + t.Errorf("update %d = %v, want nil or ErrVersionConflict", i, err) + } + } + if winners != 1 { + t.Fatalf("%d updates succeeded, want exactly 1 (errs: %v)", winners, errs) + } + got, err := p.GetGlobalAccessPolicy(ctx) + if err != nil { + t.Fatalf("GetGlobalAccessPolicy failed: %v", err) + } + if got.GetMetadata().GetVersion() != pre.Version+1 { + t.Errorf("version = %d, want %d", got.GetMetadata().GetVersion(), pre.Version+1) + } + want := []string{"owner " + got.GetBindings()[0].GetMembers()[0]} + if diff := cmp.Diff(want, tuplesOn(t, p, "global", "root")); diff != "" { + t.Errorf("tuples do not match the committed row (-row +tuples):\n%s", diff) + } + }) +} diff --git a/cmd/ateapi/internal/store/atemy/actor.go b/cmd/ateapi/internal/store/atemy/actor.go new file mode 100644 index 0000000000..9148c45632 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/actor.go @@ -0,0 +1,158 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +func (p *Persistence) CreateActor(ctx context.Context, actor *ateapipb.Actor) (*ateapipb.Actor, error) { + atespace := actor.GetMetadata().GetAtespace() + name := actor.GetMetadata().GetName() + + dbActor := proto.Clone(actor).(*ateapipb.Actor) + storesql.SetCreateMetadata(dbActor.Metadata) + + protoBytes, err := proto.Marshal(dbActor) + if err != nil { + return nil, fmt.Errorf("marshaling actor: %w", err) + } + + err = p.insertChild(ctx, "actor "+atespace+"/"+name, func(tx *sql.Tx) error { return lockAtespace(ctx, tx, atespace) }, ` + INSERT INTO actors (atespace, name, uid, version, proto) + VALUES (?, ?, ?, ?, ?)`, + atespace, name, dbActor.GetMetadata().GetUid(), dbActor.GetMetadata().GetVersion(), protoBytes) + if err != nil { + return nil, err + } + return dbActor, nil +} + +func (p *Persistence) GetActor(ctx context.Context, actorRef resources.ActorRef) (*ateapipb.Actor, error) { + var protoBytes []byte + err := p.db.QueryRowContext(ctx, `SELECT proto FROM actors WHERE atespace = ? AND name = ?`, actorRef.Atespace, actorRef.Name).Scan(&protoBytes) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting actor %s/%s: %w", actorRef.Atespace, actorRef.Name, err) + } + out := &ateapipb.Actor{} + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { + return nil, fmt.Errorf("unmarshaling actor: %w", err) + } + return out, nil +} + +// UpdateActor is a compare-and-set on uid and version rather than a locked +// read, so mutate never runs while a row lock is held. +func (p *Persistence) UpdateActor(ctx context.Context, actorRef resources.ActorRef, precondition store.Precondition, mutate func(*ateapipb.Actor) error) (*ateapipb.Actor, error) { + if err := precondition.Validate(); err != nil { + return nil, err + } + atespace, name := actorRef.Atespace, actorRef.Name + var currentUID string + var currentVersion int64 + var currentBytes []byte + if err := p.db.QueryRowContext(ctx, ` + SELECT uid, version, proto FROM actors + WHERE atespace = ? AND name = ?`, atespace, name).Scan(¤tUID, ¤tVersion, ¤tBytes); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting actor %s/%s for update: %w", atespace, name, err) + } + + dbActor := &ateapipb.Actor{} + if err := storesql.UnmarshalStored(currentBytes, dbActor); err != nil { + return nil, fmt.Errorf("unmarshaling actor for update: %w", err) + } + if err := storesql.ValidateProtoMetadataMatchesColumns("actor "+actorRef.String(), dbActor.GetMetadata(), currentUID, currentVersion); err != nil { + return nil, err + } + if err := precondition.Check(dbActor.GetMetadata()); err != nil { + return nil, err + } + oldMeta := proto.CloneOf(dbActor.Metadata) + if err := mutate(dbActor); err != nil { + return nil, err + } + // Stored metadata is authoritative; discard any metadata edits made by the + // closure and derive the next revision from the state this attempt read. + storesql.SetUpdateMetadata(dbActor.Metadata, oldMeta) + + updatedBytes, err := proto.Marshal(dbActor) + if err != nil { + return nil, fmt.Errorf("marshaling actor: %w", err) + } + if err := updateGuarded(ctx, p.db, "actor "+actorRef.String(), ` + UPDATE actors + SET version = ?, proto = ? + WHERE atespace = ? AND name = ? AND uid = ? AND version = ?`, + dbActor.GetMetadata().GetVersion(), updatedBytes, atespace, name, currentUID, currentVersion); err != nil { + return nil, err + } + return dbActor, nil +} + +// DeleteActor removes the actor's egress policy with it. +func (p *Persistence) DeleteActor(ctx context.Context, actorRef resources.ActorRef, precondition store.DeletePreconditions) (*ateapipb.Actor, error) { + atespace, name := actorRef.Atespace, actorRef.Name + var protoBytes []byte + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + var err error + _, _, protoBytes, err = deleteLockedRow(ctx, tx, "actors", "atespace = ? AND name = ?", precondition, atespace, name) + if err != nil { + return err + } + if _, err := tx.ExecContext(ctx, `DELETE FROM actor_egress_policies WHERE atespace = ? AND actor_name = ?`, atespace, name); err != nil { + return fmt.Errorf("deleting egress policy of actor %s/%s: %w", atespace, name, err) + } + return nil + }) + if err != nil { + return nil, err + } + out := &ateapipb.Actor{} + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { + return nil, fmt.Errorf("unmarshaling deleted actor: %w", err) + } + return out, nil +} + +func (p *Persistence) ListActors(ctx context.Context, atespace string, opts store.ListOptions) (store.ListResponse[*ateapipb.Actor], error) { + opts, err := store.NormalizeListOptions(opts) + if err != nil { + return store.ListResponse[*ateapipb.Actor]{}, err + } + decode := func(b []byte, keys []string) (*ateapipb.Actor, error) { + a := &ateapipb.Actor{} + return a, storesql.UnmarshalRow(b, a, "actor", rowID(atespace, keys)...) + } + items, next, err := listScoped(ctx, p.db, "actors", storesql.KindActor, atespace, opts, decode) + if err != nil { + return store.ListResponse[*ateapipb.Actor]{}, err + } + return store.ListResponse[*ateapipb.Actor]{Items: items, NextPageToken: next}, nil +} diff --git a/cmd/ateapi/internal/store/atemy/actor_template.go b/cmd/ateapi/internal/store/atemy/actor_template.go new file mode 100644 index 0000000000..236249fc91 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/actor_template.go @@ -0,0 +1,148 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +func (p *Persistence) CreateActorTemplate(ctx context.Context, template *ateapipb.ActorTemplate) (*ateapipb.ActorTemplate, error) { + atespace, name := template.GetMetadata().GetAtespace(), template.GetMetadata().GetName() + dbTemplate := proto.Clone(template).(*ateapipb.ActorTemplate) + if dbTemplate.Metadata == nil { + dbTemplate.Metadata = &ateapipb.ResourceMetadata{} + } + storesql.SetCreateMetadata(dbTemplate.Metadata) + protoBytes, err := proto.Marshal(dbTemplate) + if err != nil { + return nil, fmt.Errorf("marshaling actor template: %w", err) + } + err = p.insertChild(ctx, "actor template "+atespace+"/"+name, func(tx *sql.Tx) error { return lockAtespace(ctx, tx, atespace) }, ` + INSERT INTO actor_templates (atespace, name, uid, version, proto) + VALUES (?, ?, ?, ?, ?)`, + atespace, name, dbTemplate.GetMetadata().GetUid(), dbTemplate.GetMetadata().GetVersion(), protoBytes) + if err != nil { + return nil, err + } + return dbTemplate, nil +} + +func (p *Persistence) GetActorTemplate(ctx context.Context, templateRef resources.ActorTemplateRef) (*ateapipb.ActorTemplate, error) { + var protoBytes []byte + err := p.db.QueryRowContext(ctx, `SELECT proto FROM actor_templates WHERE atespace = ? AND name = ?`, templateRef.Atespace, templateRef.Name).Scan(&protoBytes) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting actor template %s: %w", templateRef, err) + } + out := &ateapipb.ActorTemplate{} + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { + return nil, fmt.Errorf("unmarshaling actor template: %w", err) + } + return out, nil +} + +func (p *Persistence) UpdateActorTemplate(ctx context.Context, templateRef resources.ActorTemplateRef, precondition store.Precondition, mutate func(*ateapipb.ActorTemplate) error) (*ateapipb.ActorTemplate, error) { + if err := precondition.Validate(); err != nil { + return nil, err + } + var currentUID string + var currentVersion int64 + var currentBytes []byte + if err := p.db.QueryRowContext(ctx, ` + SELECT uid, version, proto FROM actor_templates + WHERE atespace = ? AND name = ?`, templateRef.Atespace, templateRef.Name).Scan(¤tUID, ¤tVersion, ¤tBytes); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting actor template %s for update: %w", templateRef, err) + } + + dbTemplate := &ateapipb.ActorTemplate{} + if err := storesql.UnmarshalStored(currentBytes, dbTemplate); err != nil { + return nil, fmt.Errorf("unmarshaling actor template for update: %w", err) + } + if err := storesql.ValidateProtoMetadataMatchesColumns("actor template "+templateRef.String(), dbTemplate.GetMetadata(), currentUID, currentVersion); err != nil { + return nil, err + } + if err := precondition.Check(dbTemplate.GetMetadata()); err != nil { + return nil, err + } + templateBeforeMutation := proto.Clone(dbTemplate).(*ateapipb.ActorTemplate) + if err := mutate(dbTemplate); err != nil { + return nil, err + } + if err := storesql.ValidateUpdateActorTemplateMutation(templateBeforeMutation, dbTemplate); err != nil { + return nil, err + } + if dbTemplate.Metadata == nil { + dbTemplate.Metadata = &ateapipb.ResourceMetadata{} + } + storesql.SetUpdateMetadata(dbTemplate.Metadata, templateBeforeMutation.GetMetadata()) + updatedBytes, err := proto.Marshal(dbTemplate) + if err != nil { + return nil, fmt.Errorf("marshaling actor template: %w", err) + } + if err := updateGuarded(ctx, p.db, "actor template "+templateRef.String(), ` + UPDATE actor_templates SET version = ?, proto = ? + WHERE atespace = ? AND name = ? AND uid = ? AND version = ?`, + dbTemplate.GetMetadata().GetVersion(), updatedBytes, templateRef.Atespace, templateRef.Name, currentUID, currentVersion); err != nil { + return nil, err + } + return dbTemplate, nil +} + +func (p *Persistence) ListActorTemplates(ctx context.Context, atespace string, opts store.ListOptions) (store.ListResponse[*ateapipb.ActorTemplate], error) { + opts, err := store.NormalizeListOptions(opts) + if err != nil { + return store.ListResponse[*ateapipb.ActorTemplate]{}, err + } + decode := func(b []byte, keys []string) (*ateapipb.ActorTemplate, error) { + template := &ateapipb.ActorTemplate{} + return template, storesql.UnmarshalRow(b, template, "actor template", rowID(atespace, keys)...) + } + items, next, err := listScoped(ctx, p.db, "actor_templates", storesql.KindActorTemplate, atespace, opts, decode) + if err != nil { + return store.ListResponse[*ateapipb.ActorTemplate]{}, err + } + return store.ListResponse[*ateapipb.ActorTemplate]{Items: items, NextPageToken: next}, nil +} + +func (p *Persistence) DeleteActorTemplate(ctx context.Context, templateRef resources.ActorTemplateRef, precondition store.DeletePreconditions) (*ateapipb.ActorTemplate, error) { + var protoBytes []byte + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + var err error + _, _, protoBytes, err = deleteLockedRow(ctx, tx, "actor_templates", "atespace = ? AND name = ?", precondition, templateRef.Atespace, templateRef.Name) + return err + }) + if err != nil { + return nil, err + } + out := &ateapipb.ActorTemplate{} + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { + return nil, fmt.Errorf("unmarshaling deleted actor template: %w", err) + } + return out, nil +} diff --git a/cmd/ateapi/internal/store/atemy/actor_template_test.go b/cmd/ateapi/internal/store/atemy/actor_template_test.go new file mode 100644 index 0000000000..78022e5f90 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/actor_template_test.go @@ -0,0 +1,30 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "testing" + + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +func createTestActorTemplate(t *testing.T, s *Persistence, atespace, name string) { + t.Helper() + if _, err := s.CreateActorTemplate(t.Context(), &ateapipb.ActorTemplate{ + Metadata: &ateapipb.ResourceMetadata{Atespace: atespace, Name: name}, + }); err != nil { + t.Fatalf("CreateActorTemplate(%q/%q) failed: %v", atespace, name, err) + } +} diff --git a/cmd/ateapi/internal/store/atepg/pagetoken_test.go b/cmd/ateapi/internal/store/atemy/actor_test.go similarity index 59% rename from cmd/ateapi/internal/store/atepg/pagetoken_test.go rename to cmd/ateapi/internal/store/atemy/actor_test.go index 0056858af4..40d0209125 100644 --- a/cmd/ateapi/internal/store/atepg/pagetoken_test.go +++ b/cmd/ateapi/internal/store/atemy/actor_test.go @@ -12,13 +12,14 @@ // See the License for the specific language governing permissions and // limitations under the License. -package atepg +package atemy -import "testing" +import "github.com/agent-substrate/substrate/pkg/proto/ateapipb" -func TestDecodePageTokenRejectsWrongKeyShape(t *testing.T) { - token := encodePageToken(kindActor, "", []string{"only-an-atespace"}) - if _, err := decodePageToken(token, kindActor, "", 2); err == nil { - t.Fatal("decodePageToken() accepted a global actor token with only one key part") +func newTestActor(atespace, name string) *ateapipb.Actor { + return &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: atespace, Name: name}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: atespace, Name: "template-a"}, + Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, } } diff --git a/cmd/ateapi/internal/store/atemy/atemy.go b/cmd/ateapi/internal/store/atemy/atemy.go new file mode 100644 index 0000000000..df44ce8d10 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/atemy.go @@ -0,0 +1,445 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package atemy is an ate storage backend built on MySQL 8.0 or later. It +// uses only features PlanetScale's Vitess supports: no foreign keys, stored +// routines, triggers, partitioning or CREATE DATABASE, and no DDL outside the +// migrations applied at startup. +// +// Each table holds native SQL columns for fields SQL must operate on +// (primary keys, versions, pagination, update/delete preconditions) plus +// the complete protobuf message, binary-encoded, in a LONGBLOB column. +// Parent and child rows are kept consistent by the transactions that write +// them, in place of foreign keys. +package atemy + +import ( + "context" + "crypto/tls" + "crypto/x509" + "database/sql" + "errors" + "fmt" + "log/slog" + "net" + "os" + "strings" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/go-sql-driver/mysql" + "k8s.io/apimachinery/pkg/util/wait" +) + +const ( + // watchPoolMaxConns sizes the dedicated watch pool: one connection for the + // WatchWorkers poller, one for the maintenance loop, and one of headroom so + // a transiently slow poll can never gate a maintenance pass. + watchPoolMaxConns = 3 + // Migrations need one connection for the session lock and one for + // migration work. + ownerPoolMaxConns = 2 +) + +// Persistence is a service that stores ate state in MySQL. +type Persistence struct { + db *sql.DB + // watchDB serves WatchWorkers pollers, outbox retention and expired-lease + // cleanup. + watchDB *sql.DB + ownsWatchDB bool + policyManager *authz.PolicyManager + leaseTTL time.Duration + pollFailureCloseAfter time.Duration + stopMaintenance context.CancelFunc + maintenanceDone chan struct{} + // watchers are the live WatchWorkers channels. + watchers storesql.Watchers +} + +var _ store.Interface = (*Persistence)(nil) + +// ErrUnavailable reports that ateapi could not establish the initial MySQL +// connection. Callers can retry this error before startup. +var ErrUnavailable = errors.New("MySQL is unavailable") + +// TLSFiles names PEM files that secure MySQL connections. They are read again +// for every new connection, so rotated certificates take effect without a +// restart. +type TLSFiles struct { + // CAFile verifies the server certificate. Empty uses the system roots. + CAFile string + // CertFile and KeyFile hold the client certificate and key. Both may name + // the same file. Empty presents no client certificate. + CertFile string + KeyFile string +} + +func (f TLSFiles) enabled() bool { + return f.CAFile != "" || f.CertFile != "" || f.KeyFile != "" +} + +// ConnectConfig configures the MySQL pools used by Persistence. The database +// named in each DSN must already exist; atemy never creates one. +type ConnectConfig struct { + ReadWriteDSN string + // OwnerDSN applies migrations and needs DDL privileges on the database. + OwnerDSN string + TLS TLSFiles + PoolMaxConns int32 +} + +// Connect opens a read/write pool and applies migrations through an owner +// pool it closes before returning. +func Connect(ctx context.Context, config ConnectConfig) (*Persistence, error) { + if config.PoolMaxConns < 0 { + return nil, fmt.Errorf("MySQL pool maximum connections must not be negative") + } + if config.OwnerDSN == "" { + return nil, fmt.Errorf("MySQL owner connection string must not be empty") + } + readWriteConfig, err := parseConfig(config.ReadWriteDSN, config.TLS) + if err != nil { + return nil, err + } + ownerConfig, err := parseConfig(config.OwnerDSN, config.TLS) + if err != nil { + return nil, fmt.Errorf("parse MySQL owner connection string: %w", err) + } + if readWriteConfig.DBName != ownerConfig.DBName { + return nil, fmt.Errorf("MySQL read/write and owner connection strings name different databases") + } + readWrite, err := mysql.NewConnector(readWriteConfig) + if err != nil { + return nil, err + } + owner, err := mysql.NewConnector(ownerConfig) + if err != nil { + return nil, fmt.Errorf("parse MySQL owner connection string: %w", err) + } + + maxConns := storesql.DefaultMaxConns() + if config.PoolMaxConns > 0 { + maxConns = int(config.PoolMaxConns) + } + db := sql.OpenDB(readWrite) + db.SetMaxOpenConns(maxConns) + db.SetMaxIdleConns(maxConns) + db.SetConnMaxLifetime(storesql.ConnMaxLifetime) + db.SetConnMaxIdleTime(storesql.ConnMaxIdleTime) + if err := db.PingContext(ctx); err != nil { + db.Close() + return nil, fmt.Errorf("%w: pinging MySQL: %w", ErrUnavailable, err) + } + ownerDB := sql.OpenDB(owner) + ownerDB.SetMaxOpenConns(ownerPoolMaxConns) + if err := ownerDB.PingContext(ctx); err != nil { + ownerDB.Close() + db.Close() + return nil, fmt.Errorf("%w: ping MySQL owner connection: %w", ErrUnavailable, err) + } + watchDB := sql.OpenDB(readWrite) + watchDB.SetMaxOpenConns(watchPoolMaxConns) + watchDB.SetMaxIdleConns(watchPoolMaxConns) + watchDB.SetConnMaxLifetime(storesql.ConnMaxLifetime) + watchDB.SetConnMaxIdleTime(storesql.ConnMaxIdleTime) + + p, err := newPersistence(ctx, db, watchDB, ownerDB) + ownerDB.Close() + if err != nil { + watchDB.Close() + db.Close() + return nil, err + } + p.ownsWatchDB = true + return p, nil +} + +// parseConfig parses dsn and applies the session settings atemy relies on. +func parseConfig(dsn string, files TLSFiles) (*mysql.Config, error) { + cfg, err := mysql.ParseDSN(dsn) + if err != nil { + return nil, fmt.Errorf("parsing MySQL connection string: invalid value") + } + if cfg.DBName == "" { + return nil, fmt.Errorf("MySQL connection string must name a database") + } + // OpenFGA scans TIMESTAMP columns into time.Time. + cfg.ParseTime = true + cfg.Loc = time.UTC + // RowsAffected reports matched rows, as PostgreSQL does, so a guarded + // UPDATE that rewrites identical bytes still counts as applied. + cfg.ClientFoundRows = true + // Interpolating parameters client-side saves the prepare and close round + // trips the driver would otherwise spend on every parameterized statement. + cfg.InterpolateParams = true + cfg.MultiStatements = false + // Every transaction runs at READ COMMITTED, the isolation PostgreSQL gives + // atepg. Setting it once per session spares each transaction the SET + // TRANSACTION round trip, which Vitess applies to the session anyway. + if cfg.Params == nil { + cfg.Params = map[string]string{} + } + cfg.Params["transaction_isolation"] = "'READ-COMMITTED'" + if !files.enabled() { + return cfg, nil + } + if _, err := loadTLSConfig(files, ""); err != nil { + return nil, err + } + // The driver passes BeforeConnect a fresh copy of cfg for each new + // connection, so rotated files are read again. + err = cfg.Apply(mysql.BeforeConnect(func(_ context.Context, c *mysql.Config) error { + host, _, err := net.SplitHostPort(c.Addr) + if err != nil { + host = c.Addr + } + tlsConfig, err := loadTLSConfig(files, host) + if err != nil { + return err + } + c.TLS = tlsConfig + // TLS files demand TLS; tls=preferred in the DSN must not turn that + // into an optional upgrade. + c.AllowFallbackToPlaintext = false + return nil + })) + return cfg, err +} + +func loadTLSConfig(files TLSFiles, serverName string) (*tls.Config, error) { + cfg := &tls.Config{MinVersion: tls.VersionTLS12, ServerName: serverName} + if files.CAFile != "" { + pem, err := os.ReadFile(files.CAFile) + if err != nil { + return nil, fmt.Errorf("reading MySQL CA file: %w", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(pem) { + return nil, fmt.Errorf("MySQL CA file holds no PEM certificates") + } + cfg.RootCAs = pool + } + if files.CertFile != "" || files.KeyFile != "" { + if files.CertFile == "" || files.KeyFile == "" { + return nil, fmt.Errorf("MySQL client certificate and key files must be set together") + } + cert, err := tls.LoadX509KeyPair(files.CertFile, files.KeyFile) + if err != nil { + return nil, fmt.Errorf("loading MySQL client certificate: %w", err) + } + cfg.Certificates = []tls.Certificate{cert} + } + return cfg, nil +} + +// Open returns a pool on dsn with the session settings atemy relies on, +// without connecting. Pass the result to NewPersistence. +func Open(dsn string) (*sql.DB, error) { + cfg, err := parseConfig(dsn, TLSFiles{}) + if err != nil { + return nil, err + } + c, err := mysql.NewConnector(cfg) + if err != nil { + return nil, err + } + return sql.OpenDB(c), nil +} + +// NewPersistence wraps a database opened with Open, applying pending +// migrations. Watch and owner traffic share it. +func NewPersistence(ctx context.Context, db *sql.DB) (*Persistence, error) { + return newPersistence(ctx, db, db, db) +} + +func newPersistence(ctx context.Context, db, watchDB, ownerDB *sql.DB) (*Persistence, error) { + if err := requireMySQL8(ctx, db); err != nil { + return nil, err + } + // The read/write pool's sessions run every write, and its DSN can set + // session variables, so the session checks run there. + if err := requireAutoIncrementStep(ctx, db); err != nil { + return nil, err + } + if err := requireStrictMode(ctx, db); err != nil { + return nil, err + } + if err := applyMigrations(ctx, ownerDB); err != nil { + return nil, err + } + maintenanceCtx, stopMaintenance := context.WithCancel(context.Background()) + p := &Persistence{ + db: db, + watchDB: watchDB, + leaseTTL: storesql.DefaultLeaseTTL, + pollFailureCloseAfter: outboxPollFailureCloseAfter, + stopMaintenance: stopMaintenance, + maintenanceDone: make(chan struct{}), + } + go func() { + defer close(p.maintenanceDone) + p.maintenance(maintenanceCtx) + }() + return p, nil +} + +// Close stops the maintenance loop and waits for it to exit, then closes the +// watch pool if Connect created it. It does not close the main database, +// which the caller owns. +func (p *Persistence) Close() { + p.stopMaintenance() + <-p.maintenanceDone + if p.ownsWatchDB { + p.watchDB.Close() + } +} + +func (p *Persistence) DB() *sql.DB { + return p.db +} + +// SetPolicyManager configures the authorization policy manager that writes +// OpenFGA tuples in the same transaction as access policy and atespace +// mutations. It must be set before the store serves access policy or +// atespace writes. +func (p *Persistence) SetPolicyManager(pm *authz.PolicyManager) { + p.policyManager = pm +} + +// querier is satisfied by *sql.DB and *sql.Tx. +type querier interface { + QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row + QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error) + ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error) +} + +// inList returns a parenthesized list of n placeholders for an IN clause. +func inList(n int) string { return "(?" + strings.Repeat(", ?", n-1) + ")" } + +// txRetries bounds how many times inTx reruns a transaction InnoDB chose as a +// deadlock victim. +const txRetries = 5 + +// txBackoff spaces deadlock retries so the transaction that won has time to +// commit. +func txBackoff() wait.Backoff { + return wait.Backoff{Steps: txRetries, Duration: 10 * time.Millisecond, Factor: 2, Jitter: 1} +} + +// inTx runs fn in a transaction on db, READ COMMITTED through the session +// setting parseConfig applies, and commits if fn succeeds. InnoDB resolves a +// deadlock by rolling back one transaction, which can happen even between two +// inserts of one key, so inTx runs fn again from the start, up to txRetries +// times after a backoff. fn must therefore be safe to repeat, as the store's +// update contract already requires of mutate. +func inTx(ctx context.Context, db *sql.DB, fn func(tx *sql.Tx) error) error { + backoff := txBackoff() + for { + err := runTx(ctx, db, fn) + if mysqlErrNumber(err) != errDeadlock { + return err + } + if backoff.Steps == 0 { + // The write lost to a concurrent one and the caller may retry. + return fmt.Errorf("%w: %w", store.ErrVersionConflict, err) + } + retry := time.NewTimer(backoff.Step()) + select { + case <-ctx.Done(): + retry.Stop() + return fmt.Errorf("retrying a deadlocked transaction: %w", ctx.Err()) + case <-retry.C: + } + } +} + +func runTx(ctx context.Context, db *sql.DB, fn func(tx *sql.Tx) error) error { + tx, err := db.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("beginning transaction: %w", err) + } + defer tx.Rollback() //nolint:errcheck // no-op once committed + if err := fn(tx); err != nil { + return err + } + if err := tx.Commit(); err != nil { + return fmt.Errorf("committing transaction: %w", err) + } + return nil +} + +// MySQL error numbers atemy acts on. +const ( + errDuplicateKey = 1062 + errDeadlock = 1213 + errLockNowait = 3572 +) + +func isUniqueViolation(err error) bool { return mysqlErrNumber(err) == errDuplicateKey } + +func mysqlErrNumber(err error) uint16 { + var myErr *mysql.MySQLError + if errors.As(err, &myErr) { + return myErr.Number + } + return 0 +} + +// requireOneRow reports a statement that should have touched exactly one row. +func requireOneRow(res sql.Result, what string) error { + n, err := res.RowsAffected() + if err != nil { + return fmt.Errorf("%s: %w", what, err) + } + if n != 1 { + return fmt.Errorf("%s affected %d rows, want 1", what, n) + } + return nil +} + +const ( + // Paces the maintenance loop (outbox retention and expired leases). + maintenanceInterval = time.Minute + + // Bounds a maintenance pass so a stall, such as a lock wait, aborts and + // retries next tick. + maintenancePassTimeout = 5 * time.Minute +) + +// maintenance trims the worker outbox and reaps expired leases each tick. A +// failure in one does not skip the other. +func (p *Persistence) maintenance(ctx context.Context) { + ticker := time.NewTicker(maintenanceInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + passCtx, cancel := context.WithTimeout(ctx, maintenancePassTimeout) + if err := p.trimWorkerOutboxOlderThan(passCtx, outboxRetentionAge); err != nil && ctx.Err() == nil { + slog.WarnContext(ctx, "worker outbox maintenance failed", slog.Any("err", err)) + } + if deleted, err := p.cleanupExpiredLeases(passCtx); err != nil && ctx.Err() == nil { + slog.WarnContext(ctx, "expired lease cleanup failed", slog.Int64("deleted", deleted), slog.Any("err", err)) + } else if deleted > 0 { + slog.InfoContext(ctx, "removed expired MySQL leases", slog.Int64("deleted", deleted)) + } + cancel() + } +} diff --git a/cmd/ateapi/internal/store/atemy/atemy_test.go b/cmd/ateapi/internal/store/atemy/atemy_test.go new file mode 100644 index 0000000000..fb5a21a3e5 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/atemy_test.go @@ -0,0 +1,251 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + + "github.com/go-sql-driver/mysql" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// A listing fails as a whole on one undecodable row, so the error has to name +// the row. +func TestList_NamesAnUndecodableRow(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + createTestAtespace(t, s, "team-a") + createTestActorTemplate(t, s, "team-a", "t1") + createTestTag(t, s, "team-a", "v1") + if _, err := s.CreateActor(ctx, newTestActor("team-a", "a1")); err != nil { + t.Fatalf("CreateActor failed: %v", err) + } + if _, err := s.CreateWorker(ctx, newTestWorker("w1")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + + // Not a valid encoding of any message. + bad := []byte{0xff} + for _, table := range []string{"atespaces", "actors", "actor_templates", "tags", "workers"} { + if _, err := s.db.ExecContext(ctx, "UPDATE "+table+" SET proto = ?", bad); err != nil { + t.Fatalf("corrupting %s: %v", table, err) + } + } + if _, err := s.db.ExecContext(ctx, `INSERT INTO worker_assignments (actor_uid, worker_name, proto) VALUES ('uid-1', 'w1', ?)`, bad); err != nil { + t.Fatalf("inserting assignment: %v", err) + } + + opts := store.ListOptions{PageSize: 10} + tests := []struct { + name string + list func() error + want string + }{ + {"atespaces", func() error { _, err := s.ListAtespaces(ctx, opts); return err }, "atespace team-a"}, + {"actors", func() error { _, err := s.ListActors(ctx, "team-a", opts); return err }, "actor team-a/a1"}, + {"actors, all atespaces", func() error { _, err := s.ListActors(ctx, "", opts); return err }, "actor team-a/a1"}, + {"actor templates", func() error { _, err := s.ListActorTemplates(ctx, "team-a", opts); return err }, "actor template team-a/t1"}, + {"actor templates, all atespaces", func() error { _, err := s.ListActorTemplates(ctx, "", opts); return err }, "actor template team-a/t1"}, + {"tags", func() error { _, err := s.ListTags(ctx, "team-a", opts); return err }, "tag team-a/v1"}, + {"tags, all atespaces", func() error { _, err := s.ListTags(ctx, "", opts); return err }, "tag team-a/v1"}, + {"workers", func() error { _, err := s.ListWorkers(ctx, opts); return err }, "worker w1"}, + {"worker assignments", func() error { _, err := s.ListWorkerAssignments(ctx, "w1", opts); return err }, "actor uid-1"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + err := tc.list() + if err == nil || !strings.Contains(err.Error(), tc.want) { + t.Errorf("list error = %v, want one naming %q", err, tc.want) + } + }) + } +} + +// Key columns use utf8mb4_0900_bin, so names that a case-insensitive or +// PAD SPACE collation would fold together stay distinct, as they do in +// PostgreSQL, and listings and their page tokens follow byte order. +func TestKeys_CompareByteForByte(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + names := []string{"team", "Team", "TEAM", "team ", "a", "B"} + for _, name := range names { + createTestAtespace(t, s, name) + } + for _, name := range names { + got, err := s.GetAtespace(ctx, name) + if err != nil { + t.Fatalf("GetAtespace(%q) failed: %v", name, err) + } + if got.GetMetadata().GetName() != name { + t.Errorf("GetAtespace(%q) returned %q", name, got.GetMetadata().GetName()) + } + } + want := slices.Sorted(slices.Values(names)) + listed := listByOne(t, func(opts store.ListOptions) (store.ListResponse[*ateapipb.Atespace], error) { + return s.ListAtespaces(ctx, opts) + }, func(a *ateapipb.Atespace) string { return a.GetMetadata().GetName() }) + if !slices.Equal(listed, want) { + t.Errorf("atespaces listed in order %q, want byte order %q", listed, want) + } + + // The unscoped listing pages over the composite (atespace, name) key. + actorNames := []string{"x", "X", "x "} + var wantActors []string + for _, atespace := range []string{"team", "Team"} { + for _, name := range actorNames { + if _, err := s.CreateActor(ctx, newTestActor(atespace, name)); err != nil { + t.Fatalf("CreateActor(%q/%q) failed: %v", atespace, name, err) + } + wantActors = append(wantActors, atespace+"/"+name) + } + } + slices.Sort(wantActors) + listedActors := listByOne(t, func(opts store.ListOptions) (store.ListResponse[*ateapipb.Actor], error) { + return s.ListActors(ctx, "", opts) + }, func(a *ateapipb.Actor) string { return a.GetMetadata().GetAtespace() + "/" + a.GetMetadata().GetName() }) + if !slices.Equal(listedActors, wantActors) { + t.Errorf("actors listed in order %q, want byte order %q", listedActors, wantActors) + } + if _, err := s.GetActor(ctx, resources.ActorRef{Atespace: "team", Name: "x "}); !errors.Is(err, store.ErrNotFound) { + t.Errorf("GetActor with an extra trailing space = %v, want ErrNotFound", err) + } +} + +// listByOne lists one item per page through the last page and returns the +// key of each item in order. +func listByOne[T any](t *testing.T, list func(store.ListOptions) (store.ListResponse[T], error), key func(T) string) []string { + t.Helper() + var keys []string + opts := store.ListOptions{PageSize: 1} + for { + page, err := list(opts) + if err != nil { + t.Fatalf("listing failed: %v", err) + } + for _, item := range page.Items { + keys = append(keys, key(item)) + } + if opts.PageToken = page.NextPageToken; opts.PageToken == "" { + return keys + } + } +} + +// selectForUpdate locks one atespace row for the rest of tx. +func selectForUpdate(ctx context.Context, tx *sql.Tx, name string) error { + var one int + return tx.QueryRowContext(ctx, `SELECT 1 FROM atespaces WHERE name = ? FOR UPDATE`, name).Scan(&one) +} + +// TestInTx_RetriesADeadlockVictim runs two transactions that lock two rows in +// opposite orders. InnoDB rolls one back, and inTx must run it again so both +// commit. +func TestInTx_RetriesADeadlockVictim(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + createTestAtespace(t, s, "a") + createTestAtespace(t, s, "b") + + aLocked := make(chan struct{}) + bLocked := make(chan struct{}) + var signalA, signalB sync.Once + var attempts atomic.Int32 + errs := make([]error, 2) + var wg sync.WaitGroup + wg.Go(func() { + errs[0] = inTx(ctx, s.db, func(tx *sql.Tx) error { + attempts.Add(1) + if err := selectForUpdate(ctx, tx, "a"); err != nil { + return err + } + signalA.Do(func() { close(aLocked) }) + <-bLocked + return selectForUpdate(ctx, tx, "b") + }) + }) + wg.Go(func() { + errs[1] = inTx(ctx, s.db, func(tx *sql.Tx) error { + attempts.Add(1) + <-aLocked + if err := selectForUpdate(ctx, tx, "b"); err != nil { + return err + } + signalB.Do(func() { close(bLocked) }) + return selectForUpdate(ctx, tx, "a") + }) + }) + wg.Wait() + + for i, err := range errs { + if err != nil { + t.Errorf("transaction %d = %v, want the deadlock victim retried to a commit", i, err) + } + } + if got := attempts.Load(); got != 3 { + t.Errorf("transactions ran %d attempts, want 3 (one victim retried once)", got) + } +} + +// Deadlock retries stop after txRetries or when ctx ends. +func TestInTx_StopsRetryingDeadlocks(t *testing.T) { + s := setupMySQLPersistence(t) + deadlock := &mysql.MySQLError{Number: errDeadlock, Message: "Deadlock found when trying to get lock"} + + attempts := 0 + err := inTx(t.Context(), s.db, func(*sql.Tx) error { + attempts++ + return deadlock + }) + if !errors.Is(err, store.ErrVersionConflict) || !errors.Is(err, deadlock) { + t.Errorf("inTx after repeated deadlocks = %v, want ErrVersionConflict wrapping the deadlock", err) + } + if attempts != txRetries+1 { + t.Errorf("inTx ran %d attempts, want %d", attempts, txRetries+1) + } + + ctx, cancel := context.WithCancel(t.Context()) + attempts = 0 + err = inTx(ctx, s.db, func(*sql.Tx) error { + attempts++ + cancel() + return deadlock + }) + if !errors.Is(err, context.Canceled) || attempts != 1 { + t.Errorf("inTx with ctx canceled after a deadlock = %v after %d attempts, want context.Canceled after 1", err, attempts) + } +} + +// A lock wait timeout is not a deadlock, so inTx returns it as is. +func TestInTx_ReturnsLockWaitTimeout(t *testing.T) { + lockWait := &mysql.MySQLError{Number: 1205, Message: "Lock wait timeout exceeded"} + attempts := 0 + err := inTx(t.Context(), requireDB(t), func(*sql.Tx) error { + attempts++ + return lockWait + }) + if attempts != 1 || !errors.Is(err, lockWait) || errors.Is(err, store.ErrVersionConflict) { + t.Errorf("inTx returning a lock wait timeout = %v after %d attempts, want it as is after 1", err, attempts) + } +} diff --git a/cmd/ateapi/internal/store/atemy/atespace.go b/cmd/ateapi/internal/store/atemy/atespace.go new file mode 100644 index 0000000000..1daea4de6d --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/atespace.go @@ -0,0 +1,137 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +func (p *Persistence) CreateAtespace(ctx context.Context, atespace *ateapipb.Atespace) (*ateapipb.Atespace, error) { + name := atespace.GetMetadata().GetName() + + dbAtespace := proto.Clone(atespace).(*ateapipb.Atespace) + storesql.SetCreateMetadata(dbAtespace.Metadata) + + protoBytes, err := proto.Marshal(dbAtespace) + if err != nil { + return nil, fmt.Errorf("marshaling atespace: %w", err) + } + + err = inTx(ctx, p.db, func(tx *sql.Tx) error { + _, err := tx.ExecContext(ctx, ` + INSERT INTO atespaces (name, uid, version, proto) + VALUES (?, ?, ?, ?)`, + name, dbAtespace.GetMetadata().GetUid(), dbAtespace.GetMetadata().GetVersion(), protoBytes) + if isUniqueViolation(err) { + return store.ErrAlreadyExists + } + if err != nil { + return fmt.Errorf("inserting atespace %q: %w", name, err) + } + return nil + }) + if err != nil { + return nil, err + } + return dbAtespace, nil +} + +func (p *Persistence) GetAtespace(ctx context.Context, name string) (*ateapipb.Atespace, error) { + var protoBytes []byte + err := p.db.QueryRowContext(ctx, `SELECT proto FROM atespaces WHERE name = ?`, name).Scan(&protoBytes) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting atespace %q: %w", name, err) + } + out := &ateapipb.Atespace{} + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { + return nil, fmt.Errorf("unmarshaling atespace: %w", err) + } + return out, nil +} + +func (p *Persistence) ListAtespaces(ctx context.Context, opts store.ListOptions) (store.ListResponse[*ateapipb.Atespace], error) { + opts, err := store.NormalizeListOptions(opts) + if err != nil { + return store.ListResponse[*ateapipb.Atespace]{}, err + } + token, err := storesql.DecodePageToken(opts.PageToken, storesql.KindAtespace, "", 1) + if err != nil { + return store.ListResponse[*ateapipb.Atespace]{}, err + } + query, args := `SELECT name, proto FROM atespaces ORDER BY name LIMIT ?`, []any{int64(opts.PageSize) + 1} + if len(token.Last) > 0 { + query, args = `SELECT name, proto FROM atespaces WHERE name > ? ORDER BY name LIMIT ?`, []any{token.Last[0], int64(opts.PageSize) + 1} + } + items, next, err := listPage(ctx, p.db, storesql.KindAtespace, "", opts.PageSize, func(b []byte, keys []string) (*ateapipb.Atespace, error) { + a := &ateapipb.Atespace{} + return a, storesql.UnmarshalRow(b, a, "atespace", keys...) + }, query, args...) + if err != nil { + return store.ListResponse[*ateapipb.Atespace]{}, err + } + return store.ListResponse[*ateapipb.Atespace]{Items: items, NextPageToken: next}, nil +} + +// DeleteAtespace refuses an atespace that still holds actors, actor templates +// or tags, and removes its access policy and authorization tuples with it. +func (p *Persistence) DeleteAtespace(ctx context.Context, name string, precondition store.DeletePreconditions) (*ateapipb.Atespace, error) { + var protoBytes []byte + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + var err error + _, _, protoBytes, err = deleteLockedRow(ctx, tx, "atespaces", "name = ?", precondition, name) + if err != nil { + return err + } + // The exclusive lock deleteLockedRow took waits out every child insert + // that locked this atespace, so these reads see all of its children. + var nonEmpty bool + if err := tx.QueryRowContext(ctx, ` + SELECT EXISTS (SELECT 1 FROM actors WHERE atespace = ?) + OR EXISTS (SELECT 1 FROM actor_templates WHERE atespace = ?) + OR EXISTS (SELECT 1 FROM tags WHERE atespace = ?)`, name, name, name).Scan(&nonEmpty); err != nil { + return fmt.Errorf("checking atespace %q for children: %w", name, err) + } + if nonEmpty { + return store.ErrFailedPrecondition + } + if _, err := tx.ExecContext(ctx, `DELETE FROM atespace_access_policies WHERE atespace_name = ?`, name); err != nil { + return fmt.Errorf("deleting access policy of atespace %q: %w", name, err) + } + if err := p.policyManager.DeleteAtespacePolicies(ctx, authz.SQLTx(tx), name); err != nil { + return fmt.Errorf("deleting authorization tuples for atespace %q: %w", name, err) + } + return nil + }) + if err != nil { + return nil, err + } + out := &ateapipb.Atespace{} + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { + return nil, fmt.Errorf("unmarshaling deleted atespace: %w", err) + } + return out, nil +} diff --git a/cmd/ateapi/internal/store/atemy/common_test.go b/cmd/ateapi/internal/store/atemy/common_test.go new file mode 100644 index 0000000000..0ac34c8966 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/common_test.go @@ -0,0 +1,199 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "testing" + "time" + + "github.com/go-sql-driver/mysql" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// clearAll empties every table so the next test starts from an empty store +// without paying for a fresh database. Nothing in production mass-deletes +// state, so the statements live here rather than on Persistence. +func clearAll(t *testing.T, p *Persistence) { + t.Helper() + for _, stmt := range []string{ + `DELETE FROM atespaces`, + `DELETE FROM global_access_policy`, + `DELETE FROM atespace_access_policies`, + `DELETE FROM actors`, + `DELETE FROM actor_egress_policies`, + `DELETE FROM actor_templates`, + `DELETE FROM tags`, + `DELETE FROM workers`, + `DELETE FROM worker_assignments`, + `DELETE FROM leases`, + // TRUNCATE also restarts seq at 1, so no seq skipped by an earlier + // test reads as a pending write to the next test's watchers. + `TRUNCATE TABLE worker_outbox`, + `UPDATE worker_outbox_trim SET seq = 0`, + `DELETE FROM tuple`, + `DELETE FROM changelog`, + } { + if _, err := p.db.ExecContext(context.Background(), stmt); err != nil { + t.Fatalf("clearing tables (%s): %v", stmt, err) + } + } +} + +func setupMySQLPersistence(t *testing.T) *Persistence { + t.Helper() + return setupPersistenceOn(t, requireDB(t)) +} + +// setupPersistenceOn is setupMySQLPersistence over db, which may be a +// vtgate rather than the shared MySQL container. +func setupPersistenceOn(t *testing.T, db *sql.DB) *Persistence { + t.Helper() + ctx := context.Background() + p, err := NewPersistence(ctx, db) + if err != nil { + t.Fatalf("NewPersistence failed: %v", err) + } + t.Cleanup(p.Close) + clearAll(t, p) + setTestPolicyManager(t, p) + return p +} + +// setTestPolicyManager gives p an OpenFGA-backed PolicyManager, as the server +// always does. +func setTestPolicyManager(t *testing.T, p *Persistence) { + t.Helper() + fgaServer, err := authz.NewOpenFGAServer(authz.MySQLBackend(p.db)) + if err != nil { + t.Fatalf("NewOpenFGAServer failed: %v", err) + } + t.Cleanup(fgaServer.Close) + _, policyManager, err := authz.New(t.Context(), authz.MySQLBackend(p.db), fgaServer, nil) + if err != nil { + t.Fatalf("authz.New failed: %v", err) + } + p.SetPolicyManager(policyManager) +} + +func newTestAtespace(name string) *ateapipb.Atespace { + return &ateapipb.Atespace{Metadata: &ateapipb.ResourceMetadata{Name: name}} +} + +func createTestAtespace(t *testing.T, s *Persistence, name string) { + t.Helper() + if _, err := s.CreateAtespace(context.Background(), newTestAtespace(name)); err != nil { + t.Fatalf("CreateAtespace(%q) failed: %v", name, err) + } +} + +// newReplica opens a second Persistence on the shared database, standing in +// for another ateapi replica. Its watchers see only polled outbox rows, never +// the commit-time publish of writes made through another Persistence. +func newReplica(t *testing.T, p *Persistence) *Persistence { + t.Helper() + replica, err := NewPersistence(t.Context(), p.db) + if err != nil { + t.Fatalf("NewPersistence for a replica failed: %v", err) + } + t.Cleanup(replica.Close) + return replica +} + +// adminDSN returns a root connection string for database on the shared +// container. The testcontainers module gives root the application password. +func adminDSN(t *testing.T, database string) string { + t.Helper() + requireDB(t) + cfg, err := mysql.ParseDSN(containerDSN) + if err != nil { + t.Fatalf("parsing container DSN: %v", err) + } + cfg.User = "root" + cfg.DBName = database + return cfg.FormatDSN() +} + +// openAdmin opens a root pool on database, for statements the application +// user may not run: creating databases and reading server-wide lock state. +func openAdmin(t *testing.T, database string) *sql.DB { + t.Helper() + db, err := Open(adminDSN(t, database)) + if err != nil { + t.Fatalf("opening admin pool: %v", err) + } + t.Cleanup(func() { db.Close() }) + return db +} + +// createTestDatabase creates an empty database, dropped when the test ends, +// and returns a root connection string for it. +func createTestDatabase(t *testing.T, name string) string { + t.Helper() + admin := openAdmin(t, "mysql") + if _, err := admin.ExecContext(t.Context(), "DROP DATABASE IF EXISTS `"+name+"`"); err != nil { + t.Fatalf("dropping database %s: %v", name, err) + } + if _, err := admin.ExecContext(t.Context(), "CREATE DATABASE `"+name+"`"); err != nil { + t.Fatalf("creating database %s: %v", name, err) + } + t.Cleanup(func() { + if _, err := admin.ExecContext(context.Background(), "DROP DATABASE IF EXISTS `"+name+"`"); err != nil { + t.Errorf("dropping database %s: %v", name, err) + } + }) + return adminDSN(t, name) +} + +// waitForLockWait blocks until a transaction is waiting on a row lock, so a +// test can order a commit after another transaction has blocked. It fails if +// done delivers first: the operation expected to block finished instead. +func waitForLockWait(t *testing.T, done <-chan error) { + t.Helper() + admin := openAdmin(t, "mysql") + deadline := time.Now().Add(10 * time.Second) + for { + select { + case err := <-done: + t.Fatalf("operation finished with %v, want it blocked on a row lock", err) + default: + } + var waiting int + // performance_schema is live; information_schema.INNODB_TRX is a + // cache that frequent polling never refreshes. + if err := admin.QueryRowContext(t.Context(), `SELECT COUNT(*) FROM performance_schema.data_lock_waits`).Scan(&waiting); err != nil { + t.Fatalf("reading lock waits: %v", err) + } + if waiting > 0 { + return + } + if time.Now().After(deadline) { + t.Fatal("no transaction ever waited on a row lock") + } + time.Sleep(10 * time.Millisecond) + } +} + +func countRows(t *testing.T, p *Persistence, query string, args ...any) int { + t.Helper() + var n int + if err := p.db.QueryRowContext(t.Context(), query, args...).Scan(&n); err != nil { + t.Fatalf("counting rows (%s): %v", query, err) + } + return n +} diff --git a/cmd/ateapi/internal/store/atemy/connect_test.go b/cmd/ateapi/internal/store/atemy/connect_test.go new file mode 100644 index 0000000000..32b094ba76 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/connect_test.go @@ -0,0 +1,367 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "errors" + "math/big" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/go-sql-driver/mysql" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// Connect builds a dedicated watch pool, and Close releases it but not the +// caller's read/write pool. +func TestConnect_DedicatedPools(t *testing.T) { + requireDB(t) + ctx := t.Context() + + p, err := Connect(ctx, ConnectConfig{ReadWriteDSN: containerDSN, OwnerDSN: containerDSN}) + if err != nil { + t.Fatalf("Connect failed: %v", err) + } + defer p.DB().Close() + closed := false + defer func() { + if !closed { + p.Close() + } + }() + + if p.watchDB == p.db || !p.ownsWatchDB { + t.Fatal("Connect must own a dedicated watch pool") + } + for _, tc := range []struct { + name string + got int + want int + }{ + {"read/write", p.db.Stats().MaxOpenConnections, storesql.DefaultMaxConns()}, + {"watch", p.watchDB.Stats().MaxOpenConnections, watchPoolMaxConns}, + } { + if tc.got != tc.want { + t.Errorf("%s pool max connections = %d, want %d", tc.name, tc.got, tc.want) + } + } + + clearAll(t, p) + watch, err := p.WatchWorkers(ctx) + if err != nil { + t.Fatalf("WatchWorkers failed: %v", err) + } + defer watch.Close() + if _, err := p.CreateWorker(ctx, newTestWorker("watchpool-worker")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if event := receive(t, watch); event.Type != store.WorkerEventCreated || event.Worker.GetMetadata().GetName() != "watchpool-worker" { + t.Fatalf("unexpected event %v %s", event.Type, event.Worker.GetMetadata().GetName()) + } + watch.Close() + + p.Close() + closed = true + if err := p.watchDB.PingContext(ctx); err == nil { + t.Error("watch pool still open after Close") + } + if err := p.DB().PingContext(ctx); err != nil { + t.Errorf("Close closed the caller's read/write pool: %v", err) + } +} + +// TestConnect_SeparatesRuntimeAndDDLPrivileges runs Connect with a +// read/write user that holds only DML privileges and an owner user that +// holds DDL privileges, as a production install does. +func TestConnect_SeparatesRuntimeAndDDLPrivileges(t *testing.T) { + const database = "atemy_privileges" + createTestDatabase(t, database) + admin := openAdmin(t, "mysql") + ctx := t.Context() + for _, stmt := range []string{ + `DROP USER IF EXISTS 'atemy_rw'@'%', 'atemy_ddl'@'%'`, + `CREATE USER 'atemy_rw'@'%' IDENTIFIED BY 'rw-password'`, + `CREATE USER 'atemy_ddl'@'%' IDENTIFIED BY 'ddl-password'`, + "GRANT SELECT, INSERT, UPDATE, DELETE ON `" + database + "`.* TO 'atemy_rw'@'%'", + "GRANT ALL ON `" + database + "`.* TO 'atemy_ddl'@'%'", + } { + if _, err := admin.ExecContext(ctx, stmt); err != nil { + t.Fatalf("preparing users (%s): %v", stmt, err) + } + } + t.Cleanup(func() { + if _, err := admin.ExecContext(context.Background(), `DROP USER IF EXISTS 'atemy_rw'@'%', 'atemy_ddl'@'%'`); err != nil { + t.Errorf("dropping users: %v", err) + } + }) + userDSN := func(user, password string) string { + cfg, err := mysql.ParseDSN(containerDSN) + if err != nil { + t.Fatalf("parsing container DSN: %v", err) + } + cfg.User, cfg.Passwd, cfg.DBName = user, password, database + return cfg.FormatDSN() + } + + p, err := Connect(ctx, ConnectConfig{ + ReadWriteDSN: userDSN("atemy_rw", "rw-password"), + OwnerDSN: userDSN("atemy_ddl", "ddl-password"), + PoolMaxConns: 20, + }) + if err != nil { + t.Fatalf("Connect failed: %v", err) + } + defer p.DB().Close() + defer p.Close() + if got := p.db.Stats().MaxOpenConnections; got != 20 { + t.Errorf("read/write pool max connections = %d, want the configured 20", got) + } + setTestPolicyManager(t, p) + + if _, err := p.CreateAtespace(ctx, newTestAtespace("runtime-write")); err != nil { + t.Fatalf("read/write operation failed: %v", err) + } + if _, err := p.CreateAtespaceAccessPolicy(ctx, "runtime-write", &ateapipb.AccessPolicy{ + Bindings: []*ateapipb.Binding{{Role: authz.RoleEditor, Members: []string{"user:bob"}}}, + }); err != nil { + t.Fatalf("access policy write through OpenFGA tables failed: %v", err) + } + if _, err := p.CreateWorker(ctx, newTestWorker("runtime-worker")); err != nil { + t.Fatalf("outbox write failed: %v", err) + } + if err := p.trimWorkerOutboxOlderThan(ctx, 0); err != nil { + t.Fatalf("outbox retention failed: %v", err) + } + lease, err := p.AcquireLease(ctx, "runtime-lease") + if err != nil { + t.Fatalf("AcquireLease failed: %v", err) + } + lease.Close() + if _, err := p.db.ExecContext(ctx, `CREATE TABLE forbidden (id INT)`); err == nil { + t.Fatal("read/write user created a table") + } +} + +func TestConnect_RejectsInvalidConfig(t *testing.T) { + dir := t.TempDir() + certPath := filepath.Join(dir, "bundle.pem") + caPath := filepath.Join(dir, "ca.pem") + writeCredentialBundle(t, certPath, caPath, 1) + notPEM := filepath.Join(dir, "not-pem.txt") + if err := os.WriteFile(notPEM, []byte("not a certificate"), 0o600); err != nil { + t.Fatalf("writing file: %v", err) + } + const dsn = "atemy:hunter2@tcp(127.0.0.1:1)/atemy" + + for _, tc := range []struct { + name string + config ConnectConfig + want string + }{ + {"negative pool size", ConnectConfig{ReadWriteDSN: dsn, OwnerDSN: dsn, PoolMaxConns: -1}, "must not be negative"}, + {"no owner DSN", ConnectConfig{ReadWriteDSN: dsn}, "owner connection string must not be empty"}, + {"unparsable DSN", ConnectConfig{ReadWriteDSN: "atemy:hunter2@tcp(127.0.0.1:1)/atemy?timeout=forever", OwnerDSN: dsn}, "invalid value"}, + {"read/write DSN without a database", ConnectConfig{ReadWriteDSN: "atemy:hunter2@tcp(127.0.0.1:1)/", OwnerDSN: dsn}, "must name a database"}, + {"owner DSN names another database", ConnectConfig{ReadWriteDSN: dsn, OwnerDSN: "atemy:hunter2@tcp(127.0.0.1:1)/other"}, "name different databases"}, + {"CA file without certificates", ConnectConfig{ReadWriteDSN: dsn, OwnerDSN: dsn, TLS: TLSFiles{CAFile: notPEM}}, "holds no PEM certificates"}, + {"certificate without key", ConnectConfig{ReadWriteDSN: dsn, OwnerDSN: dsn, TLS: TLSFiles{CertFile: certPath}}, "must be set together"}, + {"unloadable key pair", ConnectConfig{ReadWriteDSN: dsn, OwnerDSN: dsn, TLS: TLSFiles{CertFile: notPEM, KeyFile: notPEM}}, "loading MySQL client certificate"}, + } { + t.Run(tc.name, func(t *testing.T) { + p, err := Connect(t.Context(), tc.config) + if err == nil { + p.Close() + t.Fatalf("Connect succeeded, want an error containing %q", tc.want) + } + if !strings.Contains(err.Error(), tc.want) { + t.Errorf("Connect error = %v, want one containing %q", err, tc.want) + } + if strings.Contains(err.Error(), "hunter2") { + t.Errorf("Connect error %q exposes the password", err) + } + if errors.Is(err, ErrUnavailable) { + t.Errorf("Connect error %v is retryable, want a configuration error", err) + } + }) + } +} + +// TestConnect_UsesTLSFiles shows the files secure real connections: the +// container's auto-generated certificate is not signed by the test CA, so +// the handshake fails. +func TestConnect_UsesTLSFiles(t *testing.T) { + requireDB(t) + dir := t.TempDir() + certPath := filepath.Join(dir, "bundle.pem") + caPath := filepath.Join(dir, "ca.pem") + writeCredentialBundle(t, certPath, caPath, 1) + + p, err := Connect(t.Context(), ConnectConfig{ReadWriteDSN: containerDSN, OwnerDSN: containerDSN, TLS: TLSFiles{CAFile: caPath}}) + if err == nil { + p.DB().Close() + p.Close() + t.Fatal("Connect trusted a server certificate the CA file did not sign") + } + if !errors.Is(err, ErrUnavailable) || !strings.Contains(err.Error(), "certificate") { + t.Errorf("Connect error = %v, want ErrUnavailable from certificate verification", err) + } +} + +func TestParseConfig_OverridesDSNSessionSettings(t *testing.T) { + cfg, err := parseConfig("atemy:pw@tcp(db.example:3306)/atemy?multiStatements=true&parseTime=false", TLSFiles{}) + if err != nil { + t.Fatalf("parseConfig failed: %v", err) + } + if !cfg.ParseTime || cfg.MultiStatements { + t.Errorf("parseTime = %t, multiStatements = %t; want the DSN's false and true overridden", cfg.ParseTime, cfg.MultiStatements) + } +} + +// TestOpen_SessionsRunReadCommitted checks the isolation every transaction +// inherits from the session, the level atepg runs at. +func TestOpen_SessionsRunReadCommitted(t *testing.T) { + db := requireDB(t) + var isolation string + if err := db.QueryRowContext(t.Context(), `SELECT @@SESSION.transaction_isolation`).Scan(&isolation); err != nil { + t.Fatalf("reading session isolation: %v", err) + } + if isolation != "READ-COMMITTED" { + t.Errorf("session isolation = %q, want READ-COMMITTED", isolation) + } +} + +// loadTLSConfig returns the files on disk at the time of the call, and the +// connector calls it for each new connection. +func TestLoadTLSConfig_ReadsCurrentFiles(t *testing.T) { + dir := t.TempDir() + bundlePath := filepath.Join(dir, "credential-bundle.pem") + rootPath := filepath.Join(dir, "trust-bundle.pem") + writeCredentialBundle(t, bundlePath, rootPath, 1) + files := TLSFiles{CAFile: rootPath, CertFile: bundlePath, KeyFile: bundlePath} + + mycfg, err := parseConfig("atemy:pw@tcp(mysql.ate-system.svc:3306)/atemy", files) + if err != nil { + t.Fatalf("parseConfig failed: %v", err) + } + c, err := mysql.NewConnector(mycfg) + if err != nil { + t.Fatalf("NewConnector failed: %v", err) + } + cfg, err := loadTLSConfig(files, "mysql.ate-system.svc") + if err != nil { + t.Fatalf("loadTLSConfig: %v", err) + } + if got := clientCertSerial(t, cfg.Certificates); got != 1 { + t.Fatalf("client certificate serial = %d, want 1", got) + } + if cfg.ServerName != "mysql.ate-system.svc" || cfg.MinVersion != tls.VersionTLS12 { + t.Errorf("TLS config server name %q, minimum version %x; want mysql.ate-system.svc and TLS 1.2", cfg.ServerName, cfg.MinVersion) + } + + writeCredentialBundle(t, bundlePath, rootPath, 2) + cfg, err = loadTLSConfig(files, "mysql.ate-system.svc") + if err != nil { + t.Fatalf("loadTLSConfig after rotation: %v", err) + } + if got := clientCertSerial(t, cfg.Certificates); got != 2 { + t.Errorf("client certificate serial after rotation = %d, want 2", got) + } + if !cfg.RootCAs.Equal(rootPool(t, rootPath)) { + t.Error("root CAs after rotation are not the ones on disk") + } + + // A missing file fails the connection before any network traffic. + if err := os.Remove(rootPath); err != nil { + t.Fatalf("removing CA file: %v", err) + } + if _, err := c.Connect(t.Context()); err == nil || !strings.Contains(err.Error(), "reading MySQL CA file") { + t.Errorf("connector.Connect without its CA file = %v, want a CA file error", err) + } +} + +// writeCredentialBundle writes a self-signed certificate with the given serial +// to certPath in the layout of a Kubernetes credential bundle (private key +// first, then the chain) and its certificate alone to rootPath. +func writeCredentialBundle(t *testing.T, certPath, rootPath string, serial int64) { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatalf("generating key: %v", err) + } + template := &x509.Certificate{ + SerialNumber: big.NewInt(serial), + Subject: pkix.Name{CommonName: "mysql.ate-system.svc"}, + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + BasicConstraintsValid: true, + IsCA: true, + } + der, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key) + if err != nil { + t.Fatalf("signing certificate: %v", err) + } + keyDER, err := x509.MarshalPKCS8PrivateKey(key) + if err != nil { + t.Fatalf("marshaling key: %v", err) + } + certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}) + keyPEM := pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: keyDER}) + if err := os.WriteFile(certPath, append(keyPEM, certPEM...), 0o600); err != nil { + t.Fatalf("writing credential bundle: %v", err) + } + if err := os.WriteFile(rootPath, certPEM, 0o644); err != nil { + t.Fatalf("writing trust bundle: %v", err) + } +} + +func clientCertSerial(t *testing.T, certs []tls.Certificate) int64 { + t.Helper() + if len(certs) != 1 { + t.Fatalf("got %d client certificates, want 1", len(certs)) + } + leaf, err := x509.ParseCertificate(certs[0].Certificate[0]) + if err != nil { + t.Fatalf("parsing client certificate: %v", err) + } + return leaf.SerialNumber.Int64() +} + +func rootPool(t *testing.T, path string) *x509.CertPool { + t.Helper() + pemBytes, err := os.ReadFile(path) + if err != nil { + t.Fatalf("reading trust bundle: %v", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(pemBytes) { + t.Fatal("trust bundle holds no certificates") + } + return pool +} diff --git a/cmd/ateapi/internal/store/atemy/contract_test.go b/cmd/ateapi/internal/store/atemy/contract_test.go new file mode 100644 index 0000000000..023a8fedb3 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/contract_test.go @@ -0,0 +1,31 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "testing" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storecontract" +) + +// TestContractSuite runs the backend-neutral store.Interface assertions +// against a real MySQL instance. +func TestContractSuite(t *testing.T) { + storecontract.RunContractTests(t, func(t *testing.T) store.Interface { + t.Helper() + return setupMySQLPersistence(t) + }) +} diff --git a/cmd/ateapi/internal/store/atemy/egress_policy.go b/cmd/ateapi/internal/store/atemy/egress_policy.go new file mode 100644 index 0000000000..08e09c2154 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/egress_policy.go @@ -0,0 +1,126 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +// CreateEgressPolicy returns ErrFailedPrecondition if the actor does not exist. +func (p *Persistence) CreateEgressPolicy(ctx context.Context, actorRef resources.ActorRef, policy *ateapipb.EgressPolicy) (*ateapipb.EgressPolicy, error) { + dbPolicy := proto.Clone(policy).(*ateapipb.EgressPolicy) + storesql.SetCreateMetadata(dbPolicy.Metadata) + protoBytes, err := proto.Marshal(dbPolicy) + if err != nil { + return nil, fmt.Errorf("marshaling egress policy: %w", err) + } + err = p.insertChild(ctx, "egress policy for "+actorRef.String(), func(tx *sql.Tx) error { + return lockParent(ctx, tx, `SELECT 1 FROM actors WHERE atespace = ? AND name = ? FOR SHARE`, actorRef.Atespace, actorRef.Name) + }, ` + INSERT INTO actor_egress_policies (atespace, actor_name, uid, version, proto) + VALUES (?, ?, ?, ?, ?)`, actorRef.Atespace, actorRef.Name, dbPolicy.GetMetadata().GetUid(), dbPolicy.GetMetadata().GetVersion(), protoBytes) + if err != nil { + return nil, err + } + return dbPolicy, nil +} + +func (p *Persistence) GetEgressPolicy(ctx context.Context, actorRef resources.ActorRef) (*ateapipb.EgressPolicy, error) { + return getEgressPolicyRow(ctx, p.db, actorRef) +} + +func (p *Persistence) UpdateEgressPolicy(ctx context.Context, actorRef resources.ActorRef, precondition store.Precondition, mutate func(*ateapipb.EgressPolicy) error) (*ateapipb.EgressPolicy, error) { + if err := precondition.Validate(); err != nil { + return nil, err + } + dbPolicy, err := getEgressPolicyRow(ctx, p.db, actorRef) + if err != nil { + return nil, err + } + currentUID := dbPolicy.GetMetadata().GetUid() + currentVersion := dbPolicy.GetMetadata().GetVersion() + if err := precondition.Check(dbPolicy.GetMetadata()); err != nil { + return nil, err + } + oldMeta := proto.CloneOf(dbPolicy.Metadata) + if err := mutate(dbPolicy); err != nil { + return nil, err + } + dbPolicy.Metadata = oldMeta + storesql.SetUpdateMetadata(dbPolicy.Metadata, oldMeta) + protoBytes, err := proto.Marshal(dbPolicy) + if err != nil { + return nil, fmt.Errorf("marshaling updated egress policy: %w", err) + } + if err := updateGuarded(ctx, p.db, "egress policy for "+actorRef.String(), ` + UPDATE actor_egress_policies SET version = ?, proto = ? + WHERE atespace = ? AND actor_name = ? AND uid = ? AND version = ?`, + dbPolicy.GetMetadata().GetVersion(), protoBytes, actorRef.Atespace, actorRef.Name, currentUID, currentVersion); err != nil { + return nil, err + } + return dbPolicy, nil +} + +func (p *Persistence) DeleteEgressPolicy(ctx context.Context, actorRef resources.ActorRef, precondition store.DeletePreconditions) (*ateapipb.EgressPolicy, error) { + var ( + uid string + version int64 + protoBytes []byte + ) + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + var err error + uid, version, protoBytes, err = deleteLockedRow(ctx, tx, "actor_egress_policies", "atespace = ? AND actor_name = ?", precondition, actorRef.Atespace, actorRef.Name) + return err + }) + if err != nil { + return nil, err + } + return unmarshalEgressPolicy(uid, version, protoBytes) +} + +func getEgressPolicyRow(ctx context.Context, q querier, actorRef resources.ActorRef) (*ateapipb.EgressPolicy, error) { + var uid string + var version int64 + var protoBytes []byte + if err := q.QueryRowContext(ctx, ` + SELECT uid, version, proto FROM actor_egress_policies + WHERE atespace = ? AND actor_name = ?`, actorRef.Atespace, actorRef.Name).Scan(&uid, &version, &protoBytes); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting egress policy: %w", err) + } + return unmarshalEgressPolicy(uid, version, protoBytes) +} + +func unmarshalEgressPolicy(uid string, version int64, protoBytes []byte) (*ateapipb.EgressPolicy, error) { + policy := &ateapipb.EgressPolicy{} + if err := storesql.UnmarshalStored(protoBytes, policy); err != nil { + return nil, fmt.Errorf("unmarshaling egress policy: %w", err) + } + if err := storesql.ValidateProtoMetadataMatchesColumns("egress policy", policy.GetMetadata(), uid, version); err != nil { + return nil, err + } + return policy, nil +} diff --git a/cmd/ateapi/internal/store/atemy/lease.go b/cmd/ateapi/internal/store/atemy/lease.go new file mode 100644 index 0000000000..76e9065036 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/lease.go @@ -0,0 +1,146 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "fmt" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" +) + +// Lease expiry is decided by the database clock. UTC_TIMESTAMP is fixed at +// the start of each statement, which for these single-statement writes is the +// moment PostgreSQL's clock_timestamp() would report. + +func (p *Persistence) AcquireLease(ctx context.Context, key string) (*store.Lease, error) { + return storesql.LeaseSQL{ + Database: "MySQL", + Acquire: p.acquireLease, + Renew: p.renewLease, + Release: p.releaseLease, + }.AcquireLease(ctx, key, p.leaseTTL) +} + +// acquireLease takes over an expired row, or inserts one if the key has none. +// It takes two statements because with CLIENT_FOUND_ROWS an unchanged +// duplicate in INSERT ... ON DUPLICATE KEY UPDATE counts as affected, like an +// insert. A racing acquirer loses on the row lock or the duplicate key, or on +// a deadlock among inserters after cleanup deleted the row. +func (p *Persistence) acquireLease(ctx context.Context, key, token string, ttl time.Duration) (bool, error) { + res, err := p.db.ExecContext(ctx, ` + UPDATE leases + SET token = ?, expires_at = UTC_TIMESTAMP(6) + INTERVAL ? MICROSECOND + WHERE lease_key = ? AND expires_at <= UTC_TIMESTAMP(6)`, token, ttl.Microseconds(), key) + if err != nil { + return false, fmt.Errorf("acquiring lease for %q: %w", key, err) + } + if n, err := res.RowsAffected(); err != nil { + return false, fmt.Errorf("acquiring lease for %q: %w", key, err) + } else if n == 1 { + return true, nil + } + if _, err := p.db.ExecContext(ctx, ` + INSERT INTO leases (lease_key, token, expires_at) + VALUES (?, ?, UTC_TIMESTAMP(6) + INTERVAL ? MICROSECOND)`, key, token, ttl.Microseconds()); err != nil { + if isUniqueViolation(err) || mysqlErrNumber(err) == errDeadlock { + return false, nil + } + return false, fmt.Errorf("acquiring lease for %q: %w", key, err) + } + return true, nil +} + +func (p *Persistence) renewLease(ctx context.Context, key, token string, ttl time.Duration) (bool, error) { + res, err := p.db.ExecContext(ctx, ` + UPDATE leases + SET expires_at = UTC_TIMESTAMP(6) + INTERVAL ? MICROSECOND + WHERE lease_key = ? AND token = ? AND expires_at > UTC_TIMESTAMP(6)`, ttl.Microseconds(), key, token) + if err != nil { + return false, fmt.Errorf("renewing lease for %q: %w", key, err) + } + n, err := res.RowsAffected() + if err != nil { + return false, fmt.Errorf("renewing lease for %q: %w", key, err) + } + return n == 1, nil +} + +// releaseLease expires the row instead of deleting it. Inserters racing for a +// deleted key deadlock with each other, while takeovers of an expired row +// queue on its lock. Cleanup deletes the row later. +func (p *Persistence) releaseLease(ctx context.Context, key, token string) error { + if _, err := p.db.ExecContext(ctx, ` + UPDATE leases SET expires_at = UTC_TIMESTAMP(6) - INTERVAL 1 MICROSECOND + WHERE lease_key = ? AND token = ?`, key, token); err != nil { + return fmt.Errorf("releasing lease for %q: %w", key, err) + } + return nil +} + +// cleanupExpiredLeases deletes expired lease rows and returns how many it +// removed. Acquisition reclaims an expired row for its own key by itself, so +// this only keeps rows for keys nobody asks for again from accumulating. It +// runs from the maintenance loop. +// +// Rows are taken in batches with SKIP LOCKED, so a pass never waits on a +// concurrent acquire reclaiming a row or on another replica's pass, and two +// replicas cleaning at once delete disjoint rows. A pass keeps going while +// batches come back full and stops on the first short one. +func (p *Persistence) cleanupExpiredLeases(ctx context.Context) (int64, error) { + var deleted int64 + for { + var batch int64 + err := inTx(ctx, p.watchDB, func(tx *sql.Tx) error { + rows, err := tx.QueryContext(ctx, ` + SELECT lease_key FROM leases + WHERE expires_at <= UTC_TIMESTAMP(6) + LIMIT ? + FOR UPDATE SKIP LOCKED`, storesql.LeaseCleanupBatch) + if err != nil { + return err + } + var keys []any + for rows.Next() { + var key string + if err := rows.Scan(&key); err != nil { + rows.Close() + return err + } + keys = append(keys, key) + } + rows.Close() + if err := rows.Err(); err != nil || len(keys) == 0 { + return err + } + res, err := tx.ExecContext(ctx, `DELETE FROM leases WHERE lease_key IN `+inList(len(keys)), keys...) + if err != nil { + return err + } + batch, err = res.RowsAffected() + return err + }) + if err != nil { + return deleted, fmt.Errorf("deleting expired leases: %w", err) + } + deleted += batch + if batch < storesql.LeaseCleanupBatch { + return deleted, nil + } + } +} diff --git a/cmd/ateapi/internal/store/atemy/lease_test.go b/cmd/ateapi/internal/store/atemy/lease_test.go new file mode 100644 index 0000000000..8a6fc86ec0 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/lease_test.go @@ -0,0 +1,349 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "errors" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" +) + +// countLeases returns how many lease rows exist for key. +func countLeases(t *testing.T, s *Persistence, key string) int { + t.Helper() + return countRows(t, s, `SELECT COUNT(*) FROM leases WHERE lease_key = ?`, key) +} + +// seedExpiredLeases inserts n expired leases named expired-1 through +// expired-n. +func seedExpiredLeases(t *testing.T, s *Persistence, n int) { + t.Helper() + ctx := t.Context() + conn, err := s.db.Conn(ctx) + if err != nil { + t.Fatalf("acquiring connection: %v", err) + } + defer conn.Close() + if _, err := conn.ExecContext(ctx, `SET SESSION cte_max_recursion_depth = ?`, n+1); err != nil { + t.Fatalf("raising recursion depth: %v", err) + } + if _, err := conn.ExecContext(ctx, ` + INSERT INTO leases (lease_key, token, expires_at) + WITH RECURSIVE n (i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < ?) + SELECT CONCAT('expired-', i), 'old', UTC_TIMESTAMP(6) - INTERVAL 1 MINUTE FROM n`, n); err != nil { + t.Fatalf("seeding leases: %v", err) + } +} + +// TestAcquireLease_LeavesOtherKeysExpiredRows pins the acquisition path down +// to its own key: an expired row for some other key is the maintenance +// loop's to remove, not one more DELETE on every workflow's critical path. +func TestAcquireLease_LeavesOtherKeysExpiredRows(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + if _, err := s.db.ExecContext(ctx, ` + INSERT INTO leases (lease_key, token, expires_at) VALUES + ('expired', 'old', UTC_TIMESTAMP(6) - INTERVAL 1 MINUTE)`); err != nil { + t.Fatalf("seeding leases: %v", err) + } + lease, err := s.AcquireLease(ctx, "new") + if err != nil { + t.Fatalf("AcquireLease: %v", err) + } + defer lease.Close() + + if got := countLeases(t, s, "expired"); got != 1 { + t.Errorf("expired row for another key after AcquireLease: got %d, want 1 (left for cleanupExpiredLeases)", got) + } +} + +func TestCleanupExpiredLeases_RemovesOnlyExpiredRows(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + if _, err := s.db.ExecContext(ctx, ` + INSERT INTO leases (lease_key, token, expires_at) VALUES + ('expired', 'old', UTC_TIMESTAMP(6) - INTERVAL 1 MINUTE), + ('active', 'live', UTC_TIMESTAMP(6) + INTERVAL 1 HOUR)`); err != nil { + t.Fatalf("seeding leases: %v", err) + } + + deleted, err := s.cleanupExpiredLeases(ctx) + if err != nil { + t.Fatalf("cleanupExpiredLeases: %v", err) + } + if deleted != 1 { + t.Errorf("deleted = %d, want 1", deleted) + } + if expired, active := countLeases(t, s, "expired"), countLeases(t, s, "active"); expired != 0 || active != 1 { + t.Errorf("lease counts = expired:%d active:%d, want 0 and 1", expired, active) + } +} + +// TestCleanupExpiredLeases_DrainsAcrossBatches seeds more expired rows than +// one batch holds and checks a single pass keeps going until they are gone. +func TestCleanupExpiredLeases_DrainsAcrossBatches(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + const seeded = storesql.LeaseCleanupBatch*2 + 7 + seedExpiredLeases(t, s, seeded) + + deleted, err := s.cleanupExpiredLeases(ctx) + if err != nil { + t.Fatalf("cleanupExpiredLeases: %v", err) + } + if deleted != seeded { + t.Errorf("deleted = %d, want %d", deleted, seeded) + } + if remaining := countRows(t, s, `SELECT COUNT(*) FROM leases`); remaining != 0 { + t.Errorf("rows left after a full pass: %d, want 0", remaining) + } +} + +// TestCleanupExpiredLeases_SkipsLockedRows holds a row lock on one expired +// lease, as a concurrent acquire reclaiming it or another replica's pass +// would, and checks the pass returns without waiting on it and takes the +// row on the next pass once the lock is gone. +func TestCleanupExpiredLeases_SkipsLockedRows(t *testing.T) { + testCleanupSkipsLockedRows(t, setupMySQLPersistence(t)) +} + +func testCleanupSkipsLockedRows(t *testing.T, s *Persistence) { + t.Helper() + ctx := t.Context() + if _, err := s.db.ExecContext(ctx, ` + INSERT INTO leases (lease_key, token, expires_at) VALUES + ('locked', 'old', UTC_TIMESTAMP(6) - INTERVAL 1 MINUTE), + ('free', 'old', UTC_TIMESTAMP(6) - INTERVAL 1 MINUTE)`); err != nil { + t.Fatalf("seeding leases: %v", err) + } + holder, err := s.db.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("beginning holder transaction: %v", err) + } + defer holder.Rollback() //nolint:errcheck // no-op once committed + var one int + if err := holder.QueryRowContext(ctx, `SELECT 1 FROM leases WHERE lease_key = 'locked' FOR UPDATE`).Scan(&one); err != nil { + t.Fatalf("locking row: %v", err) + } + + passCtx, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + deleted, err := s.cleanupExpiredLeases(passCtx) + if err != nil { + t.Fatalf("cleanupExpiredLeases with a locked row: %v", err) + } + if deleted != 1 { + t.Errorf("deleted with one row locked = %d, want 1", deleted) + } + if locked, free := countLeases(t, s, "locked"), countLeases(t, s, "free"); locked != 1 || free != 0 { + t.Errorf("lease counts = locked:%d free:%d, want 1 and 0", locked, free) + } + + if err := holder.Commit(); err != nil { + t.Fatalf("releasing row lock: %v", err) + } + deleted, err = s.cleanupExpiredLeases(ctx) + if err != nil { + t.Fatalf("cleanupExpiredLeases after unlock: %v", err) + } + if deleted != 1 || countLeases(t, s, "locked") != 0 { + t.Errorf("after unlock: deleted = %d and %d rows left, want 1 and 0", deleted, countLeases(t, s, "locked")) + } +} + +func TestAcquireLease_ExpiresAfterHolderStops(t *testing.T) { + s := setupMySQLPersistence(t) + s.leaseTTL = 300 * time.Millisecond + holderCtx, cancelHolder := context.WithCancel(t.Context()) + lease, err := s.AcquireLease(holderCtx, "test-lease") + if err != nil { + t.Fatalf("AcquireLease failed: %v", err) + } + // Canceling the holder stops renewal without calling Close, modeling a + // process that disappeared and left its lease to expire. + cancelHolder() + select { + case <-lease.Context().Done(): + case <-time.After(time.Second): + t.Fatal("lease context was not cancelled with its holder") + } + if _, err := s.AcquireLease(t.Context(), "test-lease"); !errors.Is(err, store.ErrLeaseConflict) { + t.Fatalf("AcquireLease before the TTL ran out = %v, want ErrLeaseConflict", err) + } + + deadline := time.Now().Add(10 * time.Second) + for { + newLease, err := s.AcquireLease(t.Context(), "test-lease") + if err == nil { + newLease.Close() + return + } + if !errors.Is(err, store.ErrLeaseConflict) { + t.Fatalf("AcquireLease after the holder stopped: %v", err) + } + if time.Now().After(deadline) { + t.Fatal("lease was never released by expiry") + } + time.Sleep(100 * time.Millisecond) + } +} + +// The holder renews the lease past its TTL, and the lease's context ends once +// another token holds the row. +func TestAcquireLease_RenewsUntilTheRowChangesHands(t *testing.T) { + s := setupMySQLPersistence(t) + s.leaseTTL = time.Second + ctx := t.Context() + lease, err := s.AcquireLease(ctx, "renewed-lease") + if err != nil { + t.Fatalf("AcquireLease failed: %v", err) + } + defer lease.Close() + + time.Sleep(2 * s.leaseTTL) + if err := lease.Context().Err(); err != nil { + t.Fatalf("lease context ended while its holder was renewing: %v", err) + } + if other, err := s.AcquireLease(ctx, "renewed-lease"); !errors.Is(err, store.ErrLeaseConflict) { + if other != nil { + other.Close() + } + t.Fatalf("AcquireLease after %v of renewal = %v, want ErrLeaseConflict", 2*s.leaseTTL, err) + } + + if _, err := s.db.ExecContext(ctx, `UPDATE leases SET token = 'usurper' WHERE lease_key = 'renewed-lease'`); err != nil { + t.Fatalf("replacing the lease token: %v", err) + } + select { + case <-lease.Context().Done(): + case <-time.After(5 * time.Second): + t.Fatal("lease context outlived the loss of its row") + } +} + +// raceAcquire runs racers concurrent acquisitions of key and returns how many +// won. Winners hold their lease until every racer has tried, so later racers +// cannot win sequentially. +func raceAcquire(t *testing.T, s *Persistence, key string, racers int) int { + t.Helper() + winners := make(chan *store.Lease, racers) + start := make(chan struct{}) + var wg sync.WaitGroup + for i := range racers { + wg.Go(func() { + <-start + lease, err := s.AcquireLease(t.Context(), key) + if err != nil { + if !errors.Is(err, store.ErrLeaseConflict) { + t.Errorf("AcquireLease racer %d failed: %v", i, err) + } + return + } + winners <- lease + }) + } + close(start) + wg.Wait() + close(winners) + won := len(winners) + for lease := range winners { + lease.Close() + } + return won +} + +// TestAcquireLease_ConcurrentTakeover races many goroutines to take over an +// already-expired lease, which they all find through the UPDATE path, and +// asserts exactly one wins. +func TestAcquireLease_ConcurrentTakeover(t *testing.T) { + s := setupMySQLPersistence(t) + s.leaseTTL = time.Millisecond + holderCtx, cancelHolder := context.WithCancel(t.Context()) + initial, err := s.AcquireLease(holderCtx, "contested-lease") + if err != nil { + t.Fatalf("seeding initial lease failed: %v", err) + } + cancelHolder() + <-initial.Context().Done() + deadline := time.Now().Add(5 * time.Second) + for countRows(t, s, `SELECT COUNT(*) FROM leases WHERE lease_key = 'contested-lease' AND expires_at <= UTC_TIMESTAMP(6)`) != 1 { + if time.Now().After(deadline) { + t.Fatal("seeded lease never expired") + } + time.Sleep(5 * time.Millisecond) + } + s.leaseTTL = 10 * time.Second + + if got := raceAcquire(t, s, "contested-lease", 20); got != 1 { + t.Errorf("expected exactly 1 racer to win the expired lease, got %d", got) + } +} + +// TestAcquireLease_ConcurrentFirstAcquire races many goroutines to acquire a +// key with no row, which they all find through the INSERT path; the +// duplicate key must leave exactly one winner. +func TestAcquireLease_ConcurrentFirstAcquire(t *testing.T) { + s := setupMySQLPersistence(t) + if got := raceAcquire(t, s, "fresh-lease", 20); got != 1 { + t.Errorf("expected exactly 1 racer to win a fresh lease, got %d", got) + } +} + +// TestAcquireLease_ChurnKeepsOneHolder has many clients acquire and release +// one key at once. Every failure must read as a lost race, and the lease must +// never have two holders. +func TestAcquireLease_ChurnKeepsOneHolder(t *testing.T) { + s := setupMySQLPersistence(t) + var holders, maxHolders, acquired atomic.Int32 + var wg sync.WaitGroup + for g := range 16 { + wg.Go(func() { + for i := range 25 { + lease, err := s.AcquireLease(t.Context(), "churned-lease") + if err != nil { + if !errors.Is(err, store.ErrLeaseConflict) { + t.Errorf("client %d attempt %d: AcquireLease = %v, want nil or ErrLeaseConflict", g, i, err) + } + continue + } + acquired.Add(1) + n := holders.Add(1) + for { + m := maxHolders.Load() + if n <= m || maxHolders.CompareAndSwap(m, n) { + break + } + } + // Hold the lease briefly so overlapping holders would be seen. + time.Sleep(time.Millisecond) + holders.Add(-1) + lease.Close() + } + }) + } + wg.Wait() + if got := maxHolders.Load(); got != 1 { + t.Errorf("at most %d clients held the lease at once, want 1", got) + } + if acquired.Load() == 0 { + t.Error("no client ever acquired the lease") + } +} diff --git a/cmd/ateapi/internal/store/atemy/list.go b/cmd/ateapi/internal/store/atemy/list.go new file mode 100644 index 0000000000..68ce255c8e --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/list.go @@ -0,0 +1,108 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" +) + +// listPage runs a keyset page query whose rows hold len(keys) ordering +// columns followed by the proto, decodes each row, and trims the extra row +// the query fetched to learn whether another page follows. The query must +// select pageSize+1 rows. +func listPage[T any](ctx context.Context, q querier, kind storesql.Kind, scope string, pageSize int32, decode func(b []byte, keys []string) (T, error), query string, args ...any) ([]T, string, error) { + rows, err := q.QueryContext(ctx, query, args...) + if err != nil { + return nil, "", fmt.Errorf("listing %s rows: %w", kind, err) + } + defer rows.Close() + + cols, err := rows.Columns() + if err != nil { + return nil, "", fmt.Errorf("listing %s rows: %w", kind, err) + } + var items []T + var lastKeys [][]string + for rows.Next() { + keys := make([]string, len(cols)-1) + var protoBytes []byte + dest := make([]any, 0, len(cols)) + for i := range keys { + dest = append(dest, &keys[i]) + } + dest = append(dest, &protoBytes) + if err := rows.Scan(dest...); err != nil { + return nil, "", fmt.Errorf("scanning %s row: %w", kind, err) + } + item, err := decode(protoBytes, keys) + if err != nil { + return nil, "", err + } + items = append(items, item) + lastKeys = append(lastKeys, keys) + } + if err := rows.Err(); err != nil { + return nil, "", fmt.Errorf("listing %s rows: %w", kind, err) + } + if len(items) <= int(pageSize) { + return items, "", nil + } + return items[:pageSize], storesql.EncodePageToken(kind, scope, lastKeys[pageSize-1]), nil +} + +// rowID names a row of an atespace-scoped listing in errors. A scoped listing +// selects only the name; a global one selects atespace and name. +func rowID(atespace string, keys []string) []string { + if atespace != "" { + return append([]string{atespace}, keys...) + } + return keys +} + +// listScoped pages through an atespace-owned table keyed by (atespace, name): +// one atespace ordered by name, or every atespace ordered by (atespace, name) +// when atespace is empty. table is a trusted SQL identifier. +func listScoped[T any](ctx context.Context, q querier, table string, kind storesql.Kind, atespace string, opts store.ListOptions, decode func(b []byte, keys []string) (T, error)) ([]T, string, error) { + limit := int64(opts.PageSize) + 1 + if atespace != "" { + token, err := storesql.DecodePageToken(opts.PageToken, kind, atespace, 1) + if err != nil { + return nil, "", err + } + query, args := `SELECT name, proto FROM `+table+` WHERE atespace = ? ORDER BY name LIMIT ?`, []any{atespace, limit} + if len(token.Last) == 1 { + query, args = `SELECT name, proto FROM `+table+` WHERE atespace = ? AND name > ? ORDER BY name LIMIT ?`, []any{atespace, token.Last[0], limit} + } + return listPage(ctx, q, kind, atespace, opts.PageSize, decode, query, args...) + } + token, err := storesql.DecodePageToken(opts.PageToken, kind, "", 2) + if err != nil { + return nil, "", err + } + query, args := `SELECT atespace, name, proto FROM `+table+` ORDER BY atespace, name LIMIT ?`, []any{limit} + if len(token.Last) == 2 { + // Expanded rather than a row comparison so MySQL plans a primary + // key range scan. + query = `SELECT atespace, name, proto FROM ` + table + ` + WHERE atespace > ? OR (atespace = ? AND name > ?) + ORDER BY atespace, name LIMIT ?` + args = []any{token.Last[0], token.Last[0], token.Last[1], limit} + } + return listPage(ctx, q, kind, "", opts.PageSize, decode, query, args...) +} diff --git a/cmd/ateapi/internal/store/atemy/main_test.go b/cmd/ateapi/internal/store/atemy/main_test.go new file mode 100644 index 0000000000..393900a295 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/main_test.go @@ -0,0 +1,106 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "fmt" + "os" + "sync" + "testing" + "time" + + "github.com/testcontainers/testcontainers-go/modules/mysql" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/dockerenv" +) + +// One MySQL container serves every test in this package, and clearAll isolates +// them, so tests here are not safe to run with -parallel. +var ( + containerOnce sync.Once + containerDB *sql.DB + containerDSN string + containerMySQL *mysql.MySQLContainer + containerErr error +) + +func TestMain(m *testing.M) { + code := m.Run() + if containerDB != nil { + containerDB.Close() + } + if containerMySQL != nil { + if err := containerMySQL.Terminate(context.Background()); err != nil { + fmt.Fprintf(os.Stderr, "terminating MySQL testcontainer: %v\n", err) + if code == 0 { + code = 1 + } + } + } + os.Exit(code) +} + +func requireDB(t *testing.T) *sql.DB { + t.Helper() + containerOnce.Do(func() { + ctx := context.Background() + if err := dockerenv.Configure(ctx); err != nil { + containerErr = err + return + } + c, err := mysql.Run(ctx, "mysql:8.4", + mysql.WithDatabase("atemy"), + mysql.WithUsername("atemy"), + mysql.WithPassword("atemy"), + ) + if err != nil { + containerErr = err + return + } + containerMySQL = c + dsn, err := c.ConnectionString(ctx) + if err != nil { + containerErr = err + return + } + containerDSN = dsn + db, err := Open(dsn) + if err != nil { + containerErr = err + return + } + var pingErr error + for range 30 { + if pingErr = db.PingContext(ctx); pingErr == nil { + break + } + time.Sleep(500 * time.Millisecond) + } + if pingErr != nil { + containerErr = fmt.Errorf("pinging MySQL testcontainer after retries: %w", pingErr) + return + } + containerDB = db + }) + if containerErr != nil { + if dockerenv.Required() { + t.Fatalf("MySQL testcontainer unavailable and required (CI or REQUIRE_DOCKER is set): %v", containerErr) + } + t.Skipf("MySQL testcontainer unavailable (requires Docker): %v", containerErr) + } + return containerDB +} diff --git a/cmd/ateapi/internal/store/atemy/migrations/000001_initial.sql b/cmd/ateapi/internal/store/atemy/migrations/000001_initial.sql new file mode 100644 index 0000000000..120dd4926f --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/migrations/000001_initial.sql @@ -0,0 +1,122 @@ +-- Copyright 2026 Google LLC +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. + +-- +goose Up + +-- MySQL commits each DDL statement on its own, so a failed run can leave tables +-- with no ledger row. Every statement is idempotent so the next startup +-- completes the file. + +-- No table declares a foreign key, because PlanetScale disables them by +-- default. atemy enforces every parent and child relationship in the +-- transaction that writes the child or deletes the parent. Key columns use a +-- binary collation so names compare byte for byte, as text equality does in +-- PostgreSQL. + +CREATE TABLE IF NOT EXISTS atespaces ( + name VARCHAR(255) NOT NULL, + uid VARCHAR(255) NOT NULL, + version BIGINT NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (name) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; + +CREATE TABLE IF NOT EXISTS actors ( + atespace VARCHAR(255) NOT NULL, + name VARCHAR(255) NOT NULL, + uid VARCHAR(255) NOT NULL, + version BIGINT NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (atespace, name) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; + +CREATE TABLE IF NOT EXISTS actor_egress_policies ( + atespace VARCHAR(255) NOT NULL, + actor_name VARCHAR(255) NOT NULL, + uid VARCHAR(255) NOT NULL, + version BIGINT NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (atespace, actor_name) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; + +CREATE TABLE IF NOT EXISTS actor_templates ( + atespace VARCHAR(255) NOT NULL, + name VARCHAR(255) NOT NULL, + uid VARCHAR(255) NOT NULL, + version BIGINT NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (atespace, name) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; + +CREATE TABLE IF NOT EXISTS tags ( + atespace VARCHAR(255) NOT NULL, + name VARCHAR(255) NOT NULL, + uid VARCHAR(255) NOT NULL, + version BIGINT NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (atespace, name) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; + +-- Workers are global-scoped and named by their Kubernetes pod UID, so name +-- alone is the primary key. +CREATE TABLE IF NOT EXISTS workers ( + name VARCHAR(255) NOT NULL, + uid VARCHAR(255) NOT NULL, + version BIGINT NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (name), + UNIQUE KEY workers_uid (uid) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; + +-- One row per Actor, keyed by Actor UID because an Actor has at most one +-- Worker. Kept separate from workers so Worker reads, writes, and watch events +-- do not grow with occupancy. The primary key finds an Actor's Worker; +-- worker_name lists a Worker's Actors. +CREATE TABLE IF NOT EXISTS worker_assignments ( + actor_uid VARCHAR(255) NOT NULL, + worker_name VARCHAR(255) NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (actor_uid), + KEY worker_assignments_worker_idx (worker_name) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; + +-- Transactional outbox backing WatchWorkers. Each worker write inserts one +-- row as its last statement, taking the next seq. Retention deletes the oldest +-- rows and records the greatest deleted seq in worker_outbox_trim. +CREATE TABLE IF NOT EXISTS worker_outbox ( + seq BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, + created_at DATETIME(6) NOT NULL, + payload LONGBLOB NOT NULL, + PRIMARY KEY (seq) +); + +-- Holds one row with the greatest seq retention has deleted. Watchers compare +-- it with their cursor to detect deleted rows they had not consumed. +CREATE TABLE IF NOT EXISTS worker_outbox_trim ( + id TINYINT UNSIGNED NOT NULL, + seq BIGINT UNSIGNED NOT NULL, + PRIMARY KEY (id), + CONSTRAINT worker_outbox_trim_singleton CHECK (id = 1) +); + +INSERT INTO worker_outbox_trim (id, seq) VALUES (1, 0) + ON DUPLICATE KEY UPDATE id = id; + +CREATE TABLE IF NOT EXISTS leases ( + lease_key VARCHAR(512) NOT NULL, + token VARCHAR(255) NOT NULL, + expires_at DATETIME(6) NOT NULL, + PRIMARY KEY (lease_key), + KEY leases_expires_at_idx (expires_at) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; diff --git a/cmd/ateapi/internal/store/atemy/migrations/000002_openfga.sql b/cmd/ateapi/internal/store/atemy/migrations/000002_openfga.sql new file mode 100644 index 0000000000..2aaac1b088 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/migrations/000002_openfga.sql @@ -0,0 +1,76 @@ +-- Copyright 2026 Google LLC and The OpenFGA Authors +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. + +-- +goose Up + +-- The OpenFGA storage schema after github.com/openfga/openfga MySQL migrations +-- 001 through 008, matching the PostgreSQL schema pinned in atepg. It shares +-- the Goose directory with Substrate tables so resource mutations and +-- authorization tuple updates commit in one MySQL transaction. + +CREATE TABLE IF NOT EXISTS tuple ( + store CHAR(26) NOT NULL, + object_type VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + object_id VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + relation VARCHAR(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + _user VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + user_type VARCHAR(7) NOT NULL, + ulid CHAR(26) NOT NULL, + inserted_at TIMESTAMP NOT NULL, + condition_name VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL, + condition_context LONGBLOB, + PRIMARY KEY (store, object_type, object_id, relation, _user), + UNIQUE KEY idx_tuple_ulid (ulid), + KEY idx_user_lookup (store, _user, relation, object_type, object_id) +) DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS authorization_model ( + store CHAR(26) NOT NULL, + authorization_model_id CHAR(26) NOT NULL, + type VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + type_definition BLOB, + schema_version VARCHAR(5) NOT NULL DEFAULT '1.0', + serialized_protobuf LONGBLOB, + PRIMARY KEY (store, authorization_model_id, type) +) DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS store ( + id CHAR(26) NOT NULL, + name VARCHAR(64) NOT NULL, + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NULL, + deleted_at TIMESTAMP NULL, + PRIMARY KEY (id) +) DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS assertion ( + store CHAR(26) NOT NULL, + authorization_model_id CHAR(26) NOT NULL, + assertions BLOB, + PRIMARY KEY (store, authorization_model_id) +) DEFAULT CHARSET = utf8mb4; + +CREATE TABLE IF NOT EXISTS changelog ( + store CHAR(26) NOT NULL, + object_type VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + object_id VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + relation VARCHAR(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + _user VARCHAR(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + operation INTEGER NOT NULL, + ulid CHAR(26) NOT NULL, + inserted_at TIMESTAMP NOT NULL, + condition_name VARCHAR(256) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL, + condition_context LONGBLOB, + PRIMARY KEY (store, ulid, object_type) +) DEFAULT CHARSET = utf8mb4; diff --git a/cmd/ateapi/internal/store/atemy/migrations/000003_access_policies.sql b/cmd/ateapi/internal/store/atemy/migrations/000003_access_policies.sql new file mode 100644 index 0000000000..88c1b922cb --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/migrations/000003_access_policies.sql @@ -0,0 +1,32 @@ +-- Copyright 2026 Google LLC +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. + +-- +goose Up + +CREATE TABLE IF NOT EXISTS global_access_policy ( + id TINYINT UNSIGNED NOT NULL, + uid VARCHAR(255) NOT NULL, + version BIGINT NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (id), + CONSTRAINT global_access_policy_singleton CHECK (id = 1) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; + +CREATE TABLE IF NOT EXISTS atespace_access_policies ( + atespace_name VARCHAR(255) NOT NULL, + uid VARCHAR(255) NOT NULL, + version BIGINT NOT NULL, + proto LONGBLOB NOT NULL, + PRIMARY KEY (atespace_name) +) DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin; diff --git a/cmd/ateapi/internal/store/atemy/migrations_test.go b/cmd/ateapi/internal/store/atemy/migrations_test.go new file mode 100644 index 0000000000..6ed9a5d8d8 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/migrations_test.go @@ -0,0 +1,404 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "database/sql" + "io/fs" + "path/filepath" + "regexp" + "slices" + "strconv" + "strings" + "testing" + "time" + + "github.com/go-sql-driver/mysql" + "github.com/google/go-cmp/cmp" + "github.com/openfga/openfga/assets" + "github.com/pressly/goose/v3/database" +) + +const pinnedOpenFGAMigrationVersion = 8 + +var ( + transactionControl = regexp.MustCompile(`(?im)^\s*(BEGIN|START\s+TRANSACTION|COMMIT|ROLLBACK)\s*;`) + createTable = regexp.MustCompile(`(?i)\bCREATE\s+TABLE\b`) + createTableIfNotExists = regexp.MustCompile(`(?i)\bCREATE\s+TABLE\s+IF\s+NOT\s+EXISTS\b`) + insertStatement = regexp.MustCompile(`(?is)\bINSERT\s+INTO\b[^;]*;`) +) + +// migrationVersion parses the numeric prefix of a migration file name. +func migrationVersion(t *testing.T, name string) int { + t.Helper() + prefix, _, ok := strings.Cut(filepath.Base(name), "_") + if !ok { + t.Fatalf("unexpected migration filename %q", name) + } + v, err := strconv.Atoi(prefix) + if err != nil { + t.Fatalf("parse migration version from %q: %v", name, err) + } + return v +} + +// TestOpenFGAMigrationVersionGuard fails when a github.com/openfga/openfga bump +// adds an upstream MySQL migration. migrations/000002_openfga.sql holds the +// final state of upstream migrations 001 through 008, and +// cmd/ateapi/internal/authz/datastore_mysql.go adapts upstream queries to run +// on the caller's transaction. Port each new upstream migration and recheck +// those queries. +func TestOpenFGAMigrationVersionGuard(t *testing.T) { + entries, err := fs.ReadDir(assets.EmbedMigrations, assets.MySQLMigrationDir) + if err != nil { + t.Fatalf("read embedded OpenFGA migrations: %v", err) + } + var maxVersion int + for _, entry := range entries { + if entry.IsDir() || filepath.Ext(entry.Name()) != ".sql" { + continue + } + maxVersion = max(maxVersion, migrationVersion(t, entry.Name())) + } + if maxVersion != pinnedOpenFGAMigrationVersion { + t.Fatalf( + "OpenFGA embedded MySQL migrations are at version %d, but 000002_openfga.sql and cmd/ateapi/internal/authz/datastore_mysql.go are pinned to version %d; port any new OpenFGA DDL to cmd/ateapi/internal/store/atemy/migrations/ and verify the MySQL datastore before updating pinnedOpenFGAMigrationVersion", + maxVersion, pinnedOpenFGAMigrationVersion, + ) + } +} + +func TestMigrationPolicy(t *testing.T) { + err := fs.WalkDir(migrationFiles, "migrations", func(path string, entry fs.DirEntry, err error) error { + if err != nil || entry.IsDir() { + return err + } + if !strings.HasSuffix(path, ".sql") { + return nil + } + data, err := fs.ReadFile(migrationFiles, path) + if err != nil { + return err + } + sql := string(data) + upperSQL := strings.ToUpper(sql) + if strings.Count(sql, "-- +goose Up") != 1 { + t.Errorf("%s must contain exactly one Goose Up annotation", path) + } + if strings.Contains(upperSQL, "-- +GOOSE DOWN") { + t.Errorf("%s must not contain a Goose Down migration", path) + } + if strings.Contains(upperSQL, "-- +GOOSE NO TRANSACTION") { + t.Errorf("%s must let Goose run it in a transaction", path) + } + // MySQL commits each DDL statement on its own, so a file that fails + // partway must be safe to run again from the top. + statements := stripSQLComments(sql) + if creates, guarded := createTable.FindAllStringIndex(statements, -1), createTableIfNotExists.FindAllStringIndex(statements, -1); len(creates) != len(guarded) { + t.Errorf("%s has %d CREATE TABLE statements but only %d use IF NOT EXISTS", path, len(creates), len(guarded)) + } + for _, stmt := range insertStatement.FindAllString(statements, -1) { + if !strings.Contains(strings.ToUpper(stmt), "ON DUPLICATE KEY UPDATE") { + t.Errorf("%s seeds rows without ON DUPLICATE KEY UPDATE: %q", path, stmt) + } + } + if strings.Contains(upperSQL, "-- +GOOSE ENVSUB") { + t.Errorf("%s must not use Goose environment substitution", path) + } + if transactionControl.MatchString(sql) { + t.Errorf("%s must let Goose control the transaction", path) + } + return nil + }) + if err != nil { + t.Fatalf("check MySQL migrations: %v", err) + } +} + +// stripSQLComments drops whole-line -- comments, including Goose annotations. +func stripSQLComments(sql string) string { + var kept []string + for line := range strings.Lines(sql) { + if !strings.HasPrefix(strings.TrimSpace(line), "--") { + kept = append(kept, line) + } + } + return strings.Join(kept, "") +} + +// Tables Goose did not create are refused rather than adopted by CREATE TABLE +// IF NOT EXISTS. +func TestMigrations_RejectTablesWithoutALedger(t *testing.T) { + createTestDatabase(t, "atemy_unversioned") + db := openAdmin(t, "atemy_unversioned") + if _, err := db.ExecContext(t.Context(), `CREATE TABLE atespaces (name VARCHAR(255) PRIMARY KEY)`); err != nil { + t.Fatalf("creating a stray atespaces table: %v", err) + } + if _, err := NewPersistence(t.Context(), db); err == nil || !strings.Contains(err.Error(), "without a migration ledger") { + t.Fatalf("NewPersistence over tables without a ledger = %v, want a refusal", err) + } +} + +// TestMigrations_ResumeAPartialInitialMigration simulates a crash partway +// through 000001: MySQL committed some of its tables and a seed row, but +// Goose recorded no version. The next startup must complete the file. +func TestMigrations_ResumeAPartialInitialMigration(t *testing.T) { + createTestDatabase(t, "atemy_partial_migration") + db := openAdmin(t, "atemy_partial_migration") + ctx := t.Context() + initial, err := fs.ReadFile(migrationFiles, "migrations/000001_initial.sql") + if err != nil { + t.Fatalf("reading 000001: %v", err) + } + // Goose creates its ledger, holding version 0, before it runs 000001. + ledger, err := database.NewStore(database.DialectMySQL, migrationTableName) + if err != nil { + t.Fatalf("creating the migration store: %v", err) + } + if err := ledger.CreateVersionTable(ctx, db); err != nil { + t.Fatalf("creating the migration ledger: %v", err) + } + if err := ledger.Insert(ctx, db, database.InsertRequest{Version: 0}); err != nil { + t.Fatalf("recording version 0: %v", err) + } + var ran []string + for stmt := range strings.SplitSeq(stripSQLComments(string(initial)), ";") { + if stmt = strings.TrimSpace(stmt); stmt == "" { + continue + } + if _, err := db.ExecContext(ctx, stmt); err != nil { + t.Fatalf("running %q: %v", stmt, err) + } + ran = append(ran, stmt) + if strings.Contains(stmt, "INSERT INTO worker_outbox_trim") { + break + } + } + if len(ran) < 3 { + t.Fatalf("ran %d statements of 000001 before the seed row, want a partial run", len(ran)) + } + var tables int + if err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = DATABASE() AND table_name = 'leases'`).Scan(&tables); err != nil || tables != 0 { + t.Fatalf("leases table count = %d, %v; want 0 so the run is partial", tables, err) + } + + p, err := NewPersistence(ctx, db) + if err != nil { + t.Fatalf("NewPersistence after a partial migration failed: %v", err) + } + p.Close() + if diff := cmp.Diff(embeddedMigrationVersions(t), appliedMigrationVersions(t, db)); diff != "" { + t.Fatalf("applied migration versions (-embedded +applied):\n%s", diff) + } + var trimRows int + if err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM worker_outbox_trim`).Scan(&trimRows); err != nil || trimRows != 1 { + t.Fatalf("worker_outbox_trim rows = %d, %v; want 1", trimRows, err) + } +} + +func embeddedMigrationVersions(t *testing.T) []int64 { + t.Helper() + names, err := fs.Glob(migrationFiles, "migrations/*.sql") + if err != nil { + t.Fatalf("listing migrations: %v", err) + } + var versions []int64 + for _, name := range names { + versions = append(versions, int64(migrationVersion(t, name))) + } + slices.Sort(versions) + return versions +} + +func appliedMigrationVersions(t *testing.T, db *sql.DB) []int64 { + t.Helper() + rows, err := db.QueryContext(t.Context(), ` + SELECT version_id FROM schema_migrations + WHERE version_id > 0 AND is_applied ORDER BY version_id`) + if err != nil { + t.Fatalf("reading applied migration versions: %v", err) + } + defer rows.Close() + var versions []int64 + for rows.Next() { + var v int64 + if err := rows.Scan(&v); err != nil { + t.Fatalf("scanning migration version: %v", err) + } + versions = append(versions, v) + } + if err := rows.Err(); err != nil { + t.Fatalf("reading applied migration versions: %v", err) + } + return versions +} + +func TestMigrationsConcurrentStartup(t *testing.T) { + const database = "atemy_concurrent_startup" + dsn := createTestDatabase(t, database) + errs := make(chan error, 2) + for range 2 { + go func() { + p, err := Connect(t.Context(), ConnectConfig{ReadWriteDSN: dsn, OwnerDSN: dsn}) + if p != nil { + p.Close() + p.DB().Close() + } + errs <- err + }() + } + for range 2 { + if err := <-errs; err != nil { + t.Fatalf("Connect failed: %v", err) + } + } + // Every migration applied once and no more: two racing starts must not each + // record the same version. + if diff := cmp.Diff(embeddedMigrationVersions(t), appliedMigrationVersions(t, openAdmin(t, database))); diff != "" { + t.Fatalf("applied migration versions (-embedded +applied):\n%s", diff) + } +} + +func TestMigrationsWaitForInProgressMigration(t *testing.T) { + const database = "atemy_migration_lock_wait" + createTestDatabase(t, database) + ctx := t.Context() + admin := openAdmin(t, "mysql") + lockConn, err := admin.Conn(ctx) + if err != nil { + t.Fatalf("acquiring lock connection: %v", err) + } + defer lockConn.Close() + lockName := migrationLockName(database) + var got int + if err := lockConn.QueryRowContext(ctx, `SELECT GET_LOCK(?, 0)`, lockName).Scan(&got); err != nil || got != 1 { + t.Fatalf("taking migration lock = %d, %v; want 1", got, err) + } + + db := openAdmin(t, database) + result := make(chan error, 1) + go func() { result <- applyMigrations(ctx, db) }() + + deadline := time.Now().Add(10 * time.Second) + for { + var pending int + if err := admin.QueryRowContext(ctx, ` + SELECT COUNT(*) FROM performance_schema.metadata_locks + WHERE OBJECT_TYPE = 'USER LEVEL LOCK' AND OBJECT_NAME = ? AND LOCK_STATUS = 'PENDING'`, lockName).Scan(&pending); err != nil { + t.Fatalf("reading pending locks: %v", err) + } + if pending > 0 { + break + } + if time.Now().After(deadline) { + t.Fatal("migration never waited on the held lock") + } + time.Sleep(10 * time.Millisecond) + } + select { + case err := <-result: + t.Fatalf("migration returned while its lock was held: %v", err) + default: + } + if versions := appliedMigrationVersions(t, db); len(versions) != 0 { + t.Fatalf("migrations %v applied while another session held the lock", versions) + } + + var released int + if err := lockConn.QueryRowContext(ctx, `SELECT RELEASE_LOCK(?)`, lockName).Scan(&released); err != nil || released != 1 { + t.Fatalf("releasing migration lock = %d, %v; want 1", released, err) + } + select { + case err := <-result: + if err != nil { + t.Fatalf("migration failed after the lock was released: %v", err) + } + case <-time.After(30 * time.Second): + t.Fatal("migration did not finish after the lock was released") + } + if diff := cmp.Diff(embeddedMigrationVersions(t), appliedMigrationVersions(t, db)); diff != "" { + t.Fatalf("applied migration versions (-embedded +applied):\n%s", diff) + } +} + +// A schema migrated by a newer binary is ahead of this one's files. An older +// replica in a rolling deploy must still start against it. +func TestMigrations_StartAgainstASchemaAhead(t *testing.T) { + createTestDatabase(t, "atemy_migration_ahead") + db := openAdmin(t, "atemy_migration_ahead") + ctx := t.Context() + p, err := NewPersistence(ctx, db) + if err != nil { + t.Fatalf("creating current schema: %v", err) + } + p.Close() + if _, err := db.ExecContext(ctx, ` + INSERT INTO schema_migrations (version_id, is_applied) + SELECT MAX(version_id) + 1, TRUE FROM schema_migrations`); err != nil { + t.Fatalf("setting ahead migration state: %v", err) + } + p, err = NewPersistence(ctx, db) + if err != nil { + t.Fatalf("NewPersistence against a schema ahead failed: %v", err) + } + p.Close() +} + +// Multi-primary replication hands out AUTO_INCREMENT values in steps, which +// watchers would track as gaps that never fill. +func TestRequireAutoIncrementStep(t *testing.T) { + requireDB(t) + db, err := Open(containerDSN) + if err != nil { + t.Fatalf("opening pool: %v", err) + } + defer db.Close() + // One connection, kept, so the session setting applies to every query. + db.SetMaxOpenConns(1) + db.SetMaxIdleConns(1) + ctx := t.Context() + if err := requireAutoIncrementStep(ctx, db); err != nil { + t.Fatalf("requireAutoIncrementStep with the default step = %v, want nil", err) + } + if _, err := db.ExecContext(ctx, `SET SESSION auto_increment_increment = 2`); err != nil { + t.Fatalf("setting auto_increment_increment: %v", err) + } + if err := requireAutoIncrementStep(ctx, db); err == nil || !strings.Contains(err.Error(), "auto_increment_increment = 1, got 2") { + t.Errorf("requireAutoIncrementStep with a step of 2 = %v, want a step error", err) + } + if _, err := NewPersistence(ctx, db); err == nil || !strings.Contains(err.Error(), "auto_increment_increment") { + t.Errorf("NewPersistence with a step of 2 = %v, want a step error", err) + } +} + +// The read/write DSN can turn strict mode off for the sessions that write, +// while the owner pool that migrates stays strict. +func TestNewPersistence_RequiresStrictReadWriteSessions(t *testing.T) { + ctx := t.Context() + owner := requireDB(t) + cfg, err := mysql.ParseDSN(containerDSN) + if err != nil { + t.Fatalf("parsing DSN: %v", err) + } + cfg.Params = map[string]string{"sql_mode": "''"} + db, err := Open(cfg.FormatDSN()) + if err != nil { + t.Fatalf("opening pool: %v", err) + } + defer db.Close() + if _, err := newPersistence(ctx, db, db, owner); err == nil || !strings.Contains(err.Error(), "strict sql_mode") { + t.Errorf("newPersistence with a non-strict read/write pool = %v, want a strict mode error", err) + } +} diff --git a/cmd/ateapi/internal/store/atemy/outbox.go b/cmd/ateapi/internal/store/atemy/outbox.go new file mode 100644 index 0000000000..9f88822241 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/outbox.go @@ -0,0 +1,515 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// The worker outbox: every worker write appends one event row to +// worker_outbox in the same transaction (writeAndAppendEvent), per-replica +// watchers poll it with a seq cursor (WatchWorkers), and the maintenance loop +// deletes rows past retention and records how far it deleted, so lagging +// watchers detect the loss and resync (trimWorkerOutboxOlderThan). +// +// seq is an AUTO_INCREMENT key taken by each write's last statement, so +// writes never wait on one another, as with atepg's xids. A seq is assigned +// before its transaction commits, though, and InnoDB exposes its oldest open +// transaction only through INNODB_TRX, which Vitess does not serve, so a +// poller can see seq N+1 before seq N commits. +// +// The poller delivers what it sees and remembers each skipped seq as +// pending, delivering it when it commits. A pending seq is dropped only once +// a locking probe shows no write holds it, which means it rolled back. This +// reorders events only across workers: two writes to one worker hold its row +// lock in turn, so the later write's seq is assigned after the earlier one +// commits. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + "log/slog" + "slices" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// writeAndAppendEvent runs fn inside a transaction, then--only if fn reports a +// worker worth publishing--appends the event to the worker_outbox table in the +// same transaction, so watchers see it if and only if the transaction commits. +// fn returns the worker the event carries, or nil to skip the event; it is +// returned from writeAndAppendEvent so callers get back what actually +// committed. +func (p *Persistence) writeAndAppendEvent(ctx context.Context, eventType store.WorkerEventType, fn func(ctx context.Context, tx *sql.Tx) (*ateapipb.Worker, error)) (*ateapipb.Worker, error) { + var worker *ateapipb.Worker + var payload []byte + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + payload = nil + var err error + worker, err = fn(ctx, tx) + if err != nil || worker == nil { + return err + } + payload, err = storesql.MarshalWorkerEvent(eventType, worker) + if err != nil { + return fmt.Errorf("marshaling worker event: %w", err) + } + // Last, so the seq is assigned as close to commit as possible: a seq + // waiting on its commit is a gap pollers must track. + if _, err := tx.ExecContext(ctx, ` + INSERT INTO worker_outbox (created_at, payload) + VALUES (UTC_TIMESTAMP(6), ?)`, payload); err != nil { + return fmt.Errorf("appending worker outbox: %w", err) + } + return nil + }) + if err != nil { + return nil, err + } + if payload != nil { + p.watchers.Publish(ctx, payload) + } + return worker, nil +} + +const ( + // Bound worker-event delivery latency. + outboxPollInterval = 50 * time.Millisecond + + // Caps rows per poll and per retention transaction. A larger burst carries + // over to the next poll. + outboxBatch = 1024 + + // Minimum time retention keeps outbox rows. + outboxRetentionAge = 15 * time.Minute + + // How long a skipped seq stays pending before a probe asks whether a write + // still holds it. It covers the moment inside a write's INSERT between + // taking the seq and locking the new row, when neither shows. + outboxGapGrace = 2 * time.Second + + // How far back a new watch looks for seqs still committing. A write that + // took its seq longer ago than this and commits after the watch starts is + // not delivered to that watch. + outboxSubscribeWindow = 5 * time.Minute + + // Caps the skipped seqs a watcher tracks; past it the watcher resyncs. + outboxMaxPending = 4096 + + // Bounds stale-serving during polling outages: after this duration of + // uninterrupted failures, the watch closes and forces a full cache relist. + outboxPollFailureCloseAfter = 30 * time.Second +) + +// trimWorkerOutboxOlderThan deletes, in batches, the longest seq prefix of +// rows created more than age ago by the database clock. Each batch raises the +// trim mark to its greatest seq in the same transaction. The replica that +// locks the trim row runs the pass; the others skip it, as atepg's elected +// retention does. +func (p *Persistence) trimWorkerOutboxOlderThan(ctx context.Context, age time.Duration) error { + for { + var expiredSeqs []any + err := inTx(ctx, p.watchDB, func(tx *sql.Tx) error { + expiredSeqs = nil + var mark uint64 + err := tx.QueryRowContext(ctx, `SELECT seq FROM worker_outbox_trim WHERE id = 1 FOR UPDATE SKIP LOCKED`).Scan(&mark) + if errors.Is(err, sql.ErrNoRows) { + return nil // another replica is maintaining; next tick retries + } + if err != nil { + return fmt.Errorf("electing outbox retention: %w", err) + } + rows, err := tx.QueryContext(ctx, ` + SELECT seq, created_at < UTC_TIMESTAMP(6) - INTERVAL ? MICROSECOND + FROM worker_outbox ORDER BY seq LIMIT ?`, age.Microseconds(), outboxBatch) + if err != nil { + return fmt.Errorf("reading expired outbox rows: %w", err) + } + for rows.Next() { + var seq uint64 + var expired bool + if err := rows.Scan(&seq, &expired); err != nil { + rows.Close() + return fmt.Errorf("scanning outbox row: %w", err) + } + if !expired { + break + } + expiredSeqs = append(expiredSeqs, seq) + } + rows.Close() + if err := rows.Err(); err != nil { + return fmt.Errorf("reading expired outbox rows: %w", err) + } + if len(expiredSeqs) == 0 { + return nil + } + // Only the rows read, so a write still committing below the last + // seq read is never waited on. A watcher waiting on such a row + // closes once the trim mark passes it. + if _, err := tx.ExecContext(ctx, `DELETE FROM worker_outbox WHERE seq IN `+inList(len(expiredSeqs)), expiredSeqs...); err != nil { + return fmt.Errorf("deleting expired outbox rows: %w", err) + } + if _, err := tx.ExecContext(ctx, `UPDATE worker_outbox_trim SET seq = GREATEST(seq, ?) WHERE id = 1`, expiredSeqs[len(expiredSeqs)-1]); err != nil { + return fmt.Errorf("recording outbox trim mark: %w", err) + } + return nil + }) + if err != nil || len(expiredSeqs) < outboxBatch { + return err + } + } +} + +// outboxMarks are the values a poll checks against its cursor, read after +// the poll's rows. +type outboxMarks struct { + // trim is the greatest seq retention has deleted. + trim uint64 + // head is the greatest seq written: the larger of the greatest stored + // seq and trim. + head uint64 + // server identifies the MySQL server that answered (@@server_uuid). A + // failover to another server can lose writes it never received. + server string +} + +func (p *Persistence) readOutboxMarks(ctx context.Context) (outboxMarks, error) { + var m outboxMarks + err := p.watchDB.QueryRowContext(ctx, ` + SELECT t.seq, GREATEST(t.seq, COALESCE((SELECT MAX(seq) FROM worker_outbox), 0)), @@server_uuid + FROM worker_outbox_trim AS t WHERE t.id = 1`).Scan(&m.trim, &m.head, &m.server) + return m, err +} + +// outboxCursor is a watcher's position: every seq at or below seq was +// delivered, written before the watch, or is pending. +type outboxCursor struct { + seq uint64 + // pending holds skipped seqs below seq, each with when it was skipped. + pending map[uint64]time.Time + server string +} + +// wait records the seqs strictly between lo and hi as pending, stopping one +// past outboxMaxPending so closeReason reports the overflow. +func (c *outboxCursor) wait(lo, hi uint64, now time.Time) { + for s := lo + 1; s < hi && len(c.pending) <= outboxMaxPending; s++ { + c.pending[s] = now + } +} + +// Reasons a watch closes for a resync, logged as is. +const ( + resyncServerChanged = "database server changed under the outbox; closing watch for resync" + resyncLostWrites = "worker outbox moved behind the watch cursor; closing watch for resync" + resyncFellBehind = "worker watch fell behind outbox retention; closing for resync" + resyncTooManyPending = "worker watch is waiting on too many uncommitted outbox rows; closing for resync" + resyncPendingTrimmed = "outbox retention deleted a row the worker watch was waiting for; closing for resync" +) + +// closeReason reports why marks force a resync, with the attributes to log, +// or "" if they do not. +func (c *outboxCursor) closeReason(m outboxMarks) (string, []any) { + switch { + case m.server != c.server: + return resyncServerChanged, []any{slog.String("was", c.server), slog.String("now", m.server)} + case m.head < c.seq: + return resyncLostWrites, []any{slog.Uint64("cursor_seq", c.seq), slog.Uint64("head_seq", m.head)} + case m.trim > c.seq: + return resyncFellBehind, []any{slog.Uint64("cursor_seq", c.seq)} + case len(c.pending) > outboxMaxPending: + return resyncTooManyPending, []any{slog.Int("pending", len(c.pending))} + } + for s := range c.pending { + if s <= m.trim { + return resyncPendingTrimmed, []any{slog.Uint64("pending_seq", s), slog.Uint64("trim_seq", m.trim)} + } + } + return "", nil +} + +// WatchWorkers subscribes by polling the worker_outbox table with a seq +// cursor. The cursor starts at the greatest seq written, and the seqs below it +// still uncommitted start pending, so the watch sees every write that commits +// after it subscribes. +// +// This process's own writes are published at commit, ahead of the poll, and +// again on the poll, so consumers must reconcile versions and tolerate +// duplicates. +// +// The watch closes to force the consumer to resync from the primary tables +// when it may have missed an event: it fell behind retention, the server +// changed or lost writes, or it waits on too many uncommitted seqs. Unlike +// atepg's UNLOGGED outbox, worker_outbox is durable, so a restart of the same +// server loses no events. +func (p *Persistence) WatchWorkers(ctx context.Context) (*store.WorkerWatch, error) { + watchCtx, cancel := context.WithCancel(ctx) + + cursor, err := p.subscribeWorkerOutbox(watchCtx) + if err != nil { + cancel() + return nil, err + } + + ch := make(chan store.WorkerEvent, 128) + // Committed writes in this process are published straight onto ch, ahead + // of the poll that would carry them. + p.watchers.Add(ch) + go func() { + defer func() { + p.watchers.Remove(ch) + close(ch) + }() + ticker := time.NewTicker(outboxPollInterval) + defer ticker.Stop() + // failingSince bounds how long consumers serve stale state during an outage. + var failingSince time.Time + for { + select { + case <-watchCtx.Done(): + return + case <-ticker.C: + } + // Drain until a batch is partial. Sleeping between full batches would + // cap throughput and cause unrecoverable lag during bursts. + for { + batch, full, resync, err := p.pollWorkerOutbox(watchCtx, cursor) + if err != nil { + if watchCtx.Err() != nil { + return + } + // Transient failure: retry next tick. Persistent failure (past the + // deadline): close the watch to flip workercache not-ready, forcing + // callers to fail fast instead of serving a frozen fleet view. + if failingSince.IsZero() { + failingSince = time.Now() + } else if time.Since(failingSince) > p.pollFailureCloseAfter { + slog.WarnContext(watchCtx, "worker outbox polling has failed persistently; closing watch", + slog.Duration("failing_for", time.Since(failingSince)), slog.Any("err", err)) + return + } + slog.WarnContext(watchCtx, "worker outbox poll failed", slog.Any("err", err)) + break + } + failingSince = time.Time{} + if resync { + return // pollWorkerOutbox logged why + } + for _, r := range batch { + event, err := storesql.UnmarshalWorkerEvent(r.payload) + if err != nil { + // Close to force a relist. Skipping it would cause silent + // data loss. The fresh watch starts at the current head, + // naturally bypassing the corrupt row to prevent a boot loop. + slog.ErrorContext(watchCtx, "worker event unmarshal failed; closing watch for resync", + slog.Uint64("seq", r.seq), slog.Any("err", err)) + return + } + select { + case ch <- event: + case <-watchCtx.Done(): + return + } + } + if !full { + break + } + } + } + }() + return store.NewWorkerWatch(ch, cancel), nil +} + +// subscribeWorkerOutbox places a new watch at the greatest seq written. Seqs +// missing among the rows written in the last outboxSubscribeWindow, and +// between the oldest of them and the row before it, may belong to writes still +// committing, so they start pending. Rows present are not delivered. +func (p *Persistence) subscribeWorkerOutbox(ctx context.Context) (*outboxCursor, error) { + marks, err := p.readOutboxMarks(ctx) + if err != nil { + return nil, fmt.Errorf("reading worker outbox cursor: %w", err) + } + cursor := &outboxCursor{seq: marks.head, pending: map[uint64]time.Time{}, server: marks.server} + now := time.Now() + // above is the lowest recent row seen so far; the walk ends at the first + // row older than the window, or at the trim mark. + above, recent := marks.head+1, true + for recent { + rows, err := p.watchDB.QueryContext(ctx, ` + SELECT seq, created_at >= UTC_TIMESTAMP(6) - INTERVAL ? MICROSECOND + FROM worker_outbox WHERE seq > ? AND seq < ? + ORDER BY seq DESC LIMIT ?`, outboxSubscribeWindow.Microseconds(), marks.trim, above, outboxBatch) + if err != nil { + return nil, fmt.Errorf("reading recent worker outbox rows: %w", err) + } + n := 0 + for recent && rows.Next() { + var seq uint64 + if err := rows.Scan(&seq, &recent); err != nil { + rows.Close() + return nil, fmt.Errorf("reading recent worker outbox rows: %w", err) + } + if above <= marks.head { + cursor.wait(seq, above, now) + } + if recent { + above = seq + } + n++ + } + rows.Close() + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("reading recent worker outbox rows: %w", err) + } + if n < outboxBatch { + break + } + } + if recent && above <= marks.head { + // Every row down to the trim mark is recent. + cursor.wait(marks.trim, above, now) + } + if len(cursor.pending) > outboxMaxPending { + return nil, fmt.Errorf("subscribing to the worker outbox: %s", resyncTooManyPending) + } + return cursor, nil +} + +type outboxRow struct { + seq uint64 + payload []byte +} + +// pollWorkerOutbox returns the rows to deliver, in seq order: pending seqs +// that have committed, then up to outboxBatch rows past the cursor, and +// advances cursor past them. full reports that more rows may follow; resync +// reports, after logging why, that the watch must close instead. +// +// The marks are read after the rows. A trim that commits between the reads +// raises the trim mark, so a row missing from the batch is always detected. +// Pending rows are read after new rows, so any write to a worker that the +// new rows carry finds that worker's earlier pending write committed too. +func (p *Persistence) pollWorkerOutbox(ctx context.Context, cursor *outboxCursor) (batch []outboxRow, full, resync bool, err error) { + if err := p.dropRolledBack(ctx, cursor, time.Now()); err != nil { + return nil, false, false, err + } + fresh, err := p.queryOutboxRows(ctx, ` + SELECT seq, payload FROM worker_outbox + WHERE seq > ? ORDER BY seq LIMIT ?`, cursor.seq, outboxBatch) + if err != nil { + return nil, false, false, fmt.Errorf("reading new worker outbox rows: %w", err) + } + var late []outboxRow + if len(cursor.pending) > 0 { + seqs := make([]any, 0, len(cursor.pending)) + for s := range cursor.pending { + seqs = append(seqs, s) + } + late, err = p.queryOutboxRows(ctx, ` + SELECT seq, payload FROM worker_outbox + WHERE seq IN `+inList(len(seqs))+` ORDER BY seq`, seqs...) + if err != nil { + return nil, false, false, fmt.Errorf("reading pending worker outbox rows: %w", err) + } + } + marks, err := p.readOutboxMarks(ctx) + if err != nil { + return nil, false, false, fmt.Errorf("reading worker outbox marks: %w", err) + } + + now := time.Now() + for _, r := range late { + delete(cursor.pending, r.seq) + } + for _, r := range fresh { + cursor.wait(cursor.seq, r.seq, now) + cursor.seq = r.seq + } + if reason, attrs := cursor.closeReason(marks); reason != "" { + slog.WarnContext(ctx, reason, attrs...) + return nil, false, true, nil + } + return slices.Concat(late, fresh), len(fresh) == outboxBatch, false, nil +} + +// dropRolledBack stops waiting on pending seqs whose writes rolled back. A +// write that took a seq holds an implicit lock on its row until it commits or +// rolls back, so a NOWAIT locking read fails while any candidate is still +// committing. When it succeeds, a candidate it does not return has no row and +// no writer, and a candidate it does return has committed. pollWorkerOutbox +// probes before reading, so a failed probe leaves the cursor unchanged and a +// committed candidate is delivered by the same poll. +func (p *Persistence) dropRolledBack(ctx context.Context, cursor *outboxCursor, now time.Time) error { + var candidates []any + for s, at := range cursor.pending { + if now.Sub(at) > outboxGapGrace { + candidates = append(candidates, s) + } + } + if len(candidates) == 0 { + return nil + } + // A scan would lock, and fail NOWAIT on, rows other writes hold. + rows, err := p.watchDB.QueryContext(ctx, ` + SELECT seq FROM worker_outbox FORCE INDEX (PRIMARY) + WHERE seq IN `+inList(len(candidates))+` + FOR SHARE NOWAIT`, candidates...) + if mysqlErrNumber(err) == errLockNowait { + return nil // a write is still committing; probe again next poll + } + if err != nil { + return fmt.Errorf("probing pending worker outbox rows: %w", err) + } + defer rows.Close() + committed := map[uint64]bool{} + for rows.Next() { + var seq uint64 + if err := rows.Scan(&seq); err != nil { + return fmt.Errorf("probing pending worker outbox rows: %w", err) + } + committed[seq] = true + } + if err := rows.Err(); err != nil { + if mysqlErrNumber(err) == errLockNowait { + return nil + } + return fmt.Errorf("probing pending worker outbox rows: %w", err) + } + for _, c := range candidates { + if s := c.(uint64); !committed[s] { + delete(cursor.pending, s) + } + } + return nil +} + +func (p *Persistence) queryOutboxRows(ctx context.Context, query string, args ...any) ([]outboxRow, error) { + rows, err := p.watchDB.QueryContext(ctx, query, args...) + if err != nil { + return nil, err + } + defer rows.Close() + var out []outboxRow + for rows.Next() { + var r outboxRow + if err := rows.Scan(&r.seq, &r.payload); err != nil { + return nil, err + } + out = append(out, r) + } + return out, rows.Err() +} diff --git a/cmd/ateapi/internal/store/atemy/outbox_test.go b/cmd/ateapi/internal/store/atemy/outbox_test.go new file mode 100644 index 0000000000..f13b8157e7 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/outbox_test.go @@ -0,0 +1,786 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "fmt" + "slices" + "sync" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/testing/protocmp" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +func workerPayload(t *testing.T, eventType store.WorkerEventType, worker *ateapipb.Worker) []byte { + t.Helper() + payload, err := storesql.MarshalWorkerEvent(eventType, worker) + if err != nil { + t.Fatalf("marshaling event for %q: %v", worker.GetMetadata().GetName(), err) + } + return payload +} + +// insertOutboxRow appends payload on q, stamped age before now, the way +// writeAndAppendEvent does, and returns its seq. +func insertOutboxRow(t *testing.T, q querier, payload []byte, age time.Duration) uint64 { + t.Helper() + res, err := q.ExecContext(t.Context(), ` + INSERT INTO worker_outbox (created_at, payload) + VALUES (UTC_TIMESTAMP(6) - INTERVAL ? MICROSECOND, ?)`, age.Microseconds(), payload) + if err != nil { + t.Fatalf("appending outbox row: %v", err) + } + seq, err := res.LastInsertId() + if err != nil { + t.Fatalf("reading appended seq: %v", err) + } + return uint64(seq) +} + +// beginOutboxWrite opens a transaction that appends an event for worker and +// stays open, as a worker write between its append and its commit does. +func beginOutboxWrite(t *testing.T, p *Persistence, eventType store.WorkerEventType, worker *ateapipb.Worker) (*sql.Tx, uint64) { + t.Helper() + tx, err := p.db.BeginTx(t.Context(), &sql.TxOptions{Isolation: sql.LevelReadCommitted}) + if err != nil { + t.Fatalf("BeginTx failed: %v", err) + } + t.Cleanup(func() { tx.Rollback() }) //nolint:errcheck // no-op once committed + return tx, insertOutboxRow(t, tx, workerPayload(t, eventType, worker), 0) +} + +// appendRawEvents commits n outbox rows carrying payload in one statement, +// stamped age before now, and returns the first and last seq. +func appendRawEvents(t *testing.T, p *Persistence, n int, payload []byte, age time.Duration) (first, last uint64) { + t.Helper() + ctx := t.Context() + conn, err := p.db.Conn(ctx) + if err != nil { + t.Fatalf("acquiring connection: %v", err) + } + defer conn.Close() + if _, err := conn.ExecContext(ctx, `SET SESSION cte_max_recursion_depth = ?`, n+1); err != nil { + t.Fatalf("raising recursion depth: %v", err) + } + if _, err := conn.ExecContext(ctx, ` + INSERT INTO worker_outbox (created_at, payload) + WITH RECURSIVE r (i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM r WHERE i < ?) + SELECT UTC_TIMESTAMP(6) - INTERVAL ? MICROSECOND, ? FROM r`, + n, age.Microseconds(), payload); err != nil { + t.Fatalf("appending outbox rows: %v", err) + } + if err := conn.QueryRowContext(ctx, `SELECT LAST_INSERT_ID()`).Scan(&first); err != nil { + t.Fatalf("reading first seq: %v", err) + } + return first, first + uint64(n) - 1 +} + +// outboxHead returns the greatest stored seq. +func outboxHead(t *testing.T, p *Persistence) uint64 { + t.Helper() + var seq uint64 + if err := p.db.QueryRowContext(t.Context(), `SELECT COALESCE(MAX(seq), 0) FROM worker_outbox`).Scan(&seq); err != nil { + t.Fatalf("reading outbox head: %v", err) + } + return seq +} + +func trimMark(t *testing.T, p *Persistence) uint64 { + t.Helper() + var seq uint64 + if err := p.db.QueryRowContext(t.Context(), `SELECT seq FROM worker_outbox_trim WHERE id = 1`).Scan(&seq); err != nil { + t.Fatalf("reading trim mark: %v", err) + } + return seq +} + +func outboxSeqs(t *testing.T, p *Persistence) []uint64 { + t.Helper() + rows, err := p.db.QueryContext(t.Context(), `SELECT seq FROM worker_outbox ORDER BY seq`) + if err != nil { + t.Fatalf("reading outbox: %v", err) + } + defer rows.Close() + var seqs []uint64 + for rows.Next() { + var seq uint64 + if err := rows.Scan(&seq); err != nil { + t.Fatalf("scanning outbox row: %v", err) + } + seqs = append(seqs, seq) + } + if err := rows.Err(); err != nil { + t.Fatalf("reading outbox: %v", err) + } + return seqs +} + +func seqRange(first, last uint64) []uint64 { + var seqs []uint64 + for seq := first; seq <= last; seq++ { + seqs = append(seqs, seq) + } + return seqs +} + +// receive returns the next event, failing if the watch closes or stays quiet. +func receive(t *testing.T, watch *store.WorkerWatch) store.WorkerEvent { + t.Helper() + select { + case event, ok := <-watch.Events: + if !ok { + t.Fatal("watch closed, want an event") + } + return event + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for an event") + } + return store.WorkerEvent{} +} + +// requireClosedNext fails unless the watch's next signal is its close. +func requireClosedNext(t *testing.T, watch *store.WorkerWatch, why string) { + t.Helper() + select { + case event, ok := <-watch.Events: + if ok { + t.Fatalf("delivered event for %q, want the watch to close: %s", event.Worker.GetMetadata().GetName(), why) + } + case <-time.After(5 * time.Second): + t.Fatalf("watch stayed open: %s", why) + } +} + +func watchWorkers(t *testing.T, p *Persistence) *store.WorkerWatch { + t.Helper() + watch, err := p.WatchWorkers(t.Context()) + if err != nil { + t.Fatalf("WatchWorkers failed: %v", err) + } + t.Cleanup(watch.Close) + return watch +} + +// TestWorkerEvent_OnlyAfterCommit proves a worker write's outbox row shares +// the write's transaction: a rolled-back write delivers nothing and the seq it +// took, a gap the watcher waits on, does not close the watch. +func TestWorkerEvent_OnlyAfterCommit(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + // A replica's watcher sees only polled rows, so a rolled-back row that + // leaked would arrive from the outbox. + watch := watchWorkers(t, newReplica(t, s)) + + worker := newTestWorker("6e4d2f81-b3a9-4c05-8e72-1f9d4a0c7b63") + tx, rolledBack := beginOutboxWrite(t, s, store.WorkerEventCreated, newTestWorker("rolled-back")) + if err := tx.Rollback(); err != nil { + t.Fatalf("Rollback failed: %v", err) + } + + created, err := s.CreateWorker(ctx, worker) + if err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if created.GetMetadata().GetName() == "" || outboxHead(t, s) <= rolledBack { + t.Fatalf("committed write did not take a seq past the rolled-back %d", rolledBack) + } + event := receive(t, watch) + if event.Type != store.WorkerEventCreated { + t.Errorf("event type = %v, want WorkerEventCreated", event.Type) + } + if diff := cmp.Diff(created, event.Worker, protocmp.Transform()); diff != "" { + t.Errorf("event worker mismatch (-want +got):\n%s", diff) + } + + // The rolled-back seq does not close the watch. + if _, err := s.CreateWorker(ctx, newTestWorker("after-rollback")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if got := receive(t, watch).Worker.GetMetadata().GetName(); got != "after-rollback" { + t.Errorf("delivered %q, want after-rollback", got) + } +} + +// TestWatchWorkers_OutOfOrderCommitNotSkipped commits seq N+1 while seq N is +// still open. The watcher delivers N+1, remembers N as pending, and delivers +// N late once it commits, without losing it or closing the watch. +func TestWatchWorkers_OutOfOrderCommitNotSkipped(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + watch := watchWorkers(t, newReplica(t, s)) + + txA, seqA := beginOutboxWrite(t, s, store.WorkerEventCreated, newTestWorker("first-seq-late-commit")) + if _, err := s.CreateWorker(ctx, newTestWorker("second-seq-early-commit")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if got := outboxHead(t, s); got <= seqA { + t.Fatalf("early commit took seq %d, want one past the open %d", got, seqA) + } + if got := receive(t, watch).Worker.GetMetadata().GetName(); got != "second-seq-early-commit" { + t.Fatalf("delivered %q, want second-seq-early-commit", got) + } + + if err := txA.Commit(); err != nil { + t.Fatalf("committing the first writer: %v", err) + } + if got := receive(t, watch).Worker.GetMetadata().GetName(); got != "first-seq-late-commit" { + t.Fatalf("delivered %q, want the late first-seq-late-commit", got) + } + if _, err := s.CreateWorker(ctx, newTestWorker("after-late-commit")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if got := receive(t, watch).Worker.GetMetadata().GetName(); got != "after-late-commit" { + t.Errorf("delivered %q, want after-late-commit", got) + } +} + +// Writes to one worker take its row lock in turn, so the later write's seq is +// assigned after the earlier one commits. When the earlier write is pending, +// both arrive in write order, the earlier one delivered late in the same poll. +func TestWatchWorkers_KeepsPerWorkerOrder(t *testing.T) { + ipsOf := func(t *testing.T, watch *store.WorkerWatch, name string, n int) []string { + t.Helper() + var ips []string + for len(ips) < n { + event := receive(t, watch) + if event.Worker.GetMetadata().GetName() == name { + ips = append(ips, event.Worker.GetIps()...) + } + } + return ips + } + s := setupMySQLPersistence(t) + ctx := t.Context() + created, err := s.CreateWorker(ctx, newTestWorker("ordered")) + if err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + watch := watchWorkers(t, newReplica(t, s)) + + // The first write holds the worker's row lock past its append. + first := proto.CloneOf(created) + first.Ips = []string{"10.0.0.1"} + tx, err := s.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadCommitted}) + if err != nil { + t.Fatalf("BeginTx failed: %v", err) + } + defer tx.Rollback() //nolint:errcheck // no-op once committed + if _, err := tx.ExecContext(ctx, `UPDATE workers SET version = version WHERE name = 'ordered'`); err != nil { + t.Fatalf("locking the worker: %v", err) + } + insertOutboxRow(t, tx, workerPayload(t, store.WorkerEventUpdated, first), 0) + + // Another worker commits a later seq, so the first write's seq is + // pending. + if _, err := s.CreateWorker(ctx, newTestWorker("other")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if got := receive(t, watch).Worker.GetMetadata().GetName(); got != "other" { + t.Fatalf("delivered %q, want other", got) + } + + second := make(chan error, 1) + go func() { + stored, err := s.GetWorker(ctx, "ordered") + if err == nil { + _, err = s.UpdateWorker(ctx, "ordered", store.PreconditionFrom(stored), func(w *ateapipb.Worker) error { + w.Ips = []string{"10.0.0.2"} + return nil + }) + } + second <- err + }() + waitForLockWait(t, second) + if err := tx.Commit(); err != nil { + t.Fatalf("committing the first write: %v", err) + } + if err := <-second; err != nil { + t.Fatalf("second UpdateWorker failed: %v", err) + } + if got, want := ipsOf(t, watch, "ordered", 2), []string{"10.0.0.1", "10.0.0.2"}; !slices.Equal(got, want) { + t.Errorf("updates delivered in order %q, want %q", got, want) + } +} + +// A write in flight when a watch subscribes has a seq below a write that +// committed before the watch, so the watch starts with it pending and +// delivers it once it commits. Rows already committed are not replayed. +func TestWatchWorkers_DeliversAWriteInFlightAtSubscribe(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + // An earlier row, outside the gap window, as on a quiet system. + insertOutboxRow(t, s.db, workerPayload(t, store.WorkerEventCreated, newTestWorker("long-ago")), 2*outboxSubscribeWindow) + + tx, inFlight := beginOutboxWrite(t, s, store.WorkerEventCreated, newTestWorker("in-flight")) + if _, err := s.CreateWorker(ctx, newTestWorker("committed-before-subscribe")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if got := outboxHead(t, s); got <= inFlight { + t.Fatalf("committed write took seq %d, want one past the in-flight %d", got, inFlight) + } + watch := watchWorkers(t, newReplica(t, s)) + + if err := tx.Commit(); err != nil { + t.Fatalf("Commit failed: %v", err) + } + if _, err := s.CreateWorker(ctx, newTestWorker("after-subscribe")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + got := []string{receive(t, watch).Worker.GetMetadata().GetName(), receive(t, watch).Worker.GetMetadata().GetName()} + if !slices.Contains(got, "in-flight") || !slices.Contains(got, "after-subscribe") { + t.Errorf("delivered %q, want in-flight and after-subscribe, and nothing written before the watch", got) + } +} + +// TestWorkerEvents_OneRowPerWrite pins the invariant the seq cursor rests on: +// each committed worker write appends exactly one outbox row, and a write +// that announces nothing appends none. +func TestWorkerEvents_OneRowPerWrite(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + + if _, err := s.CreateWorker(ctx, newTestWorker("one-row-worker")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + for i := range 10 { + stored, err := s.GetWorker(ctx, "one-row-worker") + if err != nil { + t.Fatalf("GetWorker failed: %v", err) + } + if _, err := s.UpdateWorker(ctx, "one-row-worker", store.PreconditionFrom(stored), func(*ateapipb.Worker) error { + return nil + }); err != nil { + t.Fatalf("UpdateWorker %d failed: %v", i, err) + } + } + if released, err := s.ReleaseActorFromWorker(ctx, "one-row-worker", "not-hosted"); err != nil || released != nil { + t.Fatalf("ReleaseActorFromWorker of an unhosted actor = %v, %v; want nil, nil", released, err) + } + if _, err := s.DeleteWorker(ctx, "one-row-worker", store.DeletePreconditions{}); err != nil { + t.Fatalf("DeleteWorker failed: %v", err) + } + + const writes = 12 + seqs := outboxSeqs(t, s) + if len(seqs) != writes { + t.Fatalf("%d outbox rows, want %d", len(seqs), writes) + } + if diff := cmp.Diff(seqRange(seqs[0], seqs[0]+writes-1), seqs); diff != "" { + t.Errorf("outbox seqs are not contiguous (-want +got):\n%s", diff) + } +} + +// Retention deletes rows oldest seq first and stops at the first row still +// within retention, so the trim mark always bounds a contiguous deleted +// prefix, even if a later row carries an older timestamp. +func TestTrimWorkerOutbox_TrimsOnlyTheExpiredPrefix(t *testing.T) { + testTrimsOnlyTheExpiredPrefix(t, setupMySQLPersistence(t)) +} + +func testTrimsOnlyTheExpiredPrefix(t *testing.T, s *Persistence) { + t.Helper() + ctx := t.Context() + payload := []byte("payload") + insertOutboxRow(t, s.db, payload, time.Hour) + secondOld := insertOutboxRow(t, s.db, payload, time.Hour) + fresh := insertOutboxRow(t, s.db, payload, 0) + lateOld := insertOutboxRow(t, s.db, payload, time.Hour) + + for range 2 { + if err := s.trimWorkerOutboxOlderThan(ctx, 30*time.Minute); err != nil { + t.Fatalf("trimWorkerOutboxOlderThan failed: %v", err) + } + if diff := cmp.Diff([]uint64{fresh, lateOld}, outboxSeqs(t, s)); diff != "" { + t.Errorf("outbox seqs after trim (-want +got):\n%s", diff) + } + if got := trimMark(t, s); got != secondOld { + t.Errorf("trim mark = %d, want %d", got, secondOld) + } + } + + // The production retention keeps rows younger than outboxRetentionAge. + if err := s.trimWorkerOutboxOlderThan(ctx, outboxRetentionAge); err != nil { + t.Fatalf("trimWorkerOutboxOlderThan failed: %v", err) + } + if diff := cmp.Diff([]uint64{fresh, lateOld}, outboxSeqs(t, s)); diff != "" { + t.Errorf("outbox seqs after production retention (-want +got):\n%s", diff) + } +} + +// A database that loses committed writes, as a restore from an older backup +// does, leaves the outbox behind a watcher's cursor. New writes would reuse +// seqs the cursor has passed, so the watcher must close rather than skip them. +func TestWatchWorkers_ClosesWhenTheOutboxMovesBehindTheCursor(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + watch := watchWorkers(t, newReplica(t, s)) + + if _, err := s.CreateWorker(ctx, newTestWorker("lost-worker")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if got := receive(t, watch).Worker.GetMetadata().GetName(); got != "lost-worker" { + t.Fatalf("delivered %q, want lost-worker", got) + } + if _, err := s.db.ExecContext(ctx, `DELETE FROM worker_outbox`); err != nil { + t.Fatalf("dropping the outbox rows: %v", err) + } + requireClosedNext(t, watch, "the outbox moved behind the cursor") +} + +// TestWatchWorkers_ClosesOnCorruptPayload pins close-over-skip: a payload +// that fails to decode must close the watch rather than advance past it. +func TestWatchWorkers_ClosesOnCorruptPayload(t *testing.T) { + s := setupMySQLPersistence(t) + watch := watchWorkers(t, s) + insertOutboxRow(t, s.db, []byte{0xff, 0xde, 0xad}, 0) + requireClosedNext(t, watch, "a corrupt payload was skipped") +} + +// A huge gap stops one past the cap, enough for closeReason to see it. +func TestOutboxCursor_Wait(t *testing.T) { + c := &outboxCursor{pending: map[uint64]time.Time{}} + c.wait(0, outboxMaxPending*10, time.Now()) + if got := len(c.pending); got != outboxMaxPending+1 { + t.Errorf("pending after a huge gap = %d seqs, want %d", got, outboxMaxPending+1) + } +} + +func TestOutboxCursor_CloseReason(t *testing.T) { + healthy := outboxMarks{trim: 5, head: 20, server: "a"} + tooMany := map[uint64]time.Time{} + for s := range uint64(outboxMaxPending + 1) { + tooMany[100+s] = time.Time{} + } + for _, tc := range []struct { + name string + pending map[uint64]time.Time + marks outboxMarks + closes bool + }{ + {"healthy", map[uint64]time.Time{8: {}}, healthy, false}, + {"head at the cursor", nil, outboxMarks{trim: 5, head: 10, server: "a"}, false}, + {"trim at the cursor", nil, outboxMarks{trim: 10, head: 10, server: "a"}, false}, + {"server changed", nil, outboxMarks{trim: 5, head: 20, server: "b"}, true}, + {"head behind the cursor", nil, outboxMarks{trim: 5, head: 9, server: "a"}, true}, + {"trim past the cursor", nil, outboxMarks{trim: 11, head: 20, server: "a"}, true}, + {"too many pending", tooMany, healthy, true}, + {"pending at the trim mark", map[uint64]time.Time{5: {}}, healthy, true}, + } { + t.Run(tc.name, func(t *testing.T) { + pending := tc.pending + if pending == nil { + pending = map[uint64]time.Time{} + } + c := &outboxCursor{seq: 10, pending: pending, server: "a"} + if got, _ := c.closeReason(tc.marks); (got != "") != tc.closes { + t.Errorf("closeReason(%+v) = %q, want closing: %t", tc.marks, got, tc.closes) + } + }) + } +} + +// A rolled-back write leaves a seq that never commits. The watcher drops it +// once a NOWAIT probe finds neither a row nor a writer holding it, and keeps +// every pending seq while some write is still committing, however long that +// takes, as atepg's xmin fence waits. +func TestPollWorkerOutbox_DropsOnlyRolledBackSeqs(t *testing.T) { + testDropsOnlyRolledBackSeqs(t, setupMySQLPersistence(t)) +} + +func testDropsOnlyRolledBackSeqs(t *testing.T, s *Persistence) { + t.Helper() + ctx := t.Context() + rolledBackTx, rolledBack := beginOutboxWrite(t, s, store.WorkerEventCreated, newTestWorker("rolled-back")) + if err := rolledBackTx.Rollback(); err != nil { + t.Fatalf("Rollback failed: %v", err) + } + inFlight, inFlightSeq := beginOutboxWrite(t, s, store.WorkerEventCreated, newTestWorker("in-flight")) + defer inFlight.Rollback() //nolint:errcheck // no-op once committed + if _, err := s.CreateWorker(ctx, newTestWorker("committed")); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + + cursor, err := s.subscribeWorkerOutbox(ctx) + if err != nil { + t.Fatalf("subscribeWorkerOutbox failed: %v", err) + } + if _, ok := cursor.pending[rolledBack]; !ok { + t.Fatalf("pending = %v, want the rolled-back seq %d", cursor.pending, rolledBack) + } + age := func() { + for seq := range cursor.pending { + cursor.pending[seq] = time.Now().Add(-2 * outboxGapGrace) + } + } + + age() + if _, _, resync, err := s.pollWorkerOutbox(ctx, cursor); err != nil || resync { + t.Fatalf("poll with a write in flight = resync %t, %v", resync, err) + } + for _, seq := range []uint64{rolledBack, inFlightSeq} { + if _, ok := cursor.pending[seq]; !ok { + t.Errorf("seq %d left pending while a write is in flight; pending = %v", seq, cursor.pending) + } + } + + if err := inFlight.Commit(); err != nil { + t.Fatalf("Commit failed: %v", err) + } + age() + batch, _, resync, err := s.pollWorkerOutbox(ctx, cursor) + if err != nil || resync { + t.Fatalf("poll after the commit = resync %t, %v", resync, err) + } + if len(batch) != 1 || batch[0].seq != inFlightSeq { + t.Errorf("delivered %v, want the late seq %d", batch, inFlightSeq) + } + if len(cursor.pending) != 0 { + t.Errorf("pending = %v, want the rolled-back seq dropped", cursor.pending) + } +} + +// Two replicas running retention at once: one drains every batch while the +// other skips the locked trim row, and both succeed. +func TestTrimWorkerOutbox_ConcurrentPassesAreHarmless(t *testing.T) { + s := setupMySQLPersistence(t) + replica := newReplica(t, s) + _, last := appendRawEvents(t, s, outboxBatch*2+5, []byte("payload"), time.Hour) + + errs := make([]error, 2) + var wg sync.WaitGroup + for i, p := range []*Persistence{s, replica} { + wg.Go(func() { errs[i] = p.trimWorkerOutboxOlderThan(t.Context(), time.Minute) }) + } + wg.Wait() + for i, err := range errs { + if err != nil { + t.Errorf("concurrent pass %d returned error: %v", i, err) + } + } + if seqs := outboxSeqs(t, s); len(seqs) != 0 { + t.Errorf("%d rows left after both passes, want 0", len(seqs)) + } + if got := trimMark(t, s); got != last { + t.Errorf("trim mark = %d, want %d", got, last) + } +} + +// Retention and worker writes lock disjoint rows (the trim row and old outbox +// rows against new outbox rows), so running them at once must neither +// deadlock nor fail a write. +func TestTrimWorkerOutbox_ConcurrentWritersDoNotDeadlock(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + + writerCtx, stopWriters := context.WithCancel(ctx) + defer stopWriters() + var wg sync.WaitGroup + writerErrs := make(chan error, 4) + for g := range 4 { + wg.Go(func() { + for i := 0; writerCtx.Err() == nil; i++ { + if _, err := s.CreateWorker(writerCtx, newTestWorker(fmt.Sprintf("trim-writer-%d-%d", g, i))); err != nil && writerCtx.Err() == nil { + writerErrs <- fmt.Errorf("writer %d iteration %d: %w", g, i, err) + return + } + } + }) + } + for i := range 20 { + if err := s.trimWorkerOutboxOlderThan(ctx, 0); err != nil { + t.Errorf("trim pass %d failed under concurrent writers: %v", i, err) + } + } + stopWriters() + wg.Wait() + close(writerErrs) + for err := range writerErrs { + t.Errorf("concurrent writer failed: %v", err) + } + mark := trimMark(t, s) + if seqs := outboxSeqs(t, s); len(seqs) > 0 && seqs[0] <= mark { + t.Errorf("row %d survived at or below the trim mark %d", seqs[0], mark) + } + if mark == 0 { + t.Error("no trim pass deleted anything") + } +} + +// TestWatchWorkers_ClosesWhenTrimmedPastCursor stalls the consumer so the +// poller blocks partway through its first batch, then retention deletes every +// row. On resume the watcher delivers the batch it read and closes, because +// rows it never read were deleted. +func TestWatchWorkers_ClosesWhenTrimmedPastCursor(t *testing.T) { + s := setupMySQLPersistence(t) + watch := watchWorkers(t, newReplica(t, s)) + + appendRawEvents(t, s, outboxBatch+100, workerPayload(t, store.WorkerEventUpdated, newTestWorker("lagging-worker")), 0) + deadline := time.Now().Add(10 * time.Second) + for len(watch.Events) < cap(watch.Events) { + if time.Now().After(deadline) { + t.Fatalf("watch buffered %d of %d events, want it full", len(watch.Events), cap(watch.Events)) + } + time.Sleep(10 * time.Millisecond) + } + if err := s.trimWorkerOutboxOlderThan(t.Context(), 0); err != nil { + t.Fatalf("trimWorkerOutboxOlderThan failed: %v", err) + } + if seqs := outboxSeqs(t, s); len(seqs) != 0 { + t.Fatalf("%d rows survived the trim, want 0", len(seqs)) + } + + delivered := 0 + for { + select { + case _, ok := <-watch.Events: + if !ok { + if delivered != outboxBatch { + t.Errorf("delivered %d events before closing, want the %d of the batch read before the trim", delivered, outboxBatch) + } + return + } + delivered++ + case <-time.After(5 * time.Second): + t.Fatalf("watch stayed open after %d events; rows it never read were trimmed", delivered) + } + } +} + +// TestWatchWorkers_ClosesAfterPersistentPollFailure pins the loss signal for +// polling outages: a persistent failure must close the channel within +// pollFailureCloseAfter so consumers stop serving a frozen fleet view. +func TestWatchWorkers_ClosesAfterPersistentPollFailure(t *testing.T) { + requireDB(t) + ctx := t.Context() + // Connect gives the watcher its own pool, so closing it simulates an + // outage without touching the shared pool. + p, err := Connect(ctx, ConnectConfig{ReadWriteDSN: containerDSN, OwnerDSN: containerDSN}) + if err != nil { + t.Fatalf("Connect failed: %v", err) + } + defer p.DB().Close() + defer p.Close() + p.pollFailureCloseAfter = 300 * time.Millisecond + + watch, err := p.WatchWorkers(ctx) + if err != nil { + t.Fatalf("WatchWorkers failed: %v", err) + } + defer watch.Close() + p.watchDB.Close() + + requireClosedNext(t, watch, "polling failed persistently") +} + +// TestLocalPublishReachesWatchers pins the local fast path: every worker +// event is on an active watcher's channel by the time the write call +// returns, carrying the committed state. Each read is non-blocking, so only +// the commit-time publish can satisfy it. +func TestLocalPublishReachesWatchers(t *testing.T) { + p := setupMySQLPersistence(t) + ctx := t.Context() + watch := watchWorkers(t, p) + + // nextEvent takes the next buffered event of type want, skipping outbox + // copies of earlier writes. + nextEvent := func(what string, want store.WorkerEventType) store.WorkerEvent { + t.Helper() + for { + select { + case ev, ok := <-watch.Events: + if !ok { + t.Fatalf("after %s: watch closed", what) + } + if ev.Type == want { + return ev + } + default: + t.Fatalf("after %s: no %v on the watch channel; the commit-time publish did not reach the watcher", what, want) + return store.WorkerEvent{} + } + } + } + + created, err := p.CreateWorker(ctx, newTestWorker("local-publish-worker")) + if err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + ev := nextEvent("create", store.WorkerEventCreated) + if got, want := ev.Worker.GetMetadata().GetVersion(), created.GetMetadata().GetVersion(); got != want { + t.Errorf("created event version = %d, want committed version %d", got, want) + } + + updated, err := p.UpdateWorker(ctx, created.GetMetadata().GetName(), store.PreconditionFrom(created), func(toUpdate *ateapipb.Worker) error { + toUpdate.Ips = []string{"10.0.0.9"} + return nil + }) + if err != nil { + t.Fatalf("UpdateWorker failed: %v", err) + } + ev = nextEvent("update", store.WorkerEventUpdated) + if got, want := ev.Worker.GetMetadata().GetVersion(), updated.GetMetadata().GetVersion(); got != want { + t.Errorf("updated event version = %d, want committed version %d", got, want) + } + if !slices.Equal(ev.Worker.GetIps(), []string{"10.0.0.9"}) { + t.Errorf("updated event carries Ips %q, want the committed mutation", ev.Worker.GetIps()) + } + if ev.Worker == updated { + t.Error("locally published Worker aliases the caller's returned Worker; it must be a copy") + } + + if _, err := p.DeleteWorker(ctx, created.GetMetadata().GetName(), store.DeletePreconditions{}); err != nil { + t.Fatalf("DeleteWorker failed: %v", err) + } + ev = nextEvent("delete", store.WorkerEventDeleted) + if ev.Worker.GetMetadata().GetName() != created.GetMetadata().GetName() { + t.Errorf("deleted event names worker %q, want %q", ev.Worker.GetMetadata().GetName(), created.GetMetadata().GetName()) + } +} + +// TestLocalPublishSurvivesWatchClose covers the close race the watcher +// registry exists to prevent: a write publishing concurrently with a watch +// shutting down must not send on a closed channel. +func TestLocalPublishSurvivesWatchClose(t *testing.T) { + p := setupMySQLPersistence(t) + ctx := t.Context() + for i := range 20 { + watch, err := p.WatchWorkers(ctx) + if err != nil { + t.Fatalf("WatchWorkers failed: %v", err) + } + done := make(chan struct{}) + go func() { + defer close(done) + watch.Close() + }() + w, err := p.CreateWorker(ctx, newTestWorker(fmt.Sprintf("close-race-worker-%d", i))) + if err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + if _, err := p.DeleteWorker(ctx, w.GetMetadata().GetName(), store.DeletePreconditions{}); err != nil { + t.Fatalf("DeleteWorker failed: %v", err) + } + <-done + } +} diff --git a/cmd/ateapi/internal/store/atemy/relations.go b/cmd/ateapi/internal/store/atemy/relations.go new file mode 100644 index 0000000000..6d83c2bab9 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/relations.go @@ -0,0 +1,105 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// The helpers below stand in for the foreign keys atepg declares. A child +// insert takes a shared lock on its parent row, so the parent cannot be +// deleted until the insert commits. A parent delete takes an exclusive lock +// first, so it waits for in-flight child inserts and then sees their rows. + +// lockAtespace takes a shared lock on an atespace for the rest of tx. Returns +// ErrFailedPrecondition if the atespace does not exist. +func lockAtespace(ctx context.Context, tx *sql.Tx, name string) error { + return lockParent(ctx, tx, `SELECT 1 FROM atespaces WHERE name = ? FOR SHARE`, name) +} + +func lockParent(ctx context.Context, tx *sql.Tx, query string, args ...any) error { + var found int + if err := tx.QueryRowContext(ctx, query, args...).Scan(&found); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return store.ErrFailedPrecondition + } + return fmt.Errorf("locking parent row: %w", err) + } + return nil +} + +// insertChild inserts a row whose parent lock must hold until commit, +// reporting a taken key as ErrAlreadyExists. what names the row in errors. +func (p *Persistence) insertChild(ctx context.Context, what string, lock func(tx *sql.Tx) error, insert string, args ...any) error { + return inTx(ctx, p.db, func(tx *sql.Tx) error { + if err := lock(tx); err != nil { + return err + } + if _, err := tx.ExecContext(ctx, insert, args...); err != nil { + if isUniqueViolation(err) { + return store.ErrAlreadyExists + } + return fmt.Errorf("inserting %s: %w", what, err) + } + return nil + }) +} + +// deleteLockedRow locks the row of table that where selects, checks +// precondition against it, deletes it, and returns its columns. Returns +// ErrNotFound if no row matches. table and where are trusted SQL fragments. +func deleteLockedRow(ctx context.Context, tx *sql.Tx, table, where string, precondition store.DeletePreconditions, args ...any) (uid string, version int64, protoBytes []byte, err error) { + if err := tx.QueryRowContext(ctx, `SELECT uid, version, proto FROM `+table+` WHERE `+where+` FOR UPDATE`, args...).Scan(&uid, &version, &protoBytes); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return "", 0, nil, store.ErrNotFound + } + return "", 0, nil, fmt.Errorf("locking %s row for delete: %w", table, err) + } + if err := precondition.Check(&ateapipb.ResourceMetadata{Uid: uid, Version: version}); err != nil { + return "", 0, nil, err + } + if _, err := tx.ExecContext(ctx, `DELETE FROM `+table+` WHERE `+where, args...); err != nil { + return "", 0, nil, fmt.Errorf("deleting %s row: %w", table, err) + } + return uid, version, protoBytes, nil +} + +// updateGuarded writes a new version of a row only if it still carries the +// uid and version the caller read, reporting a lost race as +// ErrVersionConflict. what names the row in errors. +func updateGuarded(ctx context.Context, q querier, what, update string, args ...any) error { + res, err := q.ExecContext(ctx, update, args...) + if err != nil { + return fmt.Errorf("updating %s: %w", what, err) + } + n, err := res.RowsAffected() + if err != nil { + return fmt.Errorf("updating %s: %w", what, err) + } + switch n { + case 0: + return store.ErrVersionConflict + case 1: + return nil + default: + return fmt.Errorf("updating %s affected %d rows, want 1", what, n) + } +} diff --git a/cmd/ateapi/internal/store/atemy/relations_test.go b/cmd/ateapi/internal/store/atemy/relations_test.go new file mode 100644 index 0000000000..e8457f7ee6 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/relations_test.go @@ -0,0 +1,331 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Tests for the transactions that stand in for foreign keys (relations.go): +// a child insert and a parent delete that race must never leave a child row +// without its parent. + +package atemy + +import ( + "errors" + "fmt" + "slices" + "sync" + "testing" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// requireNoOrphans fails if any row outlives the parent a foreign key would +// tie it to. +func requireNoOrphans(t *testing.T, p *Persistence) { + t.Helper() + for what, query := range map[string]string{ + "actors": `SELECT COUNT(*) FROM actors c + WHERE NOT EXISTS (SELECT 1 FROM atespaces p WHERE p.name = c.atespace)`, + "actor templates": `SELECT COUNT(*) FROM actor_templates c + WHERE NOT EXISTS (SELECT 1 FROM atespaces p WHERE p.name = c.atespace)`, + "tags": `SELECT COUNT(*) FROM tags c + WHERE NOT EXISTS (SELECT 1 FROM atespaces p WHERE p.name = c.atespace)`, + "atespace access policies": `SELECT COUNT(*) FROM atespace_access_policies c + WHERE NOT EXISTS (SELECT 1 FROM atespaces p WHERE p.name = c.atespace_name)`, + "egress policies": `SELECT COUNT(*) FROM actor_egress_policies c + WHERE NOT EXISTS (SELECT 1 FROM actors p WHERE p.atespace = c.atespace AND p.name = c.actor_name)`, + "worker assignments": `SELECT COUNT(*) FROM worker_assignments c + WHERE NOT EXISTS (SELECT 1 FROM workers p WHERE p.name = c.worker_name)`, + "atespace tuples": `SELECT COUNT(*) FROM tuple c + WHERE c.object_type = 'atespace' + AND NOT EXISTS (SELECT 1 FROM atespaces p WHERE p.name = c.object_id COLLATE utf8mb4_0900_bin)`, + } { + if n := countRows(t, p, query); n != 0 { + t.Errorf("%d %s outlive their parent", n, what) + } + } +} + +// raceOutcome checks the errors of one parent delete racing child creates: +// either the delete won and every create was refused, or a create won and the +// delete was refused. It reports whether the delete won. +func raceOutcome(t *testing.T, deleteErr error, createErrs []error) bool { + t.Helper() + created := 0 + for i, err := range createErrs { + switch { + case err == nil: + created++ + case !errors.Is(err, store.ErrFailedPrecondition): + t.Errorf("create %d = %v, want nil or ErrFailedPrecondition", i, err) + } + } + switch { + case deleteErr == nil && created != 0: + t.Errorf("parent delete succeeded, but %d children were created", created) + case errors.Is(deleteErr, store.ErrFailedPrecondition) && created == 0: + t.Errorf("parent delete was refused for children, but none was created") + case deleteErr != nil && !errors.Is(deleteErr, store.ErrFailedPrecondition): + t.Errorf("parent delete = %v, want nil or ErrFailedPrecondition", deleteErr) + } + return deleteErr == nil +} + +// race runs fns concurrently, releasing them together, and waits for all. +func race(fns ...func()) { + start := make(chan struct{}) + var wg sync.WaitGroup + for _, fn := range fns { + wg.Go(func() { + <-start + fn() + }) + } + close(start) + wg.Wait() +} + +func TestDeleteAtespace_RacingChildCreatesLeaveNoOrphans(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + const iterations, creators = 10, 12 + deletesWon := 0 + for i := range iterations { + atespace := fmt.Sprintf("race-%d", i) + createTestAtespace(t, s, atespace) + + createErrs := make([]error, creators) + var deleteErr error + racers := []func(){func() { + _, deleteErr = s.DeleteAtespace(ctx, atespace, store.DeletePreconditions{}) + }} + for c := range creators { + racers = append(racers, func() { + name := fmt.Sprintf("child-%d", c) + meta := &ateapipb.ResourceMetadata{Atespace: atespace, Name: name} + switch c % 3 { + case 0: + _, createErrs[c] = s.CreateActor(ctx, newTestActor(atespace, name)) + case 1: + _, createErrs[c] = s.CreateActorTemplate(ctx, &ateapipb.ActorTemplate{Metadata: meta}) + default: + _, createErrs[c] = s.CreateTag(ctx, &ateapipb.Tag{Metadata: meta, Status: &ateapipb.TagStatus{StorageLocation: "gs://bucket"}}) + } + }) + } + race(racers...) + if raceOutcome(t, deleteErr, createErrs) { + deletesWon++ + } + } + t.Logf("the delete won %d of %d races", deletesWon, iterations) + requireNoOrphans(t, s) +} + +func TestDeleteAtespace_RacingAccessPolicyCreateLeavesNoOrphans(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + policy := &ateapipb.AccessPolicy{Bindings: []*ateapipb.Binding{{Role: authz.RoleEditor, Members: []string{"user:bob"}}}} + for i := range 10 { + atespace := fmt.Sprintf("policy-race-%d", i) + createTestAtespace(t, s, atespace) + + var createErr, deleteErr error + race(func() { + _, createErr = s.CreateAtespaceAccessPolicy(ctx, atespace, policy) + }, func() { + _, deleteErr = s.DeleteAtespace(ctx, atespace, store.DeletePreconditions{}) + }) + // An access policy does not block its atespace's deletion; it goes + // with it. + if deleteErr != nil { + t.Errorf("DeleteAtespace = %v, want nil", deleteErr) + } + if createErr != nil && !errors.Is(createErr, store.ErrFailedPrecondition) { + t.Errorf("CreateAtespaceAccessPolicy = %v, want nil or ErrFailedPrecondition", createErr) + } + } + requireNoOrphans(t, s) +} + +// UpdateAtespaceAccessPolicy locks only the policy row, not the atespace, so +// its tuple writes must still land before DeleteAtespace reads the tuples to +// remove, or fail because the policy is gone. +func TestDeleteAtespace_RacingAccessPolicyUpdateLeavesNoOrphans(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + policy := &ateapipb.AccessPolicy{Bindings: []*ateapipb.Binding{{Role: authz.RoleEditor, Members: []string{"user:bob"}}}} + for i := range 10 { + atespace := fmt.Sprintf("policy-update-race-%d", i) + createTestAtespace(t, s, atespace) + created, err := s.CreateAtespaceAccessPolicy(ctx, atespace, policy) + if err != nil { + t.Fatalf("CreateAtespaceAccessPolicy failed: %v", err) + } + + var updateErr, deleteErr error + race(func() { + _, updateErr = s.UpdateAtespaceAccessPolicy(ctx, atespace, store.PreconditionFrom(created), func(ap *ateapipb.AccessPolicy) error { + ap.Bindings = []*ateapipb.Binding{{Role: authz.RoleViewer, Members: []string{"user:carol"}}} + return nil + }) + }, func() { + _, deleteErr = s.DeleteAtespace(ctx, atespace, store.DeletePreconditions{}) + }) + if deleteErr != nil { + t.Errorf("DeleteAtespace = %v, want nil", deleteErr) + } + if updateErr != nil && !errors.Is(updateErr, store.ErrNotFound) { + t.Errorf("UpdateAtespaceAccessPolicy = %v, want nil or ErrNotFound", updateErr) + } + } + requireNoOrphans(t, s) +} + +func TestDeleteActor_RacingEgressPolicyCreateLeavesNoOrphans(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + createTestAtespace(t, s, "team-a") + for i := range 20 { + actor, err := s.CreateActor(ctx, newTestActor("team-a", fmt.Sprintf("actor-%d", i))) + if err != nil { + t.Fatalf("CreateActor failed: %v", err) + } + actorRef := resources.ActorRefFromActor(actor) + + var createErr, deleteErr error + race(func() { + _, createErr = s.CreateEgressPolicy(ctx, actorRef, &ateapipb.EgressPolicy{ + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "default"}, + }) + }, func() { + _, deleteErr = s.DeleteActor(ctx, actorRef, store.DeletePreconditions{}) + }) + if deleteErr != nil { + t.Errorf("DeleteActor = %v, want nil", deleteErr) + } + if createErr != nil && !errors.Is(createErr, store.ErrFailedPrecondition) { + t.Errorf("CreateEgressPolicy = %v, want nil or ErrFailedPrecondition", createErr) + } + if _, err := s.GetEgressPolicy(ctx, actorRef); !errors.Is(err, store.ErrNotFound) { + t.Errorf("GetEgressPolicy after DeleteActor = %v, want ErrNotFound", err) + } + } + requireNoOrphans(t, s) +} + +func TestDeleteWorker_RacingBindLeavesNoOrphans(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + for i := range 20 { + name := fmt.Sprintf("bind-race-worker-%d", i) + if _, err := s.CreateWorker(ctx, newTestWorker(name)); err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + + var bindErr, deleteErr error + race(func() { + bindErr = s.BindActorToWorker(ctx, name, &ateapipb.ActorAssignment{ActorUid: fmt.Sprintf("uid-%d", i)}, nil) + }, func() { + _, deleteErr = s.DeleteWorker(ctx, name, store.DeletePreconditions{}) + }) + if deleteErr != nil { + t.Errorf("DeleteWorker = %v, want nil", deleteErr) + } + if bindErr != nil && !errors.Is(bindErr, store.ErrNotFound) { + t.Errorf("BindActorToWorker = %v, want nil or ErrNotFound", bindErr) + } + } + requireNoOrphans(t, s) +} + +// atespaceTuples returns the stored OpenFGA tuples on an atespace as +// "relation user" strings, read straight from the tuple table. +func atespaceTuples(t *testing.T, p *Persistence, name string) []string { + t.Helper() + return tuplesOn(t, p, "atespace", name) +} + +func tuplesOn(t *testing.T, p *Persistence, objectType, objectID string) []string { + t.Helper() + rows, err := p.db.QueryContext(t.Context(), `SELECT relation, _user FROM tuple WHERE object_type = ? AND object_id = ?`, objectType, objectID) + if err != nil { + t.Fatalf("reading tuples: %v", err) + } + defer rows.Close() + var out []string + for rows.Next() { + var relation, user string + if err := rows.Scan(&relation, &user); err != nil { + t.Fatalf("scanning tuple: %v", err) + } + out = append(out, relation+" "+user) + } + if err := rows.Err(); err != nil { + t.Fatalf("reading tuples: %v", err) + } + slices.Sort(out) + return out +} + +func TestDeleteAtespace_RemovesAccessPolicyAndTuples(t *testing.T) { + s := setupMySQLPersistence(t) + ctx := t.Context() + createTestAtespace(t, s, "team-a") + createTestAtespace(t, s, "team-b") + policy := &ateapipb.AccessPolicy{Bindings: []*ateapipb.Binding{{Role: authz.RoleEditor, Members: []string{"user:bob"}}}} + for _, name := range []string{"team-a", "team-b"} { + if _, err := s.CreateAtespaceAccessPolicy(ctx, name, policy); err != nil { + t.Fatalf("CreateAtespaceAccessPolicy(%s) failed: %v", name, err) + } + } + wantTuples := []string{"editor user:bob"} + if got := atespaceTuples(t, s, "team-a"); !slices.Equal(got, wantTuples) { + t.Fatalf("tuples after create = %q, want %q", got, wantTuples) + } + + // A refused delete rolls back with everything it touched. + if _, err := s.CreateActor(ctx, newTestActor("team-a", "a1")); err != nil { + t.Fatalf("CreateActor failed: %v", err) + } + if _, err := s.DeleteAtespace(ctx, "team-a", store.DeletePreconditions{}); !errors.Is(err, store.ErrFailedPrecondition) { + t.Fatalf("DeleteAtespace of a non-empty atespace = %v, want ErrFailedPrecondition", err) + } + if _, err := s.GetAtespaceAccessPolicy(ctx, "team-a"); err != nil { + t.Errorf("GetAtespaceAccessPolicy after a refused delete = %v, want the policy", err) + } + if got := atespaceTuples(t, s, "team-a"); !slices.Equal(got, wantTuples) { + t.Errorf("tuples after a refused delete = %q, want %q", got, wantTuples) + } + + if _, err := s.DeleteActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "a1"}, store.DeletePreconditions{}); err != nil { + t.Fatalf("DeleteActor failed: %v", err) + } + if _, err := s.DeleteAtespace(ctx, "team-a", store.DeletePreconditions{}); err != nil { + t.Fatalf("DeleteAtespace failed: %v", err) + } + if _, err := s.GetAtespaceAccessPolicy(ctx, "team-a"); !errors.Is(err, store.ErrNotFound) { + t.Errorf("GetAtespaceAccessPolicy after DeleteAtespace = %v, want ErrNotFound", err) + } + if got := atespaceTuples(t, s, "team-a"); len(got) != 0 { + t.Errorf("tuples after DeleteAtespace = %q, want none", got) + } + if _, err := s.GetAtespaceAccessPolicy(ctx, "team-b"); err != nil { + t.Errorf("GetAtespaceAccessPolicy(team-b) = %v, want the policy", err) + } + if got := atespaceTuples(t, s, "team-b"); !slices.Equal(got, wantTuples) { + t.Errorf("team-b tuples = %q, want %q", got, wantTuples) + } +} diff --git a/cmd/ateapi/internal/store/atemy/schema.go b/cmd/ateapi/internal/store/atemy/schema.go new file mode 100644 index 0000000000..2c47e7a198 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/schema.go @@ -0,0 +1,218 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "crypto/sha256" + "database/sql" + "embed" + "encoding/hex" + "errors" + "fmt" + "io/fs" + "strconv" + "strings" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/pressly/goose/v3" + "github.com/pressly/goose/v3/database" + "github.com/pressly/goose/v3/lock" +) + +const ( + migrationTableName = "schema_migrations" + // migrationLockTimeout bounds how long a replica waits for another + // replica's migration run. + migrationLockTimeout = 5 * time.Minute +) + +//go:embed migrations/*.sql +var migrationFiles embed.FS + +func applyMigrations(ctx context.Context, db *sql.DB) error { + if err := ctx.Err(); err != nil { + return err + } + if err := rejectUnversionedSubstrateSchema(ctx, db); err != nil { + return err + } + migrations, err := fs.Sub(migrationFiles, "migrations") + if err != nil { + return fmt.Errorf("open embedded MySQL migrations: %w", err) + } + provider, err := newMigrationProvider(ctx, db, migrations) + if err != nil { + return err + } + // provider.Close would close db, which the caller owns. + return storesql.MigrateToLatest(ctx, provider, "MySQL") +} + +// newMigrationProvider returns a Goose provider for migrations on db that +// serializes runs across replicas. +func newMigrationProvider(ctx context.Context, db *sql.DB, migrations fs.FS) (*goose.Provider, error) { + var name string + if err := db.QueryRowContext(ctx, `SELECT DATABASE()`).Scan(&name); err != nil { + return nil, fmt.Errorf("get MySQL database name: %w", err) + } + base, err := database.NewStore(database.DialectMySQL, migrationTableName) + if err != nil { + return nil, fmt.Errorf("create MySQL migration store: %w", err) + } + provider, err := goose.NewProvider( + goose.DialectCustom, + db, + migrations, + goose.WithStore(migrationStore{Store: base}), + goose.WithSessionLocker(migrationLocker{name: migrationLockName(name)}), + ) + if err != nil { + return nil, fmt.Errorf("create MySQL migration provider: %w", err) + } + return provider, nil +} + +// migrationStore is Goose's MySQL store with a version-table check that also +// works on Vitess. Goose compares table_schema to DATABASE() in a SELECT with +// no FROM clause, which vtgate answers with the keyspace name rather than the +// MySQL schema behind it, so the table never appears to exist and the next +// startup fails to create it again. +type migrationStore struct { + database.Store +} + +var _ database.StoreExtender = migrationStore{} + +func (s migrationStore) TableExists(ctx context.Context, db database.DBTxConn) (bool, error) { + var n int + if err := db.QueryRowContext(ctx, ` + SELECT COUNT(*) FROM information_schema.tables + WHERE table_schema = DATABASE() AND table_name = ?`, s.Tablename()).Scan(&n); err != nil { + return false, fmt.Errorf("check MySQL migration ledger: %w", err) + } + return n > 0, nil +} + +// requireStrictMode refuses a session that would truncate an overlong value +// with a warning instead of an error. PostgreSQL never truncates. +func requireStrictMode(ctx context.Context, db *sql.DB) error { + var mode string + if err := db.QueryRowContext(ctx, `SELECT @@SESSION.sql_mode`).Scan(&mode); err != nil { + return fmt.Errorf("get MySQL sql_mode: %w", err) + } + for m := range strings.SplitSeq(strings.ToUpper(mode), ",") { + if m == "STRICT_TRANS_TABLES" || m == "STRICT_ALL_TABLES" { + return nil + } + } + return fmt.Errorf("atemy requires a strict sql_mode (STRICT_TRANS_TABLES or STRICT_ALL_TABLES), got %q", mode) +} + +// rejectUnversionedSubstrateSchema stops Goose from adopting tables it did not +// create, as atepg does: CREATE TABLE IF NOT EXISTS would otherwise accept a +// same-named table of any shape. Goose creates the migration ledger before +// the first migration, so a partial run still has one. +func rejectUnversionedSubstrateSchema(ctx context.Context, db *sql.DB) error { + var hasMetadata, hasSubstrateTables bool + err := db.QueryRowContext(ctx, ` + SELECT + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema = DATABASE() AND table_name = ?), + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema = DATABASE() + AND table_name IN ( + 'atespaces', 'actors', 'actor_egress_policies', 'actor_templates', + 'tags', 'workers', 'worker_assignments', 'worker_outbox', + 'worker_outbox_trim', 'leases' + ))`, migrationTableName).Scan(&hasMetadata, &hasSubstrateTables) + if err != nil { + return fmt.Errorf("check MySQL migration ledger: %w", err) + } + if hasSubstrateTables && !hasMetadata { + return errors.New("unsupported MySQL schema: Substrate tables exist without a migration ledger") + } + return nil +} + +// requireAutoIncrementStep refuses a server that hands out AUTO_INCREMENT +// values in steps, as multi-primary replication does. Watchers treat every +// skipped worker outbox seq as a write still committing, so steps would leave +// them tracking gaps that never fill. +func requireAutoIncrementStep(ctx context.Context, db *sql.DB) error { + var step int + if err := db.QueryRowContext(ctx, `SELECT @@auto_increment_increment`).Scan(&step); err != nil { + return fmt.Errorf("get MySQL auto_increment_increment: %w", err) + } + if step != 1 { + return fmt.Errorf("atemy requires auto_increment_increment = 1, got %d", step) + } + return nil +} + +// requireMySQL8 reports a clear error before an older server rejects the +// schema with an opaque one. The schema needs MySQL 8.0 for utf8mb4_0900_bin, +// FOR SHARE and SKIP LOCKED. Vitess reports a version such as 8.0.40-Vitess. +// MariaDB numbers its releases 10 and up but lacks utf8mb4_0900_bin. +func requireMySQL8(ctx context.Context, db *sql.DB) error { + var version string + if err := db.QueryRowContext(ctx, `SELECT VERSION()`).Scan(&version); err != nil { + return fmt.Errorf("get MySQL version: %w", err) + } + major, _, _ := strings.Cut(version, ".") + if n, err := strconv.Atoi(major); err != nil || n < 8 || strings.Contains(strings.ToLower(version), "mariadb") { + return fmt.Errorf("atemy requires MySQL 8.0 or newer. VERSION() is %q", version) + } + return nil +} + +// migrationLockName scopes the migration lock to one database. MySQL caps +// lock names at 64 characters, so the database name is hashed. +func migrationLockName(database string) string { + sum := sha256.Sum256([]byte(database)) + return "atemy-migrations:" + hex.EncodeToString(sum[:16]) +} + +// migrationLocker serializes migration runs across replicas with a MySQL +// named lock. The lock belongs to the session, so goose holds one connection +// for the whole run, and the server releases the lock if that connection drops. +type migrationLocker struct { + name string +} + +var _ lock.SessionLocker = migrationLocker{} + +func (l migrationLocker) SessionLock(ctx context.Context, conn *sql.Conn) error { + var got sql.NullInt64 + if err := conn.QueryRowContext(ctx, `SELECT GET_LOCK(?, ?)`, l.name, int(migrationLockTimeout.Seconds())).Scan(&got); err != nil { + return fmt.Errorf("acquire MySQL migration lock: %w", err) + } + if !got.Valid || got.Int64 != 1 { + return fmt.Errorf("acquire MySQL migration lock: timed out after %s", migrationLockTimeout) + } + return nil +} + +func (l migrationLocker) SessionUnlock(ctx context.Context, conn *sql.Conn) error { + var released sql.NullInt64 + if err := conn.QueryRowContext(ctx, `SELECT RELEASE_LOCK(?)`, l.name).Scan(&released); err != nil { + return fmt.Errorf("release MySQL migration lock: %w", err) + } + if !released.Valid || released.Int64 != 1 { + return fmt.Errorf("release MySQL migration lock: this session did not hold it") + } + return nil +} diff --git a/cmd/ateapi/internal/store/atemy/tag.go b/cmd/ateapi/internal/store/atemy/tag.go new file mode 100644 index 0000000000..fda2164848 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/tag.go @@ -0,0 +1,150 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +func (p *Persistence) GetTag(ctx context.Context, tagRef resources.TagRef) (*ateapipb.Tag, error) { + atespace, name := tagRef.Atespace, tagRef.Name + var protoBytes []byte + if err := p.db.QueryRowContext(ctx, ` + SELECT proto FROM tags + WHERE atespace = ? AND name = ?`, atespace, name).Scan(&protoBytes); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting tag %s/%s: %w", atespace, name, err) + } + tag := &ateapipb.Tag{} + if err := storesql.UnmarshalStored(protoBytes, tag); err != nil { + return nil, fmt.Errorf("unmarshaling tag: %w", err) + } + return tag, nil +} + +func (p *Persistence) ListTags(ctx context.Context, atespace string, opts store.ListOptions) (store.ListResponse[*ateapipb.Tag], error) { + opts, err := store.NormalizeListOptions(opts) + if err != nil { + return store.ListResponse[*ateapipb.Tag]{}, err + } + decode := func(b []byte, keys []string) (*ateapipb.Tag, error) { + tag := &ateapipb.Tag{} + return tag, storesql.UnmarshalRow(b, tag, "tag", rowID(atespace, keys)...) + } + items, next, err := listScoped(ctx, p.db, "tags", storesql.KindTag, atespace, opts, decode) + if err != nil { + return store.ListResponse[*ateapipb.Tag]{}, err + } + return store.ListResponse[*ateapipb.Tag]{Items: items, NextPageToken: next}, nil +} + +func (p *Persistence) CreateTag(ctx context.Context, tag *ateapipb.Tag) (*ateapipb.Tag, error) { + atespace := tag.GetMetadata().GetAtespace() + name := tag.GetMetadata().GetName() + dbTag := proto.CloneOf(tag) + storesql.SetCreateMetadata(dbTag.Metadata) + protoBytes, err := proto.Marshal(dbTag) + if err != nil { + return nil, fmt.Errorf("marshaling tag: %w", err) + } + err = p.insertChild(ctx, "tag "+atespace+"/"+name, func(tx *sql.Tx) error { return lockAtespace(ctx, tx, atespace) }, ` + INSERT INTO tags (atespace, name, uid, version, proto) + VALUES (?, ?, ?, ?, ?)`, atespace, name, + dbTag.GetMetadata().GetUid(), dbTag.GetMetadata().GetVersion(), protoBytes) + if err != nil { + return nil, err + } + return dbTag, nil +} + +func (p *Persistence) UpdateTag(ctx context.Context, tagRef resources.TagRef, precondition store.Precondition, mutate func(*ateapipb.Tag) error) (*ateapipb.Tag, error) { + if err := precondition.Validate(); err != nil { + return nil, err + } + atespace, name := tagRef.Atespace, tagRef.Name + var currentUID string + var currentVersion int64 + var currentBytes []byte + if err := p.db.QueryRowContext(ctx, ` + SELECT uid, version, proto FROM tags + WHERE atespace = ? AND name = ?`, atespace, name).Scan(¤tUID, ¤tVersion, ¤tBytes); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting tag %s/%s for update: %w", atespace, name, err) + } + + dbTag := &ateapipb.Tag{} + if err := storesql.UnmarshalStored(currentBytes, dbTag); err != nil { + return nil, fmt.Errorf("unmarshaling tag: %w", err) + } + if err := storesql.ValidateProtoMetadataMatchesColumns(fmt.Sprintf("tag %s/%s", atespace, name), dbTag.GetMetadata(), currentUID, currentVersion); err != nil { + return nil, err + } + if err := precondition.Check(dbTag.GetMetadata()); err != nil { + return nil, err + } + tagBeforeMutation := proto.Clone(dbTag).(*ateapipb.Tag) + if err := mutate(dbTag); err != nil { + return nil, err + } + if err := storesql.ValidateUpdateTagMutation(tagBeforeMutation, dbTag); err != nil { + return nil, fmt.Errorf("%w: %w", store.ErrImmutableField, err) + } + // Stored metadata is authoritative; discard any metadata edits made by the + // closure and derive the next revision from the state this attempt read. + storesql.SetUpdateMetadata(dbTag.Metadata, tagBeforeMutation.GetMetadata()) + + updatedBytes, err := proto.Marshal(dbTag) + if err != nil { + return nil, fmt.Errorf("marshaling tag: %w", err) + } + if err := updateGuarded(ctx, p.db, fmt.Sprintf("tag %s/%s", atespace, name), ` + UPDATE tags + SET version = ?, proto = ? + WHERE atespace = ? AND name = ? AND uid = ? AND version = ?`, + dbTag.GetMetadata().GetVersion(), updatedBytes, atespace, name, currentUID, currentVersion); err != nil { + return nil, err + } + return dbTag, nil +} + +func (p *Persistence) DeleteTag(ctx context.Context, tagRef resources.TagRef, precondition store.DeletePreconditions) (*ateapipb.Tag, error) { + var protoBytes []byte + err := inTx(ctx, p.db, func(tx *sql.Tx) error { + var err error + _, _, protoBytes, err = deleteLockedRow(ctx, tx, "tags", "atespace = ? AND name = ?", precondition, tagRef.Atespace, tagRef.Name) + return err + }) + if err != nil { + return nil, err + } + tag := &ateapipb.Tag{} + if err := storesql.UnmarshalStored(protoBytes, tag); err != nil { + return nil, fmt.Errorf("unmarshaling deleted tag: %w", err) + } + return tag, nil +} diff --git a/cmd/ateapi/internal/store/atemy/tag_test.go b/cmd/ateapi/internal/store/atemy/tag_test.go new file mode 100644 index 0000000000..a058eb13ef --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/tag_test.go @@ -0,0 +1,38 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "testing" + + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// createTestTag creates tagName over its own copy of an actor's external +// snapshot, already finalized. +func createTestTag(t *testing.T, s *Persistence, tagAtespace, tagName string) *ateapipb.Tag { + t.Helper() + tag, err := s.CreateTag(t.Context(), &ateapipb.Tag{ + Metadata: &ateapipb.ResourceMetadata{Atespace: tagAtespace, Name: tagName}, + Scope: ateapipb.TagScope_TAG_SCOPE_ATESPACE, + Status: &ateapipb.TagStatus{ + Snapshot: &ateapipb.ExternalSnapshot{SnapshotUri: "gs://bucket/atespaces/" + tagAtespace + "/tags/" + tagName}, + }, + }) + if err != nil { + t.Fatalf("CreateTag(%s/%s) failed: %v", tagAtespace, tagName, err) + } + return tag +} diff --git a/cmd/ateapi/internal/store/atemy/vitess_test.go b/cmd/ateapi/internal/store/atemy/vitess_test.go new file mode 100644 index 0000000000..2adcfbe3bb --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/vitess_test.go @@ -0,0 +1,128 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build vitess + +package atemy + +import ( + "context" + "database/sql" + "fmt" + "testing" + "time" + + "github.com/testcontainers/testcontainers-go" + "github.com/testcontainers/testcontainers-go/wait" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/dockerenv" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storecontract" +) + +// vitessImage runs vtgate and an unsharded keyspace in one container, the +// query layer PlanetScale serves. It is built for linux/amd64 only and is +// large, so these tests sit behind the vitess build tag and CI runs them in a +// step of their own. +const vitessImage = "vitess/vttestserver:mysql80" + +// TestVitess runs the store through vtgate, so a statement Vitess rejects or +// rewrites fails here rather than on PlanetScale. Foreign keys are disallowed, +// as on a PlanetScale database by default. +func TestVitess(t *testing.T) { + db := startVitess(t) + ctx := t.Context() + + // A restart must find the migration ledger the first start wrote. + for range 2 { + p, err := NewPersistence(ctx, db) + if err != nil { + t.Fatalf("NewPersistence through vtgate failed: %v", err) + } + p.Close() + } + + storecontract.RunContractTests(t, func(t *testing.T) store.Interface { + t.Helper() + return setupPersistenceOn(t, db) + }) + // The rollback probe's FOR SHARE NOWAIT must pass through vtgate. + t.Run("DropsOnlyRolledBackSeqs", func(t *testing.T) { + testDropsOnlyRolledBackSeqs(t, setupPersistenceOn(t, db)) + }) + t.Run("TrimsOnlyTheExpiredPrefix", func(t *testing.T) { + testTrimsOnlyTheExpiredPrefix(t, setupPersistenceOn(t, db)) + }) + t.Run("CleanupSkipsLockedRows", func(t *testing.T) { + testCleanupSkipsLockedRows(t, setupPersistenceOn(t, db)) + }) +} + +func startVitess(t *testing.T) *sql.DB { + t.Helper() + ctx := context.Background() + fail := t.Skipf + if dockerenv.Required() { + fail = t.Fatalf + } + if err := dockerenv.Configure(ctx); err != nil { + fail("configuring Docker for the Vitess testcontainer: %v", err) + } + const port = "33577/tcp" // vtgate's MySQL port: PORT + 3 + c, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ + ContainerRequest: testcontainers.ContainerRequest{ + Image: vitessImage, + ImagePlatform: "linux/amd64", + Env: map[string]string{ + "PORT": "33574", + "KEYSPACES": "atemy", + "NUM_SHARDS": "1", + "MYSQL_BIND_HOST": "0.0.0.0", + "FOREIGN_KEY_MODE": "disallow", + }, + ExposedPorts: []string{port}, + WaitingFor: wait.ForListeningPort(port).WithStartupTimeout(10 * time.Minute), + }, + Started: true, + }) + if err != nil { + fail("Vitess testcontainer unavailable (requires Docker; CI or REQUIRE_DOCKER makes this fatal): %v", err) + } + t.Cleanup(func() { + if err := c.Terminate(context.Background()); err != nil { + t.Errorf("terminating Vitess testcontainer: %v", err) + } + }) + endpoint, err := c.PortEndpoint(ctx, port, "") + if err != nil { + t.Fatalf("reading the vtgate endpoint: %v", err) + } + db, err := Open(fmt.Sprintf("root@tcp(%s)/atemy", endpoint)) + if err != nil { + t.Fatalf("opening vtgate: %v", err) + } + t.Cleanup(func() { db.Close() }) + // vtgate accepts connections before the keyspace's tablet is serving. + var pingErr error + for deadline := time.Now().Add(5 * time.Minute); time.Now().Before(deadline); time.Sleep(time.Second) { + if _, pingErr = db.ExecContext(ctx, `SELECT 1 FROM dual`); pingErr == nil { + var n int + if pingErr = db.QueryRowContext(ctx, `SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = DATABASE()`).Scan(&n); pingErr == nil { + return db + } + } + } + t.Fatalf("vtgate never served the atemy keyspace: %v", pingErr) + return nil +} diff --git a/cmd/ateapi/internal/store/atemy/worker.go b/cmd/ateapi/internal/store/atemy/worker.go new file mode 100644 index 0000000000..260a58daa7 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/worker.go @@ -0,0 +1,182 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +func (p *Persistence) CreateWorker(ctx context.Context, worker *ateapipb.Worker) (*ateapipb.Worker, error) { + dbWorker := proto.Clone(worker).(*ateapipb.Worker) + if dbWorker.Metadata == nil { + dbWorker.Metadata = &ateapipb.ResourceMetadata{} + } + storesql.SetCreateMetadata(dbWorker.Metadata) + + protoBytes, err := proto.Marshal(dbWorker) + if err != nil { + return nil, fmt.Errorf("marshaling worker: %w", err) + } + + created, err := p.writeAndAppendEvent(ctx, store.WorkerEventCreated, func(ctx context.Context, tx *sql.Tx) (*ateapipb.Worker, error) { + _, err := tx.ExecContext(ctx, ` + INSERT INTO workers (name, uid, version, proto) + VALUES (?, ?, ?, ?)`, + dbWorker.GetMetadata().GetName(), dbWorker.GetMetadata().GetUid(), dbWorker.GetMetadata().GetVersion(), protoBytes) + if err != nil { + return nil, err + } + return dbWorker, nil + }) + if err != nil { + if isUniqueViolation(err) { + return nil, store.ErrAlreadyExists + } + return nil, fmt.Errorf("creating worker: %w", err) + } + return created, nil +} + +func (p *Persistence) GetWorker(ctx context.Context, name string) (*ateapipb.Worker, error) { + var protoBytes []byte + err := p.db.QueryRowContext(ctx, `SELECT proto FROM workers WHERE name = ?`, name).Scan(&protoBytes) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("getting worker %s: %w", name, err) + } + out := &ateapipb.Worker{} + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { + return nil, fmt.Errorf("unmarshaling worker: %w", err) + } + return out, nil +} + +func getWorkerRowForUpdate(ctx context.Context, tx *sql.Tx, name string) (*ateapipb.Worker, error) { + var protoBytes []byte + if err := tx.QueryRowContext(ctx, `SELECT proto FROM workers WHERE name = ? FOR UPDATE`, name).Scan(&protoBytes); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("locking worker %s for update: %w", name, err) + } + out := &ateapipb.Worker{} + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { + return nil, fmt.Errorf("unmarshaling worker: %w", err) + } + return out, nil +} + +// UpdateWorker runs mutate against the worker read FOR UPDATE inside the write +// transaction, so a concurrent writer blocks on the row lock rather than +// interleaving. That is what makes an occupancy test inside mutate a +// compare-and-set. The predicate cannot be pushed into SQL: the row stores an +// opaque marshaled proto, so assignment is not addressable in a WHERE clause. +func (p *Persistence) UpdateWorker(ctx context.Context, name string, precondition store.Precondition, mutate func(*ateapipb.Worker) error) (*ateapipb.Worker, error) { + if err := precondition.Validate(); err != nil { + return nil, err + } + return p.writeAndAppendEvent(ctx, store.WorkerEventUpdated, func(ctx context.Context, tx *sql.Tx) (*ateapipb.Worker, error) { + dbWorker, err := getWorkerRowForUpdate(ctx, tx, name) + if err != nil { + return nil, err + } + if err := precondition.Check(dbWorker.GetMetadata()); err != nil { + return nil, err + } + + oldMeta := proto.CloneOf(dbWorker.GetMetadata()) + if err := mutate(dbWorker); err != nil { + return nil, err + } + // Stored metadata is authoritative; discard any metadata edits made by + // the closure and derive the next revision from the row we locked. + storesql.SetUpdateMetadata(dbWorker.Metadata, oldMeta) + + protoBytes, err := proto.Marshal(dbWorker) + if err != nil { + return nil, fmt.Errorf("marshaling worker: %w", err) + } + res, err := tx.ExecContext(ctx, ` + UPDATE workers + SET version = ?, proto = ? + WHERE name = ?`, + dbWorker.GetMetadata().GetVersion(), protoBytes, name) + if err != nil { + return nil, fmt.Errorf("updating worker %s: %w", name, err) + } + if err := requireOneRow(res, "updating worker "+name); err != nil { + return nil, err + } + return dbWorker, nil + }) +} + +// DeleteWorker removes the worker's assignments with it. +func (p *Persistence) DeleteWorker(ctx context.Context, name string, precondition store.DeletePreconditions) (*ateapipb.Worker, error) { + return p.writeAndAppendEvent(ctx, store.WorkerEventDeleted, func(ctx context.Context, tx *sql.Tx) (*ateapipb.Worker, error) { + // Locked so the precondition checks the incarnation the DELETE removes. + deleted, err := getWorkerRowForUpdate(ctx, tx, name) + if err != nil { + return nil, err + } + if err := precondition.Check(deleted.GetMetadata()); err != nil { + return nil, err + } + if _, err := tx.ExecContext(ctx, `DELETE FROM worker_assignments WHERE worker_name = ?`, name); err != nil { + return nil, fmt.Errorf("deleting assignments of worker %s: %w", name, err) + } + res, err := tx.ExecContext(ctx, `DELETE FROM workers WHERE name = ?`, name) + if err != nil { + return nil, fmt.Errorf("deleting worker %s: %w", name, err) + } + if err := requireOneRow(res, "deleting worker "+name); err != nil { + return nil, err + } + return deleted, nil + }) +} + +func (p *Persistence) ListWorkers(ctx context.Context, opts store.ListOptions) (store.ListResponse[*ateapipb.Worker], error) { + opts, err := store.NormalizeListOptions(opts) + if err != nil { + return store.ListResponse[*ateapipb.Worker]{}, err + } + token, err := storesql.DecodePageToken(opts.PageToken, storesql.KindWorker, "", 1) + if err != nil { + return store.ListResponse[*ateapipb.Worker]{}, err + } + query, args := `SELECT name, proto FROM workers ORDER BY name LIMIT ?`, []any{int64(opts.PageSize) + 1} + if len(token.Last) > 0 { + query, args = `SELECT name, proto FROM workers WHERE name > ? ORDER BY name LIMIT ?`, []any{token.Last[0], int64(opts.PageSize) + 1} + } + items, next, err := listPage(ctx, p.db, storesql.KindWorker, "", opts.PageSize, func(b []byte, keys []string) (*ateapipb.Worker, error) { + w := &ateapipb.Worker{} + return w, storesql.UnmarshalRow(b, w, "worker", keys...) + }, query, args...) + if err != nil { + return store.ListResponse[*ateapipb.Worker]{}, err + } + return store.ListResponse[*ateapipb.Worker]{Items: items, NextPageToken: next}, nil +} diff --git a/cmd/ateapi/internal/store/atemy/worker_assignment.go b/cmd/ateapi/internal/store/atemy/worker_assignment.go new file mode 100644 index 0000000000..e0ed2b677f --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/worker_assignment.go @@ -0,0 +1,316 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" + "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +// getWorkerForUpdate reads a Worker and holds its row until the caller's +// transaction commits, which is what serializes updates to its allocation. The +// caller supplies the transaction; this only takes the lock. +func getWorkerForUpdate(ctx context.Context, tx *sql.Tx, name string) (*ateapipb.Worker, error) { + var protoBytes []byte + err := tx.QueryRowContext(ctx, `SELECT proto FROM workers WHERE name = ? FOR UPDATE`, name).Scan(&protoBytes) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, store.ErrNotFound + } + return nil, fmt.Errorf("locking worker %s: %w", name, err) + } + worker := &ateapipb.Worker{} + if err := proto.Unmarshal(protoBytes, worker); err != nil { + return nil, fmt.Errorf("unmarshaling worker: %w", err) + } + return worker, nil +} + +// saveWorker writes back a Worker whose allocation just moved, at the next +// version. This assumes the caller holds the row lock getWorkerForUpdate took. +func saveWorker(ctx context.Context, tx *sql.Tx, worker *ateapipb.Worker) error { + read := proto.CloneOf(worker.GetMetadata()) + storesql.SetUpdateMetadata(worker.GetMetadata(), worker.GetMetadata()) + protoBytes, err := proto.Marshal(worker) + if err != nil { + return fmt.Errorf("marshaling worker: %w", err) + } + // Callers hold the row lock getWorkerForUpdate took, so this matches the + // row they read. It is stated anyway so a caller that skipped the lock + // fails loudly instead of overwriting a newer Worker, and so a row that is + // gone is an error rather than an update of nothing. + res, err := tx.ExecContext(ctx, `UPDATE workers SET version = ?, proto = ? WHERE name = ? AND uid = ? AND version = ?`, + worker.GetMetadata().GetVersion(), protoBytes, read.GetName(), read.GetUid(), read.GetVersion()) + if err != nil { + return fmt.Errorf("updating worker %s: %w", read.GetName(), err) + } + if n, err := res.RowsAffected(); err != nil { + return fmt.Errorf("updating worker %s: %w", read.GetName(), err) + } else if n != 1 { + return store.ErrVersionConflict + } + return nil +} + +func (p *Persistence) BindActorToWorker(ctx context.Context, workerName string, assignment *ateapipb.ActorAssignment, admit func(*ateapipb.Worker) error) error { + actorUID := assignment.GetActorUid() + if actorUID == "" { + return fmt.Errorf("binding an assignment with no actor_uid to worker %s", workerName) + } + // The store assigns identity. atespace is empty because Workers are + // global-scoped; the name is the Actor's UID, which is also the row key. + // This is the identity a first bind gets; a rebind keeps the recorded one. + createMeta := &ateapipb.ResourceMetadata{Name: actorUID} + storesql.SetCreateMetadata(createMeta) + + _, err := p.writeAndAppendEvent(ctx, store.WorkerEventUpdated, func(ctx context.Context, tx *sql.Tx) (*ateapipb.Worker, error) { + // A retried transaction starts again from the first bind's identity. + assignment.Metadata = proto.CloneOf(createMeta) + worker, err := getWorkerForUpdate(ctx, tx, workerName) + if err != nil { + return nil, err + } + if worker.Status == nil { + worker.Status = &ateapipb.WorkerStatus{} + } + + // Read under the row lock, so an epoch raised concurrently is either + // seen here or raised after this bind commits. + assignment.WorkerEpoch = worker.GetEpoch() + assignmentBytes, err := proto.Marshal(assignment) + if err != nil { + return nil, fmt.Errorf("marshaling assignment: %w", err) + } + + // Insert first and let the duplicate key say whether the Actor was + // already bound. Checking with a read instead would miss a claim that + // commits after it, and both claims would believe they were first. A + // failed statement does not abort a MySQL transaction, so the rebind + // path below runs in the same transaction. + _, err = tx.ExecContext(ctx, ` + INSERT INTO worker_assignments (actor_uid, worker_name, proto) + VALUES (?, ?, ?)`, + actorUID, workerName, assignmentBytes) + if err != nil && !isUniqueViolation(err) { + return nil, fmt.Errorf("binding actor %s to worker %s: %w", actorUID, workerName, err) + } + if err == nil { + // A new binding needs room. The row lock holds the answer until + // commit, and a refusal rolls the insert back. + if admit != nil { + if err := admit(worker); err != nil { + return nil, err + } + } + allocated, err := resources.AddToAllocated(worker.Status.Allocated, assignment, +1) + if err != nil { + return nil, err + } + worker.Status.Allocated = allocated + if err := saveWorker(ctx, tx, worker); err != nil { + return nil, err + } + return worker, nil + } + + // Already bound. Only this path, the retried claim, pays for the read. + previous, previousWorker, err := getAssignmentRow(ctx, tx, actorUID) + if err != nil { + return nil, err + } + if previousWorker != workerName { + return nil, fmt.Errorf("actor %s is already hosted by worker %s", actorUID, previousWorker) + } + + // Subtract before adding: the Actor is already counted, and its + // declared size may have changed. + allocated, err := resources.AddToAllocated(worker.Status.Allocated, previous, -1) + if err != nil { + return nil, err + } + worker.Status.Allocated = allocated + + // Admit against the Worker without the old reservation. An + // ActorTemplate is mutable, so a replacement can be larger than what + // it replaces. + if admit != nil { + if err := admit(worker); err != nil { + return nil, err + } + } + + if allocated, err = resources.AddToAllocated(allocated, assignment, +1); err != nil { + return nil, err + } + worker.Status.Allocated = allocated + + // A rebind updates the assignment already recorded, so re-stamping it + // as a create would make a retried claim look like a new subresource. + storesql.SetUpdateMetadata(assignment.Metadata, previous.GetMetadata()) + rebindBytes, err := proto.Marshal(assignment) + if err != nil { + return nil, fmt.Errorf("marshaling rebound assignment: %w", err) + } + + // Guarded on worker_name so a claim that moved the Actor elsewhere is + // refused rather than overwritten. + rebind, err := tx.ExecContext(ctx, ` + UPDATE worker_assignments SET proto = ? + WHERE actor_uid = ? AND worker_name = ?`, + rebindBytes, actorUID, workerName) + if err != nil { + return nil, fmt.Errorf("rebinding actor %s on worker %s: %w", actorUID, workerName, err) + } + if n, err := rebind.RowsAffected(); err != nil { + return nil, fmt.Errorf("rebinding actor %s on worker %s: %w", actorUID, workerName, err) + } else if n != 1 { + return nil, fmt.Errorf("%w: actor %s left worker %s while it was being rebound", store.ErrVersionConflict, actorUID, workerName) + } + if err := saveWorker(ctx, tx, worker); err != nil { + return nil, err + } + return worker, nil + }) + return err +} + +func (p *Persistence) ReleaseActorFromWorker(ctx context.Context, workerName string, actorUID string) (*ateapipb.Worker, error) { + return p.writeAndAppendEvent(ctx, store.WorkerEventUpdated, func(ctx context.Context, tx *sql.Tx) (*ateapipb.Worker, error) { + worker, err := getWorkerForUpdate(ctx, tx, workerName) + if err != nil { + return nil, err + } + + var protoBytes []byte + err = tx.QueryRowContext(ctx, ` + SELECT proto FROM worker_assignments + WHERE actor_uid = ? AND worker_name = ? + FOR UPDATE`, actorUID, workerName).Scan(&protoBytes) + if errors.Is(err, sql.ErrNoRows) { + return nil, nil // nothing to release, and so nothing to announce + } + if err != nil { + return nil, fmt.Errorf("releasing actor %s from worker %s: %w", actorUID, workerName, err) + } + if _, err := tx.ExecContext(ctx, `DELETE FROM worker_assignments WHERE actor_uid = ? AND worker_name = ?`, actorUID, workerName); err != nil { + return nil, fmt.Errorf("releasing actor %s from worker %s: %w", actorUID, workerName, err) + } + assignment := &ateapipb.ActorAssignment{} + if err := proto.Unmarshal(protoBytes, assignment); err != nil { + return nil, fmt.Errorf("unmarshaling released assignment: %w", err) + } + + if worker.Status == nil { + worker.Status = &ateapipb.WorkerStatus{} + } + allocated, err := resources.AddToAllocated(worker.Status.Allocated, assignment, -1) + if err != nil { + return nil, err + } + worker.Status.Allocated = allocated + if err := saveWorker(ctx, tx, worker); err != nil { + return nil, err + } + return worker, nil + }) +} + +// getAssignmentRow reads the assignment for actorUID and names the worker +// holding it. +func getAssignmentRow(ctx context.Context, q querier, actorUID string) (*ateapipb.ActorAssignment, string, error) { + var ( + protoBytes []byte + workerName string + ) + err := q.QueryRowContext(ctx, `SELECT proto, worker_name FROM worker_assignments WHERE actor_uid = ?`, actorUID). + Scan(&protoBytes, &workerName) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, "", store.ErrNotFound + } + return nil, "", fmt.Errorf("getting assignment for actor %s: %w", actorUID, err) + } + assignment := &ateapipb.ActorAssignment{} + if err := proto.Unmarshal(protoBytes, assignment); err != nil { + return nil, "", fmt.Errorf("unmarshaling assignment: %w", err) + } + return assignment, workerName, nil +} + +func (p *Persistence) GetWorkerAssignment(ctx context.Context, workerName, actorUID string) (*ateapipb.ActorAssignment, error) { + assignment, holder, err := getAssignmentRow(ctx, p.db, actorUID) + if err != nil { + return nil, err + } + if holder != workerName { + return nil, store.ErrNotFound + } + return assignment, nil +} + +func (p *Persistence) ListWorkerAssignments(ctx context.Context, workerName string, opts store.ListOptions) (store.ListResponse[*ateapipb.ActorAssignment], error) { + opts, err := store.NormalizeListOptions(opts) + if err != nil { + return store.ListResponse[*ateapipb.ActorAssignment]{}, err + } + // The token is scoped to the Worker, so one cannot be replayed against + // another Worker's assignments. + token, err := storesql.DecodePageToken(opts.PageToken, storesql.KindWorkerAssign, workerName, 1) + if err != nil { + return store.ListResponse[*ateapipb.ActorAssignment]{}, err + } + limit := int64(opts.PageSize) + 1 + query, args := ` + SELECT actor_uid, proto FROM worker_assignments + WHERE worker_name = ? + ORDER BY actor_uid LIMIT ?`, []any{workerName, limit} + if len(token.Last) > 0 { + query, args = ` + SELECT actor_uid, proto FROM worker_assignments + WHERE worker_name = ? AND actor_uid > ? + ORDER BY actor_uid LIMIT ?`, []any{workerName, token.Last[0], limit} + } + items, next, err := listPage(ctx, p.db, storesql.KindWorkerAssign, workerName, opts.PageSize, func(b []byte, keys []string) (*ateapipb.ActorAssignment, error) { + assignment := &ateapipb.ActorAssignment{} + if err := proto.Unmarshal(b, assignment); err != nil { + return nil, fmt.Errorf("unmarshaling assignment of actor %s: %w", keys[0], err) + } + return assignment, nil + }, query, args...) + if err != nil { + return store.ListResponse[*ateapipb.ActorAssignment]{}, err + } + return store.ListResponse[*ateapipb.ActorAssignment]{Items: items, NextPageToken: next}, nil +} + +func (p *Persistence) FindWorkerHostingActor(ctx context.Context, actorUID string) (string, error) { + var workerName string + err := p.db.QueryRowContext(ctx, `SELECT worker_name FROM worker_assignments WHERE actor_uid = ?`, actorUID).Scan(&workerName) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return "", store.ErrNotFound + } + return "", fmt.Errorf("finding the worker hosting actor %s: %w", actorUID, err) + } + return workerName, nil +} diff --git a/cmd/ateapi/internal/store/atemy/worker_assignment_test.go b/cmd/ateapi/internal/store/atemy/worker_assignment_test.go new file mode 100644 index 0000000000..200cc55ff8 --- /dev/null +++ b/cmd/ateapi/internal/store/atemy/worker_assignment_test.go @@ -0,0 +1,75 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atemy + +import ( + "errors" + "testing" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +func newTestWorker(name string) *ateapipb.Worker { + return &ateapipb.Worker{ + Metadata: &ateapipb.ResourceMetadata{Name: name}, + WorkerNamespace: "ns", + WorkerPool: "pool", + WorkerPod: name + "-pod", + } +} + +// saveWorker checks the stored version on top of the row lock its callers +// hold, so a Worker read before another write, or before a delete, cannot +// overwrite the row. +func TestSaveWorker_RejectsAStaleCopy(t *testing.T) { + for _, tc := range []struct { + name string + after func(*testing.T, *Persistence, *ateapipb.Worker) + }{ + {"updated since read", func(t *testing.T, p *Persistence, w *ateapipb.Worker) { + if _, err := p.UpdateWorker(t.Context(), w.GetMetadata().GetName(), store.PreconditionFrom(w), func(toUpdate *ateapipb.Worker) error { + toUpdate.Ips = []string{"10.0.0.1"} + return nil + }); err != nil { + t.Fatalf("UpdateWorker failed: %v", err) + } + }}, + {"deleted since read", func(t *testing.T, p *Persistence, w *ateapipb.Worker) { + if _, err := p.DeleteWorker(t.Context(), w.GetMetadata().GetName(), store.DeletePreconditions{}); err != nil { + t.Fatalf("DeleteWorker failed: %v", err) + } + }}, + } { + t.Run(tc.name, func(t *testing.T) { + p := setupMySQLPersistence(t) + ctx := t.Context() + created, err := p.CreateWorker(ctx, newTestWorker("stale-worker")) + if err != nil { + t.Fatalf("CreateWorker failed: %v", err) + } + tc.after(t, p, created) + + tx, err := p.db.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("BeginTx failed: %v", err) + } + defer tx.Rollback() //nolint:errcheck // never committed + if err := saveWorker(ctx, tx, created); !errors.Is(err, store.ErrVersionConflict) { + t.Errorf("saveWorker() = %v, want ErrVersionConflict", err) + } + }) + } +} diff --git a/cmd/ateapi/internal/store/atepg/access_policy.go b/cmd/ateapi/internal/store/atepg/access_policy.go index 475498ddb3..d59dd9fd06 100644 --- a/cmd/ateapi/internal/store/atepg/access_policy.go +++ b/cmd/ateapi/internal/store/atepg/access_policy.go @@ -19,7 +19,9 @@ import ( "errors" "fmt" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" "google.golang.org/protobuf/proto" @@ -28,7 +30,7 @@ import ( func (p *Persistence) CreateGlobalAccessPolicy(ctx context.Context, policy *ateapipb.AccessPolicy) (*ateapipb.AccessPolicy, error) { dbPolicy := proto.Clone(policy).(*ateapipb.AccessPolicy) dbPolicy.Metadata = &ateapipb.ResourceMetadata{Name: "default"} - setCreateMetadata(dbPolicy.Metadata) + storesql.SetCreateMetadata(dbPolicy.Metadata) protoBytes, err := proto.Marshal(dbPolicy) if err != nil { return nil, fmt.Errorf("marshaling global access policy: %w", err) @@ -49,7 +51,7 @@ func (p *Persistence) CreateGlobalAccessPolicy(ctx context.Context, policy *atea } return nil, fmt.Errorf("inserting global access policy: %w", err) } - if err := p.policyManager.ReconcileGlobalBindings(ctx, tx, dbPolicy.GetBindings()); err != nil { + if err := p.policyManager.ReconcileGlobalBindings(ctx, authz.PgxTx(tx), dbPolicy.GetBindings()); err != nil { return nil, fmt.Errorf("reconciling global access policy bindings: %w", err) } if err := tx.Commit(ctx); err != nil { @@ -89,7 +91,7 @@ func (p *Persistence) UpdateGlobalAccessPolicy(ctx context.Context, precondition return nil, err } dbPolicy.Metadata = oldMeta - setUpdateMetadata(dbPolicy.Metadata, oldMeta) + storesql.SetUpdateMetadata(dbPolicy.Metadata, oldMeta) protoBytes, err := proto.Marshal(dbPolicy) if err != nil { return nil, fmt.Errorf("marshaling updated global access policy: %w", err) @@ -100,7 +102,7 @@ func (p *Persistence) UpdateGlobalAccessPolicy(ctx context.Context, precondition if err != nil { return nil, fmt.Errorf("updating global access policy: %w", err) } - if err := p.policyManager.ReconcileGlobalBindings(ctx, tx, dbPolicy.GetBindings()); err != nil { + if err := p.policyManager.ReconcileGlobalBindings(ctx, authz.PgxTx(tx), dbPolicy.GetBindings()); err != nil { return nil, fmt.Errorf("reconciling global access policy bindings: %w", err) } if err := tx.Commit(ctx); err != nil { @@ -112,7 +114,7 @@ func (p *Persistence) UpdateGlobalAccessPolicy(ctx context.Context, precondition func (p *Persistence) CreateAtespaceAccessPolicy(ctx context.Context, name string, policy *ateapipb.AccessPolicy) (*ateapipb.AccessPolicy, error) { dbPolicy := proto.Clone(policy).(*ateapipb.AccessPolicy) dbPolicy.Metadata = &ateapipb.ResourceMetadata{Name: "default"} - setCreateMetadata(dbPolicy.Metadata) + storesql.SetCreateMetadata(dbPolicy.Metadata) protoBytes, err := proto.Marshal(dbPolicy) if err != nil { return nil, fmt.Errorf("marshaling access policy: %w", err) @@ -136,7 +138,7 @@ func (p *Persistence) CreateAtespaceAccessPolicy(ctx context.Context, name strin } return nil, fmt.Errorf("inserting access policy for %s: %w", name, err) } - if err := p.policyManager.ReconcileAtespaceBindings(ctx, tx, name, dbPolicy.GetBindings()); err != nil { + if err := p.policyManager.ReconcileAtespaceBindings(ctx, authz.PgxTx(tx), name, dbPolicy.GetBindings()); err != nil { return nil, fmt.Errorf("reconciling access policy bindings for %s: %w", name, err) } if err := tx.Commit(ctx); err != nil { @@ -176,7 +178,7 @@ func (p *Persistence) UpdateAtespaceAccessPolicy(ctx context.Context, name strin return nil, err } dbPolicy.Metadata = oldMeta - setUpdateMetadata(dbPolicy.Metadata, oldMeta) + storesql.SetUpdateMetadata(dbPolicy.Metadata, oldMeta) protoBytes, err := proto.Marshal(dbPolicy) if err != nil { return nil, fmt.Errorf("marshaling updated access policy: %w", err) @@ -187,7 +189,7 @@ func (p *Persistence) UpdateAtespaceAccessPolicy(ctx context.Context, name strin if err != nil { return nil, fmt.Errorf("updating access policy for %s: %w", name, err) } - if err := p.policyManager.ReconcileAtespaceBindings(ctx, tx, name, dbPolicy.GetBindings()); err != nil { + if err := p.policyManager.ReconcileAtespaceBindings(ctx, authz.PgxTx(tx), name, dbPolicy.GetBindings()); err != nil { return nil, fmt.Errorf("reconciling access policy bindings for %s: %w", name, err) } if err := tx.Commit(ctx); err != nil { @@ -223,7 +225,7 @@ func (p *Persistence) DeleteAtespaceAccessPolicy(ctx context.Context, name strin if err != nil { return nil, err } - if err := p.policyManager.DeleteAtespacePolicies(ctx, tx, name); err != nil { + if err := p.policyManager.DeleteAtespacePolicies(ctx, authz.PgxTx(tx), name); err != nil { return nil, fmt.Errorf("deleting authorization tuples for %s: %w", name, err) } if err := tx.Commit(ctx); err != nil { @@ -247,10 +249,10 @@ func getAccessPolicyRow(ctx context.Context, q querier, query string, args ...an func unmarshalAccessPolicy(uid string, version int64, protoBytes []byte) (*ateapipb.AccessPolicy, error) { policy := &ateapipb.AccessPolicy{} - if err := unmarshalStored(protoBytes, policy); err != nil { + if err := storesql.UnmarshalStored(protoBytes, policy); err != nil { return nil, fmt.Errorf("unmarshaling access policy: %w", err) } - if err := validateProtoMetadataMatchesColumns("access policy", policy.GetMetadata(), uid, version); err != nil { + if err := storesql.ValidateProtoMetadataMatchesColumns("access policy", policy.GetMetadata(), uid, version); err != nil { return nil, err } return policy, nil diff --git a/cmd/ateapi/internal/store/atepg/access_policy_test.go b/cmd/ateapi/internal/store/atepg/access_policy_test.go index 5f646e54f0..137e402b54 100644 --- a/cmd/ateapi/internal/store/atepg/access_policy_test.go +++ b/cmd/ateapi/internal/store/atepg/access_policy_test.go @@ -27,129 +27,6 @@ import ( "google.golang.org/protobuf/testing/protocmp" ) -func TestGlobalAccessPolicy_Lifecycle(t *testing.T) { - p := setupPostgresPersistence(t) - ctx := t.Context() - - if _, err := p.GetGlobalAccessPolicy(ctx); !errors.Is(err, store.ErrNotFound) { - t.Fatalf("GetGlobalAccessPolicy before create = %v, want ErrNotFound", err) - } - - policy := &ateapipb.AccessPolicy{ - Bindings: []*ateapipb.Binding{ - {Role: authz.RoleViewer, Members: []string{"user:bob"}}, - {Role: authz.RoleOwner, Members: []string{"user:alice"}}, - }, - } - created, err := p.CreateGlobalAccessPolicy(ctx, policy) - if err != nil { - t.Fatalf("CreateGlobalAccessPolicy failed: %v", err) - } - if created.GetMetadata().GetName() != "default" || created.GetMetadata().GetVersion() != 1 || created.GetMetadata().GetUid() == "" { - t.Fatalf("unexpected created metadata: %+v", created.GetMetadata()) - } - if diff := cmp.Diff(policy.GetBindings(), created.GetBindings(), protocmp.Transform()); diff != "" { - t.Errorf("created bindings (-want +got):\n%s", diff) - } - - if _, err := p.CreateGlobalAccessPolicy(ctx, policy); !errors.Is(err, store.ErrAlreadyExists) { - t.Fatalf("second CreateGlobalAccessPolicy = %v, want ErrAlreadyExists", err) - } - - got, err := p.GetGlobalAccessPolicy(ctx) - if err != nil || !cmp.Equal(got, created, protocmp.Transform()) { - t.Fatalf("GetGlobalAccessPolicy = %v, %v; want %v", got, err, created) - } - - if _, err := p.UpdateGlobalAccessPolicy(ctx, store.Precondition{}, func(*ateapipb.AccessPolicy) error { return nil }); !errors.Is(err, store.ErrPreconditionRequired) { - t.Fatalf("UpdateGlobalAccessPolicy without precondition = %v, want ErrPreconditionRequired", err) - } - if _, err := p.UpdateGlobalAccessPolicy(ctx, store.Precondition{UID: created.GetMetadata().GetUid(), Version: 99}, func(*ateapipb.AccessPolicy) error { return nil }); !errors.Is(err, store.ErrVersionConflict) { - t.Fatalf("UpdateGlobalAccessPolicy wrong version = %v, want ErrVersionConflict", err) - } - if _, err := p.UpdateGlobalAccessPolicy(ctx, store.Precondition{UID: "wrong-uid", Version: 1}, func(*ateapipb.AccessPolicy) error { return nil }); !errors.Is(err, store.ErrUIDConflict) { - t.Fatalf("UpdateGlobalAccessPolicy wrong uid = %v, want ErrUIDConflict", err) - } - - updated, err := p.UpdateGlobalAccessPolicy(ctx, store.PreconditionFrom(created), func(toUpdate *ateapipb.AccessPolicy) error { - toUpdate.Bindings = []*ateapipb.Binding{ - {Role: authz.RoleOwner, Members: []string{"user:alice", "user:carol"}}, - } - return nil - }) - if err != nil || updated.GetMetadata().GetVersion() != 2 { - t.Fatalf("UpdateGlobalAccessPolicy = %v, %v; want version 2", updated, err) - } -} - -func TestAtespaceAccessPolicy_LifecycleAndCascade(t *testing.T) { - p := setupPostgresPersistence(t) - ctx := t.Context() - - policy := &ateapipb.AccessPolicy{ - Bindings: []*ateapipb.Binding{ - {Role: authz.RoleEditor, Members: []string{"user:bob"}}, - }, - } - if _, err := p.CreateAtespaceAccessPolicy(ctx, "missing-space", policy); !errors.Is(err, store.ErrFailedPrecondition) { - t.Fatalf("CreateAtespaceAccessPolicy on missing atespace = %v, want ErrFailedPrecondition", err) - } - - if _, err := p.CreateAtespace(ctx, newTestAtespace("team-a")); err != nil { - t.Fatalf("CreateAtespace failed: %v", err) - } - if _, err := p.GetAtespaceAccessPolicy(ctx, "team-a"); !errors.Is(err, store.ErrNotFound) { - t.Fatalf("GetAtespaceAccessPolicy before create = %v, want ErrNotFound", err) - } - - created, err := p.CreateAtespaceAccessPolicy(ctx, "team-a", policy) - if err != nil { - t.Fatalf("CreateAtespaceAccessPolicy failed: %v", err) - } - if _, err := p.CreateAtespaceAccessPolicy(ctx, "team-a", policy); !errors.Is(err, store.ErrAlreadyExists) { - t.Fatalf("second CreateAtespaceAccessPolicy = %v, want ErrAlreadyExists", err) - } - - updated, err := p.UpdateAtespaceAccessPolicy(ctx, "team-a", store.PreconditionFrom(created), func(toUpdate *ateapipb.AccessPolicy) error { - toUpdate.Bindings = []*ateapipb.Binding{ - {Role: authz.RoleViewer, Members: []string{"user:dave"}}, - } - return nil - }) - if err != nil || updated.GetMetadata().GetVersion() != 2 { - t.Fatalf("UpdateAtespaceAccessPolicy = %v, %v; want version 2", updated, err) - } - - if _, err := p.DeleteAtespaceAccessPolicy(ctx, "team-a", store.DeletePreconditions{Version: 99}); !errors.Is(err, store.ErrVersionConflict) { - t.Fatalf("DeleteAtespaceAccessPolicy wrong version = %v, want ErrVersionConflict", err) - } - if _, err := p.DeleteAtespaceAccessPolicy(ctx, "team-a", store.DeletePreconditions{UID: "wrong-uid"}); !errors.Is(err, store.ErrUIDConflict) { - t.Fatalf("DeleteAtespaceAccessPolicy wrong uid = %v, want ErrUIDConflict", err) - } - - deleted, err := p.DeleteAtespaceAccessPolicy(ctx, "team-a", store.DeletePreconditions{ - UID: updated.GetMetadata().GetUid(), - Version: updated.GetMetadata().GetVersion(), - }) - if err != nil || !cmp.Equal(deleted, updated, protocmp.Transform()) { - t.Fatalf("DeleteAtespaceAccessPolicy = %v, %v; want %v", deleted, err, updated) - } - if _, err := p.GetAtespaceAccessPolicy(ctx, "team-a"); !errors.Is(err, store.ErrNotFound) { - t.Fatalf("GetAtespaceAccessPolicy after delete = %v, want ErrNotFound", err) - } - - // Recreate policy and verify DeleteAtespace cascades to atespace_access_policies. - if _, err := p.CreateAtespaceAccessPolicy(ctx, "team-a", policy); err != nil { - t.Fatalf("re-creating access policy failed: %v", err) - } - if _, err := p.DeleteAtespace(ctx, "team-a", store.DeletePreconditions{}); err != nil { - t.Fatalf("DeleteAtespace failed: %v", err) - } - if _, err := p.GetAtespaceAccessPolicy(ctx, "team-a"); !errors.Is(err, store.ErrNotFound) { - t.Fatalf("GetAtespaceAccessPolicy after DeleteAtespace = %v, want ErrNotFound", err) - } -} - // globalTuples returns the stored OpenFGA tuples on global:root as // "relation user" strings, read straight from the tuple table. func globalTuples(t *testing.T, p *Persistence) []string { diff --git a/cmd/ateapi/internal/store/atepg/actor.go b/cmd/ateapi/internal/store/atepg/actor.go index 538d263511..def83da75c 100644 --- a/cmd/ateapi/internal/store/atepg/actor.go +++ b/cmd/ateapi/internal/store/atepg/actor.go @@ -20,6 +20,7 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" @@ -35,7 +36,7 @@ func (p *Persistence) CreateActor(ctx context.Context, actor *ateapipb.Actor) (* // mutate it in place. This breaks some of the contract tests, so we can // fix it later. dbActor := proto.Clone(actor).(*ateapipb.Actor) - setCreateMetadata(dbActor.Metadata) + storesql.SetCreateMetadata(dbActor.Metadata) protoBytes, err := proto.Marshal(dbActor) if err != nil { @@ -70,7 +71,7 @@ func (p *Persistence) GetActor(ctx context.Context, actorRef resources.ActorRef) return nil, fmt.Errorf("getting actor %s/%s: %w", actorRef.Atespace, actorRef.Name, err) } out := &ateapipb.Actor{} - if err := unmarshalStored(protoBytes, out); err != nil { + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { return nil, fmt.Errorf("unmarshaling actor: %w", err) } return out, nil @@ -94,10 +95,10 @@ func (p *Persistence) UpdateActor(ctx context.Context, actorRef resources.ActorR } dbActor := &ateapipb.Actor{} - if err := unmarshalStored(currentBytes, dbActor); err != nil { + if err := storesql.UnmarshalStored(currentBytes, dbActor); err != nil { return nil, fmt.Errorf("unmarshaling actor for update: %w", err) } - if err := validateProtoMetadataMatchesColumns("actor "+actorRef.String(), dbActor.GetMetadata(), currentUID, currentVersion); err != nil { + if err := storesql.ValidateProtoMetadataMatchesColumns("actor "+actorRef.String(), dbActor.GetMetadata(), currentUID, currentVersion); err != nil { return nil, err } if err := precondition.Check(dbActor.GetMetadata()); err != nil { @@ -109,7 +110,7 @@ func (p *Persistence) UpdateActor(ctx context.Context, actorRef resources.ActorR } // Stored metadata is authoritative; discard any metadata edits made by the // closure and derive the next revision from the state this attempt read. - setUpdateMetadata(dbActor.Metadata, oldMeta) + storesql.SetUpdateMetadata(dbActor.Metadata, oldMeta) updatedBytes, err := proto.Marshal(dbActor) if err != nil { @@ -151,7 +152,7 @@ func (p *Persistence) DeleteActor(ctx context.Context, actorRef resources.ActorR return nil, fmt.Errorf("deleting actor %s/%s: %w", atespace, name, err) } out := &ateapipb.Actor{} - if err := unmarshalStored(protoBytes, out); err != nil { + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { return nil, fmt.Errorf("unmarshaling deleted actor: %w", err) } return out, nil @@ -176,7 +177,7 @@ func (p *Persistence) ListActors(ctx context.Context, atespace string, opts stor } func (p *Persistence) listActorsScoped(ctx context.Context, atespace string, pageSize int32, pageTokenStr string) ([]*ateapipb.Actor, string, error) { - token, err := decodePageToken(pageTokenStr, kindActor, atespace, 1) + token, err := storesql.DecodePageToken(pageTokenStr, storesql.KindActor, atespace, 1) if err != nil { return nil, "", err } @@ -204,7 +205,7 @@ func (p *Persistence) listActorsScoped(ctx context.Context, atespace string, pag return nil, "", fmt.Errorf("scanning actor row: %w", err) } a := &ateapipb.Actor{} - if err := unmarshalRow(protoBytes, a, "actor", atespace, name); err != nil { + if err := storesql.UnmarshalRow(protoBytes, a, "actor", atespace, name); err != nil { return nil, "", err } result = append(result, a) @@ -217,13 +218,13 @@ func (p *Persistence) listActorsScoped(ctx context.Context, atespace string, pag var nextToken string if len(result) > int(pageSize) { result = result[:pageSize] - nextToken = encodePageToken(kindActor, atespace, []string{names[pageSize-1]}) + nextToken = storesql.EncodePageToken(storesql.KindActor, atespace, []string{names[pageSize-1]}) } return result, nextToken, nil } func (p *Persistence) listActorsGlobal(ctx context.Context, pageSize int32, pageTokenStr string) ([]*ateapipb.Actor, string, error) { - token, err := decodePageToken(pageTokenStr, kindActor, "", 2) + token, err := storesql.DecodePageToken(pageTokenStr, storesql.KindActor, "", 2) if err != nil { return nil, "", err } @@ -252,7 +253,7 @@ func (p *Persistence) listActorsGlobal(ctx context.Context, pageSize int32, page return nil, "", fmt.Errorf("scanning actor row: %w", err) } a := &ateapipb.Actor{} - if err := unmarshalRow(protoBytes, a, "actor", k.atespace, k.name); err != nil { + if err := storesql.UnmarshalRow(protoBytes, a, "actor", k.atespace, k.name); err != nil { return nil, "", err } result = append(result, a) @@ -266,7 +267,7 @@ func (p *Persistence) listActorsGlobal(ctx context.Context, pageSize int32, page if len(result) > int(pageSize) { result = result[:pageSize] last := keys[pageSize-1] - nextToken = encodePageToken(kindActor, "", []string{last.atespace, last.name}) + nextToken = storesql.EncodePageToken(storesql.KindActor, "", []string{last.atespace, last.name}) } return result, nextToken, nil } diff --git a/cmd/ateapi/internal/store/atepg/actor_template.go b/cmd/ateapi/internal/store/atepg/actor_template.go index 8a480ade6e..e3e633052d 100644 --- a/cmd/ateapi/internal/store/atepg/actor_template.go +++ b/cmd/ateapi/internal/store/atepg/actor_template.go @@ -20,6 +20,7 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" @@ -32,7 +33,7 @@ func (p *Persistence) CreateActorTemplate(ctx context.Context, template *ateapip if dbTemplate.Metadata == nil { dbTemplate.Metadata = &ateapipb.ResourceMetadata{} } - setCreateMetadata(dbTemplate.Metadata) + storesql.SetCreateMetadata(dbTemplate.Metadata) protoBytes, err := proto.Marshal(dbTemplate) if err != nil { return nil, fmt.Errorf("marshaling actor template: %w", err) @@ -63,22 +64,12 @@ func (p *Persistence) GetActorTemplate(ctx context.Context, templateRef resource return nil, fmt.Errorf("getting actor template %s: %w", templateRef, err) } out := &ateapipb.ActorTemplate{} - if err := unmarshalStored(protoBytes, out); err != nil { + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { return nil, fmt.Errorf("unmarshaling actor template: %w", err) } return out, nil } -func validateUpdateActorTemplateMutation(storedTemplate, mutatedTemplate *ateapipb.ActorTemplate) error { - if stored, mutated := storedTemplate.GetMetadata().GetAtespace(), mutatedTemplate.GetMetadata().GetAtespace(); stored != mutated { - return fmt.Errorf("metadata.atespace is immutable: mutation changed it from %q to %q", stored, mutated) - } - if stored, mutated := storedTemplate.GetMetadata().GetName(), mutatedTemplate.GetMetadata().GetName(); stored != mutated { - return fmt.Errorf("metadata.name is immutable: mutation changed it from %q to %q", stored, mutated) - } - return nil -} - func (p *Persistence) UpdateActorTemplate(ctx context.Context, templateRef resources.ActorTemplateRef, precondition store.Precondition, mutate func(*ateapipb.ActorTemplate) error) (*ateapipb.ActorTemplate, error) { if err := precondition.Validate(); err != nil { return nil, err @@ -96,10 +87,10 @@ func (p *Persistence) UpdateActorTemplate(ctx context.Context, templateRef resou } dbTemplate := &ateapipb.ActorTemplate{} - if err := unmarshalStored(currentBytes, dbTemplate); err != nil { + if err := storesql.UnmarshalStored(currentBytes, dbTemplate); err != nil { return nil, fmt.Errorf("unmarshaling actor template for update: %w", err) } - if err := validateProtoMetadataMatchesColumns("actor template "+templateRef.String(), dbTemplate.GetMetadata(), currentUID, currentVersion); err != nil { + if err := storesql.ValidateProtoMetadataMatchesColumns("actor template "+templateRef.String(), dbTemplate.GetMetadata(), currentUID, currentVersion); err != nil { return nil, err } if err := precondition.Check(dbTemplate.GetMetadata()); err != nil { @@ -109,13 +100,13 @@ func (p *Persistence) UpdateActorTemplate(ctx context.Context, templateRef resou if err := mutate(dbTemplate); err != nil { return nil, err } - if err := validateUpdateActorTemplateMutation(templateBeforeMutation, dbTemplate); err != nil { + if err := storesql.ValidateUpdateActorTemplateMutation(templateBeforeMutation, dbTemplate); err != nil { return nil, err } if dbTemplate.Metadata == nil { dbTemplate.Metadata = &ateapipb.ResourceMetadata{} } - setUpdateMetadata(dbTemplate.Metadata, templateBeforeMutation.GetMetadata()) + storesql.SetUpdateMetadata(dbTemplate.Metadata, templateBeforeMutation.GetMetadata()) updatedBytes, err := proto.Marshal(dbTemplate) if err != nil { return nil, fmt.Errorf("marshaling actor template: %w", err) @@ -146,7 +137,7 @@ func (p *Persistence) ListActorTemplates(ctx context.Context, atespace string, o if atespace != "" { keyParts = 1 } - token, err := decodePageToken(pageTokenStr, kindActorTemplate, atespace, keyParts) + token, err := storesql.DecodePageToken(pageTokenStr, storesql.KindActorTemplate, atespace, keyParts) if err != nil { return store.ListResponse[*ateapipb.ActorTemplate]{}, err } @@ -186,7 +177,7 @@ func (p *Persistence) ListActorTemplates(ctx context.Context, atespace string, o return store.ListResponse[*ateapipb.ActorTemplate]{}, fmt.Errorf("scanning actor template row: %w", err) } template := &ateapipb.ActorTemplate{} - if err := unmarshalRow(protoBytes, template, "actor template", k.atespace, k.name); err != nil { + if err := storesql.UnmarshalRow(protoBytes, template, "actor template", k.atespace, k.name); err != nil { return store.ListResponse[*ateapipb.ActorTemplate]{}, err } keys = append(keys, k) @@ -203,7 +194,7 @@ func (p *Persistence) ListActorTemplates(ctx context.Context, atespace string, o if atespace != "" { lastParts = []string{last.name} } - nextToken = encodePageToken(kindActorTemplate, atespace, lastParts) + nextToken = storesql.EncodePageToken(storesql.KindActorTemplate, atespace, lastParts) } return store.ListResponse[*ateapipb.ActorTemplate]{Items: result, NextPageToken: nextToken}, nil } @@ -226,7 +217,7 @@ func (p *Persistence) DeleteActorTemplate(ctx context.Context, templateRef resou return nil, fmt.Errorf("deleting actor template %s: %w", templateRef, err) } out := &ateapipb.ActorTemplate{} - if err := unmarshalStored(protoBytes, out); err != nil { + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { return nil, fmt.Errorf("unmarshaling deleted actor template: %w", err) } return out, nil diff --git a/cmd/ateapi/internal/store/atepg/actor_template_test.go b/cmd/ateapi/internal/store/atepg/actor_template_test.go index 0b683ce52a..f86bbe2267 100644 --- a/cmd/ateapi/internal/store/atepg/actor_template_test.go +++ b/cmd/ateapi/internal/store/atepg/actor_template_test.go @@ -16,15 +16,9 @@ package atepg import ( "context" - "errors" - "fmt" "testing" - "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" - "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" - "github.com/google/go-cmp/cmp" - "google.golang.org/protobuf/testing/protocmp" ) func createTestActorTemplate(t *testing.T, s *Persistence, atespace, name string) { @@ -35,84 +29,3 @@ func createTestActorTemplate(t *testing.T, s *Persistence, atespace, name string t.Fatalf("CreateActorTemplate(%q/%q) failed: %v", atespace, name, err) } } - -func TestUpdateActorTemplate_ConcurrentWriteReturnsConflict(t *testing.T) { - s := setupPostgresPersistence(t) - ctx := context.Background() - createTestAtespace(t, s, "team-a") - createTestActorTemplate(t, s, "team-a", "template-a") - templateRef := resources.ActorTemplateRef{Atespace: "team-a", Name: "template-a"} - created, err := s.GetActorTemplate(ctx, templateRef) - if err != nil { - t.Fatalf("GetActorTemplate failed: %v", err) - } - - mutations := 0 - _, err = s.UpdateActorTemplate(ctx, templateRef, store.PreconditionFrom(created), func(toUpdate *ateapipb.ActorTemplate) error { - mutations++ - if _, err := s.UpdateActorTemplate(ctx, templateRef, store.PreconditionFrom(created), func(concurrent *ateapipb.ActorTemplate) error { - concurrent.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid"}} - return nil - }); err != nil { - return fmt.Errorf("concurrent actor template update: %w", err) - } - toUpdate.Status = &ateapipb.ActorTemplateStatus{GoldenSnapshotStatus: &ateapipb.GoldenSnapshotStatus{ - ErrorMessage: "LosingUpdate", - }} - return nil - }) - if !errors.Is(err, store.ErrVersionConflict) { - t.Fatalf("UpdateActorTemplate error = %v, want ErrVersionConflict", err) - } - if mutations != 1 { - t.Errorf("mutation ran %d times, want 1", mutations) - } - stored, err := s.GetActorTemplate(ctx, templateRef) - if err != nil { - t.Fatalf("GetActorTemplate failed: %v", err) - } - if got := stored.GetWorkerSelector().GetMatchLabels()["tier"]; got != "paid" { - t.Errorf("worker selector tier = %q, want paid", got) - } - if got := stored.GetStatus().GetGoldenSnapshotStatus().GetErrorMessage(); got != "" { - t.Errorf("status error message = %q, want empty: losing update was persisted", got) - } -} - -func TestGetActorTemplate_BackfillsDefaults(t *testing.T) { - s := setupPostgresPersistence(t) - ctx := context.Background() - createTestAtespace(t, s, "team-a") - - if _, err := s.CreateActorTemplate(ctx, &ateapipb.ActorTemplate{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "template-a"}, - SnapshotConfig: &ateapipb.SnapshotConfig{}, - }); err != nil { - t.Fatalf("CreateActorTemplate failed: %v", err) - } - - templateRef := resources.ActorTemplateRef{Atespace: "team-a", Name: "template-a"} - got, err := s.GetActorTemplate(ctx, templateRef) - if err != nil { - t.Fatalf("GetActorTemplate failed: %v", err) - } - want := &ateapipb.SnapshotConfig{ - OnPause: ateapipb.SnapshotContentScope_SNAPSHOT_CONTENT_SCOPE_FULL, - OnCommit: ateapipb.SnapshotContentScope_SNAPSHOT_CONTENT_SCOPE_FULL, - } - if diff := cmp.Diff(want, got.GetSnapshotConfig(), protocmp.Transform()); diff != "" { - t.Errorf("GetActorTemplate did not backfill snapshot_config defaults (-want +got):\n%s", diff) - } - - // ListActorTemplates decodes rows on its own path; it must backfill too. - page, err := s.ListActorTemplates(ctx, "team-a", store.ListOptions{PageSize: 10}) - if err != nil { - t.Fatalf("ListActorTemplates failed: %v", err) - } - if len(page.Items) != 1 { - t.Fatalf("ListActorTemplates returned %d items, want 1", len(page.Items)) - } - if diff := cmp.Diff(want, page.Items[0].GetSnapshotConfig(), protocmp.Transform()); diff != "" { - t.Errorf("ListActorTemplates did not backfill snapshot_config defaults (-want +got):\n%s", diff) - } -} diff --git a/cmd/ateapi/internal/store/atepg/actor_test.go b/cmd/ateapi/internal/store/atepg/actor_test.go deleted file mode 100644 index 337a758663..0000000000 --- a/cmd/ateapi/internal/store/atepg/actor_test.go +++ /dev/null @@ -1,145 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package atepg - -import ( - "context" - "errors" - "fmt" - "testing" - - "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" - "github.com/agent-substrate/substrate/internal/resources" - "github.com/agent-substrate/substrate/pkg/proto/ateapipb" -) - -func TestUpdateActor_ConcurrentWriteReturnsConflict(t *testing.T) { - s := setupPostgresPersistence(t) - ctx := context.Background() - createTestAtespace(t, s, "team-a") - created, err := s.CreateActor(ctx, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-a"}, - ActorTemplate: &ateapipb.ObjectRef{Atespace: "default", Name: "template-a"}, - Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, - }) - if err != nil { - t.Fatalf("CreateActor failed: %v", err) - } - actorRef := resources.ActorRefFromActor(created) - - mutations := 0 - _, err = s.UpdateActor(ctx, actorRef, store.PreconditionFrom(created), func(toUpdate *ateapipb.Actor) error { - mutations++ - if _, err := s.UpdateActor(ctx, actorRef, store.PreconditionFrom(created), func(concurrent *ateapipb.Actor) error { - concurrent.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid"}} - return nil - }); err != nil { - return fmt.Errorf("concurrent actor update: %w", err) - } - toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_RUNNING - return nil - }) - if !errors.Is(err, store.ErrVersionConflict) { - t.Fatalf("UpdateActor error = %v, want ErrVersionConflict", err) - } - if mutations != 1 { - t.Errorf("mutation ran %d times, want 1", mutations) - } - stored, err := s.GetActor(ctx, actorRef) - if err != nil { - t.Fatalf("GetActor failed: %v", err) - } - if stored.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { - t.Errorf("state = %v, want SUSPENDED: losing update was persisted", stored.GetStatus().GetState()) - } - if got := stored.GetWorkerSelector().GetMatchLabels()["tier"]; got != "paid" { - t.Errorf("worker selector tier = %q, want paid", got) - } - if got, want := stored.GetMetadata().GetVersion(), created.GetMetadata().GetVersion()+1; got != want { - t.Errorf("version = %d, want %d", got, want) - } -} - -// TestCreateActor_MissingAtespace_FailedPrecondition exercises the -// foreign-key race the doc calls out: CreateActor rejects an actor whose -// atespace doesn't exist (including a concurrently-deleted one), with the -// foreign key closing the TOCTOU window around a separate existence check. -func TestCreateActor_MissingAtespace_FailedPrecondition(t *testing.T) { - s := setupPostgresPersistence(t) - ctx := context.Background() - - actor := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Name: "id1", Atespace: "no-such-atespace"}, - ActorTemplate: &ateapipb.ObjectRef{Atespace: "ns1", Name: "tmpl1"}, - Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}, - } - if _, err := s.CreateActor(ctx, actor); !errors.Is(err, store.ErrFailedPrecondition) { - t.Errorf("CreateActor with missing atespace = %v, want ErrFailedPrecondition", err) - } -} - -func TestListActors_InvalidPageToken(t *testing.T) { - s := setupPostgresPersistence(t) - ctx := context.Background() - - if _, err := s.ListActors(ctx, "", store.ListOptions{PageSize: 10, PageToken: "not-valid-base64!!"}); err == nil { - t.Errorf("ListActors with malformed page token = nil error, want an error") - } -} - -func TestListActors_CrossScopePageToken(t *testing.T) { - s := setupPostgresPersistence(t) - ctx := context.Background() - - if _, err := s.CreateAtespace(ctx, newTestAtespace("team-a")); err != nil { - t.Fatalf("CreateAtespace(team-a) failed: %v", err) - } - if _, err := s.CreateAtespace(ctx, newTestAtespace("team-b")); err != nil { - t.Fatalf("CreateAtespace(team-b) failed: %v", err) - } - for _, name := range []string{"a1", "a2"} { - if _, err := s.CreateActor(ctx, &ateapipb.Actor{Metadata: &ateapipb.ResourceMetadata{Name: name, Atespace: "team-a"}, Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED}}); err != nil { - t.Fatalf("CreateActor failed: %v", err) - } - } - - page, err := s.ListActors(ctx, "team-a", store.ListOptions{PageSize: 1}) - if err != nil { - t.Fatalf("ListActors(team-a) failed: %v", err) - } - if page.NextPageToken == "" { - t.Fatalf("expected a next page token") - } - - // A token minted for team-a must be rejected when replayed against team-b - // or against the unscoped (global) listing. - if _, err := s.ListActors(ctx, "team-b", store.ListOptions{PageSize: 1, PageToken: page.NextPageToken}); err == nil { - t.Errorf("ListActors(team-b) with team-a's token = nil error, want an error") - } - if _, err := s.ListActors(ctx, "", store.ListOptions{PageSize: 1, PageToken: page.NextPageToken}); err == nil { - t.Errorf("ListActors(all) with team-a's token = nil error, want an error") - } - - // A worker-list token must be rejected by ListAtespaces (different kind). - workerPage, err := s.ListWorkers(ctx, store.ListOptions{PageSize: 1}) - if err != nil { - t.Fatalf("ListWorkers failed: %v", err) - } - if workerPage.NextPageToken != "" { - if _, err := s.ListAtespaces(ctx, store.ListOptions{PageSize: 1, PageToken: workerPage.NextPageToken}); err == nil { - t.Errorf("ListAtespaces with a worker page token = nil error, want an error") - } - } -} diff --git a/cmd/ateapi/internal/store/atepg/atepg.go b/cmd/ateapi/internal/store/atepg/atepg.go index 63db45d5f9..3c0821739a 100644 --- a/cmd/ateapi/internal/store/atepg/atepg.go +++ b/cmd/ateapi/internal/store/atepg/atepg.go @@ -26,20 +26,15 @@ import ( "errors" "fmt" "log/slog" - "strings" - "sync" "time" "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" - "github.com/agent-substrate/substrate/cmd/ateapi/internal/defaults" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" - "github.com/google/uuid" "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgconn" "github.com/jackc/pgx/v5/pgxpool" - "google.golang.org/protobuf/proto" - "google.golang.org/protobuf/types/known/timestamppb" ) // Persistence is a service that stores ate state in PostgreSQL. @@ -67,46 +62,8 @@ type Persistence struct { pollFailureCloseAfter time.Duration stopMaintenance context.CancelFunc maintenanceDone chan struct{} - // watchMu guards watchers, the live WatchWorkers channels. - watchMu sync.Mutex - watchers map[chan store.WorkerEvent]struct{} -} - -// addWatcher enrolls a WatchWorkers channel to receive locally published events. -func (p *Persistence) addWatcher(ch chan store.WorkerEvent) { - p.watchMu.Lock() - defer p.watchMu.Unlock() - p.watchers[ch] = struct{}{} -} - -// removeWatcher unenrolls a channel. The caller must call it before closing -// the channel: once it returns, publishLocally can no longer send on it. -func (p *Persistence) removeWatcher(ch chan store.WorkerEvent) { - p.watchMu.Lock() - defer p.watchMu.Unlock() - delete(p.watchers, ch) -} - -// publishLocally hands a committed event to this process's watchers a poll -// interval ahead of the outbox, one copy each. Sends are non-blocking: a -// watcher with a full buffer is skipped and gets the event from the outbox. -func (p *Persistence) publishLocally(ctx context.Context, payload []byte) { - p.watchMu.Lock() - defer p.watchMu.Unlock() - if len(p.watchers) == 0 { - return - } - event, err := unmarshalWorkerEvent(payload) - if err != nil { - slog.ErrorContext(ctx, "decoding locally published worker event failed", slog.Any("err", err)) - return - } - for ch := range p.watchers { - select { - case ch <- store.WorkerEvent{Type: event.Type, Worker: proto.Clone(event.Worker).(*ateapipb.Worker)}: - default: - } - } + // watchers are the live WatchWorkers channels. + watchers storesql.Watchers } var _ store.Interface = (*Persistence)(nil) @@ -282,11 +239,10 @@ func newPersistence(ctx context.Context, pool, watchPool, ownerPool *pgxpool.Poo pool: pool, watchPool: watchPool, ownerPool: ownerPool, - leaseTTL: defaultLeaseTTL, + leaseTTL: storesql.DefaultLeaseTTL, pollFailureCloseAfter: outboxPollFailureCloseAfter, stopMaintenance: stopMaintenance, maintenanceDone: make(chan struct{}), - watchers: make(map[chan store.WorkerEvent]struct{}), } // Cover the partition lead before accepting writes; from then on the // maintenance loop keeps partitions ahead of the clock (and the @@ -342,53 +298,6 @@ type querier interface { Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error) } -// unmarshalStored decodes a stored proto, dropping fields this binary has no -// descriptor for. This means a newer replica can have written such a field. -// It also backfills defaults to make all resources are properly defaulted, even -// the ones stored before a field with defaults was introduced. -func unmarshalStored(b []byte, m proto.Message) error { - if err := (proto.UnmarshalOptions{DiscardUnknown: true}).Unmarshal(b, m); err != nil { - return err - } - defaults.Apply(m) - return nil -} - -// unmarshalRow is unmarshalStored for a row in a listing. A listing fails as a -// whole on one bad row, so the error names the row. -func unmarshalRow(b []byte, m proto.Message, kind string, id ...string) error { - if err := unmarshalStored(b, m); err != nil { - return fmt.Errorf("unmarshaling %s %s: %w", kind, strings.Join(id, "/"), err) - } - return nil -} - -func setCreateMetadata(metadata *ateapipb.ResourceMetadata) { - metadata.Uid = uuid.NewString() - metadata.Version = 1 - metadata.CreateTime = timestamppb.Now() - metadata.UpdateTime = metadata.CreateTime -} - -// validateProtoMetadataMatchesColumns verifies that the metadata in the database -// matches the metadata in the proto. -func validateProtoMetadataMatchesColumns(resource string, metadata *ateapipb.ResourceMetadata, uid string, version int64) error { - if metadata.GetUid() != uid { - return fmt.Errorf("%s uid projection %q does not match proto metadata uid %q", resource, uid, metadata.GetUid()) - } - if metadata.GetVersion() != version { - return fmt.Errorf("%s version projection %d does not match proto metadata version %d", resource, version, metadata.GetVersion()) - } - return nil -} - -func setUpdateMetadata(newMeta, oldMeta *ateapipb.ResourceMetadata) { - newMeta.Uid = oldMeta.Uid - newMeta.Version = oldMeta.Version + 1 - newMeta.CreateTime = oldMeta.CreateTime - newMeta.UpdateTime = timestamppb.Now() -} - func mapDeleteError(err error, uid string, version int64, precondition store.DeletePreconditions) error { if errors.Is(err, pgx.ErrNoRows) { return store.ErrNotFound diff --git a/cmd/ateapi/internal/store/atepg/atespace.go b/cmd/ateapi/internal/store/atepg/atespace.go index 9d18c70329..d83e29590f 100644 --- a/cmd/ateapi/internal/store/atepg/atespace.go +++ b/cmd/ateapi/internal/store/atepg/atespace.go @@ -19,7 +19,9 @@ import ( "errors" "fmt" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" "google.golang.org/protobuf/proto" @@ -29,7 +31,7 @@ func (p *Persistence) CreateAtespace(ctx context.Context, atespace *ateapipb.Ate name := atespace.GetMetadata().GetName() dbAtespace := proto.Clone(atespace).(*ateapipb.Atespace) - setCreateMetadata(dbAtespace.Metadata) + storesql.SetCreateMetadata(dbAtespace.Metadata) protoBytes, err := proto.Marshal(dbAtespace) if err != nil { @@ -59,7 +61,7 @@ func (p *Persistence) GetAtespace(ctx context.Context, name string) (*ateapipb.A return nil, fmt.Errorf("getting atespace %q: %w", name, err) } out := &ateapipb.Atespace{} - if err := unmarshalStored(protoBytes, out); err != nil { + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { return nil, fmt.Errorf("unmarshaling atespace: %w", err) } return out, nil @@ -71,7 +73,7 @@ func (p *Persistence) ListAtespaces(ctx context.Context, opts store.ListOptions) return store.ListResponse[*ateapipb.Atespace]{}, err } pageSize, pageTokenStr := opts.PageSize, opts.PageToken - token, err := decodePageToken(pageTokenStr, kindAtespace, "", 1) + token, err := storesql.DecodePageToken(pageTokenStr, storesql.KindAtespace, "", 1) if err != nil { return store.ListResponse[*ateapipb.Atespace]{}, err } @@ -99,7 +101,7 @@ func (p *Persistence) ListAtespaces(ctx context.Context, opts store.ListOptions) return store.ListResponse[*ateapipb.Atespace]{}, fmt.Errorf("scanning atespace row: %w", err) } a := &ateapipb.Atespace{} - if err := unmarshalRow(protoBytes, a, "atespace", name); err != nil { + if err := storesql.UnmarshalRow(protoBytes, a, "atespace", name); err != nil { return store.ListResponse[*ateapipb.Atespace]{}, err } result = append(result, a) @@ -112,7 +114,7 @@ func (p *Persistence) ListAtespaces(ctx context.Context, opts store.ListOptions) var nextToken string if len(result) > int(pageSize) { result = result[:pageSize] - nextToken = encodePageToken(kindAtespace, "", []string{names[pageSize-1]}) + nextToken = storesql.EncodePageToken(storesql.KindAtespace, "", []string{names[pageSize-1]}) } return store.ListResponse[*ateapipb.Atespace]{Items: result, NextPageToken: nextToken}, nil } @@ -143,14 +145,14 @@ func (p *Persistence) DeleteAtespace(ctx context.Context, name string, precondit } return nil, fmt.Errorf("deleting atespace %q: %w", name, err) } - if err := p.policyManager.DeleteAtespacePolicies(ctx, tx, name); err != nil { + if err := p.policyManager.DeleteAtespacePolicies(ctx, authz.PgxTx(tx), name); err != nil { return nil, fmt.Errorf("deleting authorization tuples for atespace %q: %w", name, err) } if err := tx.Commit(ctx); err != nil { return nil, fmt.Errorf("committing atespace delete %q: %w", name, err) } out := &ateapipb.Atespace{} - if err := unmarshalStored(protoBytes, out); err != nil { + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { return nil, fmt.Errorf("unmarshaling deleted atespace: %w", err) } return out, nil diff --git a/cmd/ateapi/internal/store/atepg/common_test.go b/cmd/ateapi/internal/store/atepg/common_test.go index 6edd774f3e..7c690aa7d9 100644 --- a/cmd/ateapi/internal/store/atepg/common_test.go +++ b/cmd/ateapi/internal/store/atepg/common_test.go @@ -49,12 +49,12 @@ func setupPostgresPersistence(t *testing.T) *Persistence { // always does. func setTestPolicyManager(t *testing.T, p *Persistence) { t.Helper() - fgaServer, err := authz.NewOpenFGAServer(p.pool) + fgaServer, err := authz.NewOpenFGAServer(authz.PostgresBackend(p.pool)) if err != nil { t.Fatalf("NewOpenFGAServer failed: %v", err) } t.Cleanup(fgaServer.Close) - _, policyManager, err := authz.New(t.Context(), p.pool, fgaServer, nil) + _, policyManager, err := authz.New(t.Context(), authz.PostgresBackend(p.pool), fgaServer, nil) if err != nil { t.Fatalf("authz.New failed: %v", err) } diff --git a/cmd/ateapi/internal/store/atepg/egress_policy.go b/cmd/ateapi/internal/store/atepg/egress_policy.go index 70c679656b..8b2a438f7a 100644 --- a/cmd/ateapi/internal/store/atepg/egress_policy.go +++ b/cmd/ateapi/internal/store/atepg/egress_policy.go @@ -20,6 +20,7 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" @@ -28,7 +29,7 @@ import ( func (p *Persistence) CreateEgressPolicy(ctx context.Context, actorRef resources.ActorRef, policy *ateapipb.EgressPolicy) (*ateapipb.EgressPolicy, error) { dbPolicy := proto.Clone(policy).(*ateapipb.EgressPolicy) - setCreateMetadata(dbPolicy.Metadata) + storesql.SetCreateMetadata(dbPolicy.Metadata) protoBytes, err := proto.Marshal(dbPolicy) if err != nil { return nil, fmt.Errorf("marshaling egress policy: %w", err) @@ -74,7 +75,7 @@ func (p *Persistence) UpdateEgressPolicy(ctx context.Context, actorRef resources return nil, err } dbPolicy.Metadata = oldMeta - setUpdateMetadata(dbPolicy.Metadata, oldMeta) + storesql.SetUpdateMetadata(dbPolicy.Metadata, oldMeta) protoBytes, err := proto.Marshal(dbPolicy) if err != nil { return nil, fmt.Errorf("marshaling updated egress policy: %w", err) @@ -130,10 +131,10 @@ func getEgressPolicyRow(ctx context.Context, q querier, query string, args ...an func unmarshalEgressPolicy(uid string, version int64, protoBytes []byte) (*ateapipb.EgressPolicy, error) { policy := &ateapipb.EgressPolicy{} - if err := unmarshalStored(protoBytes, policy); err != nil { + if err := storesql.UnmarshalStored(protoBytes, policy); err != nil { return nil, fmt.Errorf("unmarshaling egress policy: %w", err) } - if err := validateProtoMetadataMatchesColumns("egress policy", policy.GetMetadata(), uid, version); err != nil { + if err := storesql.ValidateProtoMetadataMatchesColumns("egress policy", policy.GetMetadata(), uid, version); err != nil { return nil, err } return policy, nil diff --git a/cmd/ateapi/internal/store/atepg/lease.go b/cmd/ateapi/internal/store/atepg/lease.go index 4303cec0fe..097a6d4629 100644 --- a/cmd/ateapi/internal/store/atepg/lease.go +++ b/cmd/ateapi/internal/store/atepg/lease.go @@ -18,72 +18,20 @@ import ( "context" "errors" "fmt" - "log/slog" "time" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" - "github.com/google/uuid" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/jackc/pgx/v5" ) -// defaultLeaseTTL is how long a lease may go unrenewed before another client -// can reclaim it. -const defaultLeaseTTL = 30 * time.Second - -// leaseCleanupBatch bounds one DELETE of a cleanup pass, so a backlog of -// expired rows drains over several short statements rather than one long -// one holding many row locks. -const leaseCleanupBatch = 1000 - func (p *Persistence) AcquireLease(ctx context.Context, key string) (*store.Lease, error) { - ttl := p.leaseTTL - token := uuid.NewString() - // Acquisition runs before any workflow step span opens, so log the - // query's duration to make this window attributable. - t := time.Now() - acquired, err := p.acquireLease(ctx, key, token, ttl) - dAcquire := time.Since(t) - slog.InfoContext(ctx, "PostgreSQL lease acquisition finished", - slog.String("key", key), - slog.Bool("acquired", acquired && err == nil), - slog.Duration("acquire", dAcquire)) - if err != nil { - return nil, err - } - if !acquired { - return nil, store.ErrLeaseConflict - } - - leaseCtx, cancel := context.WithCancel(ctx) - renewalDone := make(chan struct{}) - go func() { - defer close(renewalDone) - defer cancel() - p.renewLeaseLoop(leaseCtx, key, token, ttl) - }() - - closeFn := func() { - // Close runs after the last workflow step span ends but inside the - // operation, so log its two waits: the renewal goroutine may be - // mid-query when cancelled, and the release DELETE may wait on a - // concurrent write to the same row. - t := time.Now() - cancel() - <-renewalDone - dRenewalStop := time.Since(t) - - releaseCtx, releaseCancel := context.WithTimeout(context.Background(), 5*time.Second) - defer releaseCancel() - t = time.Now() - if err := p.releaseLease(releaseCtx, key, token); err != nil { - slog.WarnContext(releaseCtx, "failed to release PostgreSQL lease, relying on TTL to reclaim it", "key", key, "error", err) - } - slog.InfoContext(releaseCtx, "PostgreSQL lease released", - slog.String("key", key), - slog.Duration("renewal_stop", dRenewalStop), - slog.Duration("release", time.Since(t))) - } - return store.NewLease(leaseCtx, closeFn), nil + return storesql.LeaseSQL{ + Database: "PostgreSQL", + Acquire: p.acquireLease, + Renew: p.renewLease, + Release: p.releaseLease, + }.AcquireLease(ctx, key, p.leaseTTL) } // cleanupExpiredLeases deletes expired lease rows and returns how many it @@ -104,12 +52,12 @@ func (p *Persistence) cleanupExpiredLeases(ctx context.Context) (int64, error) { SELECT key FROM leases WHERE expires_at <= clock_timestamp() LIMIT $1 - FOR UPDATE SKIP LOCKED)`, leaseCleanupBatch) + FOR UPDATE SKIP LOCKED)`, storesql.LeaseCleanupBatch) if err != nil { return deleted, fmt.Errorf("deleting expired leases: %w", err) } deleted += tag.RowsAffected() - if tag.RowsAffected() < leaseCleanupBatch { + if tag.RowsAffected() < storesql.LeaseCleanupBatch { return deleted, nil } } @@ -134,68 +82,6 @@ func (p *Persistence) acquireLease(ctx context.Context, key, token string, ttl t return true, nil } -const ( - renewIntervalDivisor = 3 - renewRetryPeriodDivisor = 10 - renewDeadlineFraction = 2.0 / 3.0 -) - -func (p *Persistence) renewLeaseLoop(ctx context.Context, key, token string, ttl time.Duration) { - interval := ttl / renewIntervalDivisor - renewDeadline := time.Duration(float64(ttl) * renewDeadlineFraction) - - lastRenewed := time.Now() - timer := time.NewTimer(interval) - defer timer.Stop() - - for { - select { - case <-ctx.Done(): - return - case <-timer.C: - renewCtx, cancel := context.WithDeadline(ctx, lastRenewed.Add(renewDeadline)) - renewed := p.tryRenewLease(renewCtx, key, token, ttl) - cancel() - if !renewed { - return - } - lastRenewed = time.Now() - timer.Reset(interval) - } - } -} - -func (p *Persistence) tryRenewLease(ctx context.Context, key, token string, ttl time.Duration) bool { - retryPeriod := ttl / renewRetryPeriodDivisor - retry := time.NewTimer(0) - defer retry.Stop() - - for { - select { - case <-ctx.Done(): - if errors.Is(ctx.Err(), context.DeadlineExceeded) { - slog.WarnContext(ctx, "failed to renew PostgreSQL lease before its deadline", "key", key) - } - return false - case <-retry.C: - renewed, err := p.renewLease(ctx, key, token, ttl) - if ctx.Err() != nil { - return false - } - switch { - case err == nil && renewed: - return true - case err == nil: - slog.WarnContext(ctx, "PostgreSQL lease renewal found lease no longer owned", "key", key) - return false - default: - slog.WarnContext(ctx, "failed to renew PostgreSQL lease, retrying", "key", key, "error", err) - retry.Reset(retryPeriod) - } - } - } -} - func (p *Persistence) renewLease(ctx context.Context, key, token string, ttl time.Duration) (bool, error) { var returnedKey string err := p.pool.QueryRow(ctx, ` diff --git a/cmd/ateapi/internal/store/atepg/lease_test.go b/cmd/ateapi/internal/store/atepg/lease_test.go index a68859b909..d965280f7c 100644 --- a/cmd/ateapi/internal/store/atepg/lease_test.go +++ b/cmd/ateapi/internal/store/atepg/lease_test.go @@ -22,6 +22,7 @@ import ( "time" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" ) // countLeases returns how many lease rows exist for key. @@ -83,7 +84,7 @@ func TestCleanupExpiredLeases_RemovesOnlyExpiredRows(t *testing.T) { func TestCleanupExpiredLeases_DrainsAcrossBatches(t *testing.T) { s := setupPostgresPersistence(t) ctx := context.Background() - const seeded = leaseCleanupBatch*2 + 7 + const seeded = storesql.LeaseCleanupBatch*2 + 7 if _, err := s.pool.Exec(ctx, ` INSERT INTO leases (key, token, expires_at) SELECT 'expired-' || i, 'old', clock_timestamp() - interval '1 minute' diff --git a/cmd/ateapi/internal/store/atepg/outbox.go b/cmd/ateapi/internal/store/atepg/outbox.go index ca9bdca410..7bd8dba1b8 100644 --- a/cmd/ateapi/internal/store/atepg/outbox.go +++ b/cmd/ateapi/internal/store/atepg/outbox.go @@ -29,45 +29,11 @@ import ( "time" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" - "google.golang.org/protobuf/proto" ) -// Outbox payload format: one event-type byte followed by the binary Worker proto. -// The tag byte is read by other replicas during rolling deploys, so -// store.WorkerEventType values must stay append-only stable and fit a byte. -func marshalWorkerEvent(eventType store.WorkerEventType, worker *ateapipb.Worker) ([]byte, error) { - b, err := proto.Marshal(worker) - if err != nil { - return nil, fmt.Errorf("in proto.Marshal: %w", err) - } - return append([]byte{byte(eventType)}, b...), nil -} - -func unmarshalWorkerEvent(payload []byte) (store.WorkerEvent, error) { - if len(payload) == 0 { - return store.WorkerEvent{}, fmt.Errorf("empty worker event payload") - } - // Assert invariants at the boundary. Corrupted payloads or unknown types - // must fail here to trigger a loud resync, rather than falling through - // downstream as silent no-ops. - eventType := store.WorkerEventType(payload[0]) - switch eventType { - case store.WorkerEventCreated, store.WorkerEventUpdated, store.WorkerEventDeleted: - default: - return store.WorkerEvent{}, fmt.Errorf("unknown worker event type byte %d", payload[0]) - } - worker := &ateapipb.Worker{} - if err := unmarshalStored(payload[1:], worker); err != nil { - return store.WorkerEvent{}, fmt.Errorf("in unmarshalStored: %w", err) - } - if worker.GetMetadata().GetName() == "" { - return store.WorkerEvent{}, fmt.Errorf("worker event payload has no worker name") - } - return store.WorkerEvent{Type: eventType, Worker: worker}, nil -} - // writeAndAppendEvent runs fn inside a transaction, then--only if fn reports a // worker worth publishing--appends the event to the worker_outbox table in the // same transaction, so watchers see it if and only if the transaction commits. @@ -90,7 +56,7 @@ func (p *Persistence) writeAndAppendEvent(ctx context.Context, eventType store.W var payload []byte if worker != nil { - payload, err = marshalWorkerEvent(eventType, worker) + payload, err = storesql.MarshalWorkerEvent(eventType, worker) if err != nil { return nil, fmt.Errorf("marshaling worker event: %w", err) } @@ -103,7 +69,7 @@ func (p *Persistence) writeAndAppendEvent(ctx context.Context, eventType store.W return nil, fmt.Errorf("committing transaction: %w", err) } if payload != nil { - p.publishLocally(ctx, payload) + p.watchers.Publish(ctx, payload) } return worker, nil } @@ -445,10 +411,10 @@ func (p *Persistence) WatchWorkers(ctx context.Context) (*store.WorkerWatch, err ch := make(chan store.WorkerEvent, 128) // Committed writes in this process are published straight onto ch, ahead // of the poll that would carry them. - p.addWatcher(ch) + p.watchers.Add(ch) go func() { defer func() { - p.removeWatcher(ch) + p.watchers.Remove(ch) close(ch) }() ticker := time.NewTicker(outboxPollInterval) @@ -537,7 +503,7 @@ func (p *Persistence) WatchWorkers(ctx context.Context) (*store.WorkerWatch, err } for _, r := range batch { - event, err := unmarshalWorkerEvent(r.payload) + event, err := storesql.UnmarshalWorkerEvent(r.payload) if err != nil { // Close to force a relist. Skipping it would cause silent // data loss. The fresh watch starts at the current xmin, diff --git a/cmd/ateapi/internal/store/atepg/outbox_test.go b/cmd/ateapi/internal/store/atepg/outbox_test.go index 1e133660e2..e7f113c570 100644 --- a/cmd/ateapi/internal/store/atepg/outbox_test.go +++ b/cmd/ateapi/internal/store/atepg/outbox_test.go @@ -32,6 +32,7 @@ import ( "google.golang.org/protobuf/testing/protocmp" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) @@ -183,7 +184,7 @@ func TestWatchWorkers_OutOfOrderCommitNotSkipped(t *testing.T) { defer watch.Close() mkPayload := func(pod string) []byte { - payload, err := marshalWorkerEvent(store.WorkerEventCreated, + payload, err := storesql.MarshalWorkerEvent(store.WorkerEventCreated, &ateapipb.Worker{ Metadata: &ateapipb.ResourceMetadata{Name: pod}, WorkerNamespace: "ns", WorkerPool: "pool", WorkerPod: pod, @@ -646,7 +647,7 @@ func TestWatchWorkers_ClosesWhenTrimmedPastCursor(t *testing.T) { // Atomically append three events and trim them away unconsumed — // the watcher never gets a chance to see them, exactly as if // retention took rows a lagging watcher had not reached. - payload, err := marshalWorkerEvent(store.WorkerEventUpdated, worker) + payload, err := storesql.MarshalWorkerEvent(store.WorkerEventUpdated, worker) if err != nil { t.Fatalf("marshaling event: %v", err) } @@ -975,35 +976,6 @@ func TestWatchWorkers_BaselineDoesNotMaskOwedTrims(t *testing.T) { } } -// TestUnmarshalWorkerEvent_BoundaryAssertions pins the write-side invariants -// asserted at the read boundary: known event-type byte and a keyable worker. -func TestUnmarshalWorkerEvent_BoundaryAssertions(t *testing.T) { - valid, err := marshalWorkerEvent(store.WorkerEventUpdated, &ateapipb.Worker{ - Metadata: &ateapipb.ResourceMetadata{Name: "w1"}, - }) - if err != nil { - t.Fatalf("marshal: %v", err) - } - for name, tc := range map[string]struct { - payload []byte - wantErr bool - }{ - "valid": {valid, false}, - "empty": {nil, true}, - "unknown type byte": {[]byte{0xff, 0x00}, true}, - "type byte only": {[]byte{byte(store.WorkerEventCreated)}, true}, // empty proto = nameless - "garbage proto": {append([]byte{byte(store.WorkerEventCreated)}, 0xde, 0xad, 0xbe), true}, - "nameless worker": {func() []byte { b, _ := marshalWorkerEvent(store.WorkerEventDeleted, &ateapipb.Worker{}); return b }(), true}, - } { - t.Run(name, func(t *testing.T) { - _, err := unmarshalWorkerEvent(tc.payload) - if (err != nil) != tc.wantErr { - t.Fatalf("unmarshalWorkerEvent() err = %v, wantErr = %v", err, tc.wantErr) - } - }) - } -} - // TestWatchWorkers_ClosesOnCorruptPayload pins close-over-skip: a payload // that fails the boundary checks must close the watch (loss surfaced, // relist repairs) rather than advance the cursor past it silently. diff --git a/cmd/ateapi/internal/store/atepg/schema.go b/cmd/ateapi/internal/store/atepg/schema.go index 0ca7ac80a9..da615d1b60 100644 --- a/cmd/ateapi/internal/store/atepg/schema.go +++ b/cmd/ateapi/internal/store/atepg/schema.go @@ -20,9 +20,8 @@ import ( "errors" "fmt" "io/fs" - "log/slog" - "time" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/stdlib" "github.com/pressly/goose/v3" @@ -130,49 +129,6 @@ func rejectUnversionedSubstrateSchema(ctx context.Context, pool *pgxpool.Pool) e return nil } -func migrateToLatest(ctx context.Context, provider *goose.Provider) (migrationErr error) { - started := time.Now() - current, latest, err := provider.GetVersions(ctx) - if err != nil { - return fmt.Errorf("get PostgreSQL migration versions: %w", err) - } - starting := current - - applied := 0 - defer func() { - attributes := []any{ - slog.Int64("starting_version", starting), - slog.Int64("current_version", current), - slog.Int64("latest_version", latest), - slog.Int("applied_migrations", applied), - slog.Duration("duration", time.Since(started)), - } - if migrationErr != nil { - attributes = append(attributes, slog.Any("err", migrationErr)) - slog.ErrorContext(ctx, "PostgreSQL migrations failed", attributes...) - return - } - slog.InfoContext(ctx, "PostgreSQL migrations ready", attributes...) - }() - - results, err := provider.Up(ctx) - if err != nil { - var partial *goose.PartialError - if errors.As(err, &partial) { - applied = len(partial.Applied) - } - applyErr := fmt.Errorf("apply PostgreSQL migrations: %w", err) - failedCurrent, _, versionErr := provider.GetVersions(ctx) - if versionErr != nil { - return errors.Join(applyErr, fmt.Errorf("get PostgreSQL migration versions after a failure: %w", versionErr)) - } - current = failedCurrent - return applyErr - } - applied = len(results) - current, _, err = provider.GetVersions(ctx) - if err != nil { - return fmt.Errorf("get PostgreSQL migration versions after migration: %w", err) - } - return nil +func migrateToLatest(ctx context.Context, provider *goose.Provider) error { + return storesql.MigrateToLatest(ctx, provider, "PostgreSQL") } diff --git a/cmd/ateapi/internal/store/atepg/tag.go b/cmd/ateapi/internal/store/atepg/tag.go index c902a2bdd5..801ccca4a8 100644 --- a/cmd/ateapi/internal/store/atepg/tag.go +++ b/cmd/ateapi/internal/store/atepg/tag.go @@ -20,6 +20,7 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" @@ -38,7 +39,7 @@ func (p *Persistence) GetTag(ctx context.Context, tagRef resources.TagRef) (*ate return nil, fmt.Errorf("getting tag %s/%s: %w", atespace, name, err) } tag := &ateapipb.Tag{} - if err := unmarshalStored(protoBytes, tag); err != nil { + if err := storesql.UnmarshalStored(protoBytes, tag); err != nil { return nil, fmt.Errorf("unmarshaling tag: %w", err) } return tag, nil @@ -63,7 +64,7 @@ func (p *Persistence) ListTags(ctx context.Context, atespace string, opts store. } func (p *Persistence) listTagsScoped(ctx context.Context, atespace string, pageSize int32, pageTokenStr string) ([]*ateapipb.Tag, string, error) { - token, err := decodePageToken(pageTokenStr, kindTag, atespace, 1) + token, err := storesql.DecodePageToken(pageTokenStr, storesql.KindTag, atespace, 1) if err != nil { return nil, "", err } @@ -90,7 +91,7 @@ func (p *Persistence) listTagsScoped(ctx context.Context, atespace string, pageS return nil, "", fmt.Errorf("scanning tag row: %w", err) } tag := &ateapipb.Tag{} - if err := unmarshalRow(protoBytes, tag, "tag", atespace, name); err != nil { + if err := storesql.UnmarshalRow(protoBytes, tag, "tag", atespace, name); err != nil { return nil, "", err } result = append(result, tag) @@ -102,13 +103,13 @@ func (p *Persistence) listTagsScoped(ctx context.Context, atespace string, pageS var nextToken string if len(result) > int(pageSize) { result = result[:pageSize] - nextToken = encodePageToken(kindTag, atespace, []string{names[pageSize-1]}) + nextToken = storesql.EncodePageToken(storesql.KindTag, atespace, []string{names[pageSize-1]}) } return result, nextToken, nil } func (p *Persistence) listTagsGlobal(ctx context.Context, pageSize int32, pageTokenStr string) ([]*ateapipb.Tag, string, error) { - token, err := decodePageToken(pageTokenStr, kindTag, "", 2) + token, err := storesql.DecodePageToken(pageTokenStr, storesql.KindTag, "", 2) if err != nil { return nil, "", err } @@ -136,7 +137,7 @@ func (p *Persistence) listTagsGlobal(ctx context.Context, pageSize int32, pageTo return nil, "", fmt.Errorf("scanning tag row: %w", err) } tag := &ateapipb.Tag{} - if err := unmarshalRow(protoBytes, tag, "tag", k.atespace, k.name); err != nil { + if err := storesql.UnmarshalRow(protoBytes, tag, "tag", k.atespace, k.name); err != nil { return nil, "", err } result = append(result, tag) @@ -149,7 +150,7 @@ func (p *Persistence) listTagsGlobal(ctx context.Context, pageSize int32, pageTo if len(result) > int(pageSize) { result = result[:pageSize] last := keys[pageSize-1] - nextToken = encodePageToken(kindTag, "", []string{last.atespace, last.name}) + nextToken = storesql.EncodePageToken(storesql.KindTag, "", []string{last.atespace, last.name}) } return result, nextToken, nil } @@ -158,7 +159,7 @@ func (p *Persistence) CreateTag(ctx context.Context, tag *ateapipb.Tag) (*ateapi atespace := tag.GetMetadata().GetAtespace() name := tag.GetMetadata().GetName() dbTag := proto.CloneOf(tag) - setCreateMetadata(dbTag.Metadata) + storesql.SetCreateMetadata(dbTag.Metadata) protoBytes, err := proto.Marshal(dbTag) if err != nil { return nil, fmt.Errorf("marshaling tag: %w", err) @@ -185,25 +186,6 @@ func (p *Persistence) CreateTag(ctx context.Context, tag *ateapipb.Tag) (*ateapi return dbTag, nil } -func validateUpdateTagMutation(storedTag, mutatedTag *ateapipb.Tag) error { - if stored, mutated := storedTag.GetMetadata().GetAtespace(), mutatedTag.GetMetadata().GetAtespace(); stored != mutated { - return fmt.Errorf("metadata.atespace is immutable: mutation changed it from %q to %q", stored, mutated) - } - if stored, mutated := storedTag.GetMetadata().GetName(), mutatedTag.GetMetadata().GetName(); stored != mutated { - return fmt.Errorf("metadata.name is immutable: mutation changed it from %q to %q", stored, mutated) - } - if stored, mutated := storedTag.GetStatus().GetSnapshot(), mutatedTag.GetStatus().GetSnapshot(); stored != nil && !proto.Equal(stored, mutated) { - return fmt.Errorf("status.snapshot is immutable once set: mutation changed it from %s to %s", stored, mutated) - } - if stored, mutated := storedTag.GetStatus().GetStorageLocation(), mutatedTag.GetStatus().GetStorageLocation(); stored != mutated { - return fmt.Errorf("status.storage_location is immutable: mutation changed it from %q to %q", stored, mutated) - } - if stored, mutated := storedTag.GetStatus().GetActorTemplateUid(), mutatedTag.GetStatus().GetActorTemplateUid(); stored != mutated { - return fmt.Errorf("status.actor_template_uid is immutable: mutation changed it from %q to %q", stored, mutated) - } - return nil -} - func (p *Persistence) UpdateTag(ctx context.Context, tagRef resources.TagRef, precondition store.Precondition, mutate func(*ateapipb.Tag) error) (*ateapipb.Tag, error) { if err := precondition.Validate(); err != nil { return nil, err @@ -222,10 +204,10 @@ func (p *Persistence) UpdateTag(ctx context.Context, tagRef resources.TagRef, pr } dbTag := &ateapipb.Tag{} - if err := unmarshalStored(currentBytes, dbTag); err != nil { + if err := storesql.UnmarshalStored(currentBytes, dbTag); err != nil { return nil, fmt.Errorf("unmarshaling tag: %w", err) } - if err := validateProtoMetadataMatchesColumns(fmt.Sprintf("tag %s/%s", atespace, name), dbTag.GetMetadata(), currentUID, currentVersion); err != nil { + if err := storesql.ValidateProtoMetadataMatchesColumns(fmt.Sprintf("tag %s/%s", atespace, name), dbTag.GetMetadata(), currentUID, currentVersion); err != nil { return nil, err } if err := precondition.Check(dbTag.GetMetadata()); err != nil { @@ -237,12 +219,12 @@ func (p *Persistence) UpdateTag(ctx context.Context, tagRef resources.TagRef, pr return nil, err } // TODO: this should be done through DV and removed from here - if err := validateUpdateTagMutation(tagBeforeMutation, dbTag); err != nil { + if err := storesql.ValidateUpdateTagMutation(tagBeforeMutation, dbTag); err != nil { return nil, fmt.Errorf("%w: %w", store.ErrImmutableField, err) } // Stored metadata is authoritative; discard any metadata edits made by the // closure and derive the next revision from the state this attempt read. - setUpdateMetadata(dbTag.Metadata, oldMeta) + storesql.SetUpdateMetadata(dbTag.Metadata, oldMeta) updatedBytes, err := proto.Marshal(dbTag) if err != nil { @@ -284,7 +266,7 @@ func (p *Persistence) DeleteTag(ctx context.Context, tagRef resources.TagRef, pr return nil, fmt.Errorf("deleting tag %s/%s: %w", atespace, name, err) } tag := &ateapipb.Tag{} - if err := unmarshalStored(protoBytes, tag); err != nil { + if err := storesql.UnmarshalStored(protoBytes, tag); err != nil { return nil, fmt.Errorf("unmarshaling deleted tag: %w", err) } return tag, nil diff --git a/cmd/ateapi/internal/store/atepg/tag_test.go b/cmd/ateapi/internal/store/atepg/tag_test.go index 50ae7b4826..71976bb6e2 100644 --- a/cmd/ateapi/internal/store/atepg/tag_test.go +++ b/cmd/ateapi/internal/store/atepg/tag_test.go @@ -16,19 +16,13 @@ package atepg import ( "context" - "errors" - "fmt" "testing" - "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" - "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" - "github.com/google/go-cmp/cmp" - "google.golang.org/protobuf/testing/protocmp" ) -// createTestTag creates tagName over its own copy of an actor's -// external snapshot, already finalized. +// createTestTag creates tagName over its own copy of an actor's external +// snapshot, already finalized. func createTestTag(t *testing.T, s *Persistence, tagAtespace, tagName string) *ateapipb.Tag { t.Helper() tag, err := s.CreateTag(context.Background(), &ateapipb.Tag{ @@ -43,52 +37,3 @@ func createTestTag(t *testing.T, s *Persistence, tagAtespace, tagName string) *a } return tag } - -func TestUpdateTag_CASPreventsDeleteRecreateABA(t *testing.T) { - s := setupPostgresPersistence(t) - ctx := context.Background() - createTestAtespace(t, s, "team-a") - original := createTestTag(t, s, "team-a", "tag-a") - - mutations := 0 - var recreated *ateapipb.Tag - _, err := s.UpdateTag(ctx, resources.TagRef{Atespace: "team-a", Name: "tag-a"}, store.PreconditionFrom(original), func(toUpdate *ateapipb.Tag) error { - mutations++ - if _, err := s.DeleteTag(ctx, resources.TagRef{Atespace: "team-a", Name: "tag-a"}, store.DeletePreconditions{}); err != nil { - return fmt.Errorf("deleting original tag: %w", err) - } - recreated = createTestTag(t, s, "team-a", "tag-a") - toUpdate.Scope = ateapipb.TagScope_TAG_SCOPE_PUBLISHED - return nil - }) - if !errors.Is(err, store.ErrVersionConflict) { - t.Fatalf("UpdateTag error = %v, want ErrVersionConflict", err) - } - if mutations != 1 { - t.Errorf("guarded mutation ran %d times, want 1", mutations) - } - stored, err := s.GetTag(ctx, resources.TagRef{Atespace: "team-a", Name: "tag-a"}) - if err != nil { - t.Fatalf("GetTag failed: %v", err) - } - if diff := cmp.Diff(recreated, stored, protocmp.Transform()); diff != "" { - t.Errorf("recreated tag was overwritten (-want +got):\n%s", diff) - } -} - -func TestCreateTag_TagForeignKeyErrors(t *testing.T) { - s := setupPostgresPersistence(t) - ctx := context.Background() - createTestAtespace(t, s, "team-a") - - // A tag in an atespace that does not exist trips the tag's atespace FK. - _, err := s.CreateTag(ctx, &ateapipb.Tag{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "gone", Name: "latest"}, - Status: &ateapipb.TagStatus{ - StorageLocation: "gs://bucket", - }, - }) - if !errors.Is(err, store.ErrFailedPrecondition) { - t.Errorf("missing tag atespace error = %v, want ErrFailedPrecondition", err) - } -} diff --git a/cmd/ateapi/internal/store/atepg/worker.go b/cmd/ateapi/internal/store/atepg/worker.go index a7428e1100..4497c13856 100644 --- a/cmd/ateapi/internal/store/atepg/worker.go +++ b/cmd/ateapi/internal/store/atepg/worker.go @@ -20,6 +20,7 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" "google.golang.org/protobuf/proto" @@ -30,7 +31,7 @@ func (p *Persistence) CreateWorker(ctx context.Context, worker *ateapipb.Worker) if dbWorker.Metadata == nil { dbWorker.Metadata = &ateapipb.ResourceMetadata{} } - setCreateMetadata(dbWorker.Metadata) + storesql.SetCreateMetadata(dbWorker.Metadata) protoBytes, err := proto.Marshal(dbWorker) if err != nil { @@ -66,7 +67,7 @@ func getWorkerRow(ctx context.Context, q querier, name string) (*ateapipb.Worker return nil, fmt.Errorf("getting worker %s: %w", name, err) } out := &ateapipb.Worker{} - if err := unmarshalStored(protoBytes, out); err != nil { + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { return nil, fmt.Errorf("unmarshaling worker: %w", err) } return out, nil @@ -88,7 +89,7 @@ func getWorkerRowForUpdate(ctx context.Context, tx pgx.Tx, name string) (*ateapi return nil, fmt.Errorf("locking worker %s for update: %w", name, err) } out := &ateapipb.Worker{} - if err := unmarshalStored(protoBytes, out); err != nil { + if err := storesql.UnmarshalStored(protoBytes, out); err != nil { return nil, fmt.Errorf("unmarshaling worker: %w", err) } return out, nil @@ -121,7 +122,7 @@ func (p *Persistence) UpdateWorker(ctx context.Context, name string, preconditio } // Stored metadata is authoritative; discard any metadata edits made by // the closure and derive the next revision from the row we locked. - setUpdateMetadata(dbWorker.Metadata, oldMeta) + storesql.SetUpdateMetadata(dbWorker.Metadata, oldMeta) protoBytes, err := proto.Marshal(dbWorker) if err != nil { @@ -171,7 +172,7 @@ func (p *Persistence) ListWorkers(ctx context.Context, opts store.ListOptions) ( return store.ListResponse[*ateapipb.Worker]{}, err } pageSize, pageTokenStr := opts.PageSize, opts.PageToken - token, err := decodePageToken(pageTokenStr, kindWorker, "", 1) + token, err := storesql.DecodePageToken(pageTokenStr, storesql.KindWorker, "", 1) if err != nil { return store.ListResponse[*ateapipb.Worker]{}, err } @@ -199,7 +200,7 @@ func (p *Persistence) ListWorkers(ctx context.Context, opts store.ListOptions) ( return store.ListResponse[*ateapipb.Worker]{}, fmt.Errorf("scanning worker row: %w", err) } w := &ateapipb.Worker{} - if err := unmarshalRow(protoBytes, w, "worker", name); err != nil { + if err := storesql.UnmarshalRow(protoBytes, w, "worker", name); err != nil { return store.ListResponse[*ateapipb.Worker]{}, err } result = append(result, w) @@ -212,7 +213,7 @@ func (p *Persistence) ListWorkers(ctx context.Context, opts store.ListOptions) ( var nextToken string if len(result) > int(pageSize) { result = result[:pageSize] - nextToken = encodePageToken(kindWorker, "", []string{names[pageSize-1]}) + nextToken = storesql.EncodePageToken(storesql.KindWorker, "", []string{names[pageSize-1]}) } return store.ListResponse[*ateapipb.Worker]{Items: result, NextPageToken: nextToken}, nil } diff --git a/cmd/ateapi/internal/store/atepg/worker_assignment.go b/cmd/ateapi/internal/store/atepg/worker_assignment.go index 4e85ae595a..4359207ef8 100644 --- a/cmd/ateapi/internal/store/atepg/worker_assignment.go +++ b/cmd/ateapi/internal/store/atepg/worker_assignment.go @@ -20,6 +20,7 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storesql" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/jackc/pgx/v5" @@ -49,7 +50,7 @@ func getWorkerForUpdate(ctx context.Context, tx pgx.Tx, name string) (*ateapipb. // version. This assumes the caller holds the row lock getWorkerForUpdate took. func saveWorker(ctx context.Context, tx pgx.Tx, worker *ateapipb.Worker) error { read := proto.CloneOf(worker.GetMetadata()) - setUpdateMetadata(worker.GetMetadata(), worker.GetMetadata()) + storesql.SetUpdateMetadata(worker.GetMetadata(), worker.GetMetadata()) protoBytes, err := proto.Marshal(worker) if err != nil { return fmt.Errorf("marshaling worker: %w", err) @@ -78,7 +79,7 @@ func (p *Persistence) BindActorToWorker(ctx context.Context, workerName string, // global-scoped; the name is the Actor's UID, which is also the row key. // This is the identity a first bind gets; a rebind keeps the recorded one. assignment.Metadata = &ateapipb.ResourceMetadata{Name: actorUID} - setCreateMetadata(assignment.Metadata) + storesql.SetCreateMetadata(assignment.Metadata) _, err := p.writeAndAppendEvent(ctx, store.WorkerEventUpdated, func(ctx context.Context, tx pgx.Tx) (*ateapipb.Worker, error) { worker, err := getWorkerForUpdate(ctx, tx, workerName) @@ -160,7 +161,7 @@ func (p *Persistence) BindActorToWorker(ctx context.Context, workerName string, // A rebind updates the assignment already recorded, so re-stamping it // as a create would make a retried claim look like a new subresource. - setUpdateMetadata(assignment.Metadata, previous.GetMetadata()) + storesql.SetUpdateMetadata(assignment.Metadata, previous.GetMetadata()) rebindBytes, err := proto.Marshal(assignment) if err != nil { return nil, fmt.Errorf("marshaling rebound assignment: %w", err) @@ -272,7 +273,7 @@ func (p *Persistence) ListWorkerAssignments(ctx context.Context, workerName stri pageSize := opts.PageSize // The token is scoped to the Worker, so one cannot be replayed against // another Worker's assignments. - token, err := decodePageToken(opts.PageToken, kindWorkerAssign, workerName, 1) + token, err := storesql.DecodePageToken(opts.PageToken, storesql.KindWorkerAssign, workerName, 1) if err != nil { return store.ListResponse[*ateapipb.ActorAssignment]{}, err } @@ -313,7 +314,7 @@ func (p *Persistence) ListWorkerAssignments(ctx context.Context, workerName stri var nextToken string if len(result) > int(pageSize) { result = result[:pageSize] - nextToken = encodePageToken(kindWorkerAssign, workerName, []string{uids[pageSize-1]}) + nextToken = storesql.EncodePageToken(storesql.KindWorkerAssign, workerName, []string{uids[pageSize-1]}) } return store.ListResponse[*ateapipb.ActorAssignment]{Items: result, NextPageToken: nextToken}, nil } diff --git a/cmd/ateapi/internal/store/storecontract/contract.go b/cmd/ateapi/internal/store/storecontract/contract.go index df6b5d502f..cd59a2322c 100644 --- a/cmd/ateapi/internal/store/storecontract/contract.go +++ b/cmd/ateapi/internal/store/storecontract/contract.go @@ -35,6 +35,7 @@ import ( "google.golang.org/protobuf/testing/protocmp" "google.golang.org/protobuf/types/known/timestamppb" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -199,8 +200,8 @@ func assertPruned(t *testing.T, op string, want, got proto.Message) { // against a fresh store.Interface built by setup for each subtest. setup is // responsible for its own cleanup (e.g. via t.Cleanup). // -// PostgreSQL-specific behavior such as foreign-key races and transactional -// notifications is not covered here; see atepg's own test file for that. +// Backend-specific behavior such as worker outbox notifications is covered by +// each backend's own tests. func RunContractTests(t *testing.T, setup func(t *testing.T) store.Interface) { runActorContractTests(t, setup) runEgressPolicyContractTests(t, setup) @@ -212,6 +213,7 @@ func RunContractTests(t *testing.T, setup func(t *testing.T) store.Interface) { runLeaseContractTests(t, setup) runListOptionsContractTests(t, setup) runUnknownFieldContractTests(t, setup) + runAccessPolicyContractTests(t, setup) } func runEgressPolicyContractTests(t *testing.T, setup func(t *testing.T) store.Interface) { @@ -508,6 +510,16 @@ func runActorContractTests(t *testing.T, setup func(t *testing.T) store.Interfac } }) + t.Run("CreateActor_UnknownAtespace", func(t *testing.T) { + s := setup(t) + ctx := context.Background() + + actor := newTestSuspendedActor("team-missing", "session-1") + if _, err := s.CreateActor(ctx, actor); !errors.Is(err, store.ErrFailedPrecondition) { + t.Errorf("CreateActor in an unknown atespace = %v, want ErrFailedPrecondition", err) + } + }) + t.Run("UpdateActor_Success", func(t *testing.T) { s := setup(t) ctx := context.Background() @@ -741,6 +753,51 @@ func runActorContractTests(t *testing.T, setup func(t *testing.T) store.Interfac } }) + // A write that lands while mutate runs fails the outer update without + // retrying it, and the concurrent write survives. + t.Run("UpdateActor_ConcurrentWrite", func(t *testing.T) { + s := setup(t) + ctx := context.Background() + mustCreateAtespace(t, s, testAtespace) + created, err := s.CreateActor(ctx, newTestSuspendedActor(testAtespace, "session-1")) + if err != nil { + t.Fatalf("CreateActor failed: %v", err) + } + actorRef := resources.ActorRefFromActor(created) + + mutations := 0 + _, err = s.UpdateActor(ctx, actorRef, store.PreconditionFrom(created), func(toUpdate *ateapipb.Actor) error { + mutations++ + if _, err := s.UpdateActor(ctx, actorRef, store.PreconditionFrom(created), func(concurrent *ateapipb.Actor) error { + concurrent.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid"}} + return nil + }); err != nil { + return fmt.Errorf("concurrent actor update: %w", err) + } + toUpdate.Status.State = ateapipb.ActorState_ACTOR_STATE_RUNNING + return nil + }) + if !errors.Is(err, store.ErrVersionConflict) { + t.Fatalf("UpdateActor error = %v, want ErrVersionConflict", err) + } + if mutations != 1 { + t.Errorf("mutation ran %d times, want 1", mutations) + } + stored, err := s.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor failed: %v", err) + } + if stored.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Errorf("state = %v, want SUSPENDED: losing update was persisted", stored.GetStatus().GetState()) + } + if got := stored.GetWorkerSelector().GetMatchLabels()["tier"]; got != "paid" { + t.Errorf("worker selector tier = %q, want paid", got) + } + if got, want := stored.GetMetadata().GetVersion(), created.GetMetadata().GetVersion()+1; got != want { + t.Errorf("version = %d, want %d", got, want) + } + }) + // The store does not gate the delete on state; the workflow does, and pins // the row delete to the uid and version it last saw. t.Run("DeleteActor", func(t *testing.T) { @@ -982,6 +1039,19 @@ func runActorContractTests(t *testing.T, setup func(t *testing.T) store.Interfac t.Errorf("ListActors(all) = %v, want exactly {a1, a2, b1}", got) } + teamAPage, err := s.ListActors(ctx, "team-a", store.ListOptions{PageSize: 1}) + if err != nil || teamAPage.NextPageToken == "" { + t.Fatalf("ListActors(team-a, PageSize 1) = %v, %v; want a next page token", teamAPage, err) + } + for _, scope := range []string{"team-b", ""} { + if _, err := s.ListActors(ctx, scope, store.ListOptions{PageSize: 1, PageToken: teamAPage.NextPageToken}); !errors.Is(err, store.ErrInvalidPageToken) { + t.Errorf("ListActors(%q) with a team-a page token = %v, want ErrInvalidPageToken", scope, err) + } + } + if _, err := s.ListAtespaces(ctx, store.ListOptions{PageSize: 1, PageToken: teamAPage.NextPageToken}); !errors.Is(err, store.ErrInvalidPageToken) { + t.Errorf("ListAtespaces with an actor page token = %v, want ErrInvalidPageToken", err) + } + if _, err := s.GetActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "a1"}); err != nil { t.Errorf("GetActor(team-a, a1) failed: %v", err) } @@ -1124,6 +1194,81 @@ func runActorTemplateContractTests(t *testing.T, setup func(t *testing.T) store. } }) + t.Run("UpdateActorTemplate_ConcurrentWrite", func(t *testing.T) { + s := setup(t) + ctx := context.Background() + mustCreateAtespace(t, s, "team-a") + created, err := s.CreateActorTemplate(ctx, newTestActorTemplate("team-a", "tmpl-a")) + if err != nil { + t.Fatalf("CreateActorTemplate failed: %v", err) + } + templateRef := resources.ActorTemplateRef{Atespace: "team-a", Name: "tmpl-a"} + + mutations := 0 + _, err = s.UpdateActorTemplate(ctx, templateRef, store.PreconditionFrom(created), func(toUpdate *ateapipb.ActorTemplate) error { + mutations++ + if _, err := s.UpdateActorTemplate(ctx, templateRef, store.PreconditionFrom(created), func(concurrent *ateapipb.ActorTemplate) error { + concurrent.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"tier": "paid"}} + return nil + }); err != nil { + return fmt.Errorf("concurrent actor template update: %w", err) + } + toUpdate.Status = &ateapipb.ActorTemplateStatus{GoldenSnapshotStatus: &ateapipb.GoldenSnapshotStatus{ + ErrorMessage: "LosingUpdate", + }} + return nil + }) + if !errors.Is(err, store.ErrVersionConflict) { + t.Fatalf("UpdateActorTemplate error = %v, want ErrVersionConflict", err) + } + if mutations != 1 { + t.Errorf("mutation ran %d times, want 1", mutations) + } + stored, err := s.GetActorTemplate(ctx, templateRef) + if err != nil { + t.Fatalf("GetActorTemplate failed: %v", err) + } + if got := stored.GetWorkerSelector().GetMatchLabels()["tier"]; got != "paid" { + t.Errorf("worker selector tier = %q, want paid", got) + } + if got := stored.GetStatus().GetGoldenSnapshotStatus().GetErrorMessage(); got != "" { + t.Errorf("status error message = %q, want empty: losing update was persisted", got) + } + }) + + t.Run("ActorTemplate_BackfillsSnapshotConfigDefaults", func(t *testing.T) { + s := setup(t) + ctx := context.Background() + mustCreateAtespace(t, s, "team-a") + input := newTestActorTemplate("team-a", "tmpl-a") + input.SnapshotConfig = &ateapipb.SnapshotConfig{} + if _, err := s.CreateActorTemplate(ctx, input); err != nil { + t.Fatalf("CreateActorTemplate failed: %v", err) + } + + want := &ateapipb.SnapshotConfig{ + OnPause: ateapipb.SnapshotContentScope_SNAPSHOT_CONTENT_SCOPE_FULL, + OnCommit: ateapipb.SnapshotContentScope_SNAPSHOT_CONTENT_SCOPE_FULL, + } + got, err := s.GetActorTemplate(ctx, resources.ActorTemplateRef{Atespace: "team-a", Name: "tmpl-a"}) + if err != nil { + t.Fatalf("GetActorTemplate failed: %v", err) + } + if diff := cmp.Diff(want, got.GetSnapshotConfig(), protocmp.Transform()); diff != "" { + t.Errorf("GetActorTemplate snapshot_config (-want +got):\n%s", diff) + } + page, err := s.ListActorTemplates(ctx, "team-a", store.ListOptions{PageSize: 10}) + if err != nil { + t.Fatalf("ListActorTemplates failed: %v", err) + } + if len(page.Items) != 1 { + t.Fatalf("ListActorTemplates returned %d items, want 1", len(page.Items)) + } + if diff := cmp.Diff(want, page.Items[0].GetSnapshotConfig(), protocmp.Transform()); diff != "" { + t.Errorf("ListActorTemplates snapshot_config (-want +got):\n%s", diff) + } + }) + t.Run("ActorTemplateResources_BlockAtespaceDeletion", func(t *testing.T) { s := setup(t) ctx := context.Background() @@ -1449,6 +1594,41 @@ func runTagContractTests(t *testing.T, setup func(t *testing.T) store.Interface) } }) + // Deleting and recreating a tag under a running update must fail the + // update on the new incarnation rather than overwrite it. + t.Run("UpdateTag_DeleteRecreateConflict", func(t *testing.T) { + s := setup(t) + ctx := context.Background() + actor := seedSuspendedActor(t, s, "team-a", "actor-1") + original := storeTag(t, s, newTestInProgressTag("production", actor)) + tagRef := resources.TagRefFromTag(original) + + mutations := 0 + var recreated *ateapipb.Tag + _, err := s.UpdateTag(ctx, tagRef, store.PreconditionFrom(original), func(toUpdate *ateapipb.Tag) error { + mutations++ + if _, err := s.DeleteTag(ctx, tagRef, store.DeletePreconditions{}); err != nil { + return fmt.Errorf("deleting original tag: %w", err) + } + recreated = storeTag(t, s, newTestInProgressTag("production", actor)) + toUpdate.Scope = ateapipb.TagScope_TAG_SCOPE_PUBLISHED + return nil + }) + if !errors.Is(err, store.ErrVersionConflict) { + t.Fatalf("UpdateTag error = %v, want ErrVersionConflict", err) + } + if mutations != 1 { + t.Errorf("guarded mutation ran %d times, want 1", mutations) + } + stored, err := s.GetTag(ctx, tagRef) + if err != nil { + t.Fatalf("GetTag failed: %v", err) + } + if diff := cmp.Diff(recreated, stored, protocmp.Transform()); diff != "" { + t.Errorf("recreated tag was overwritten (-want +got):\n%s", diff) + } + }) + t.Run("ListTags_PaginationAndScope", func(t *testing.T) { s := setup(t) ctx := context.Background() @@ -3365,3 +3545,125 @@ func runUnknownFieldContractTests(t *testing.T, setup func(t *testing.T) store.I assertPruned(t, "DeleteWorker", created, deleted) }) } + +func runAccessPolicyContractTests(t *testing.T, setup func(t *testing.T) store.Interface) { + t.Helper() + + t.Run("GlobalAccessPolicy_Lifecycle", func(t *testing.T) { + s := setup(t) + ctx := context.Background() + + if _, err := s.GetGlobalAccessPolicy(ctx); !errors.Is(err, store.ErrNotFound) { + t.Fatalf("GetGlobalAccessPolicy before create = %v, want ErrNotFound", err) + } + + policy := &ateapipb.AccessPolicy{ + Bindings: []*ateapipb.Binding{ + {Role: authz.RoleViewer, Members: []string{"user:bob"}}, + {Role: authz.RoleOwner, Members: []string{"user:alice"}}, + }, + } + created, err := s.CreateGlobalAccessPolicy(ctx, policy) + if err != nil { + t.Fatalf("CreateGlobalAccessPolicy failed: %v", err) + } + if created.GetMetadata().GetName() != "default" || created.GetMetadata().GetVersion() != 1 || created.GetMetadata().GetUid() == "" { + t.Fatalf("unexpected created metadata: %+v", created.GetMetadata()) + } + if diff := cmp.Diff(policy.GetBindings(), created.GetBindings(), protocmp.Transform()); diff != "" { + t.Errorf("created bindings (-want +got):\n%s", diff) + } + if _, err := s.CreateGlobalAccessPolicy(ctx, policy); !errors.Is(err, store.ErrAlreadyExists) { + t.Fatalf("second CreateGlobalAccessPolicy = %v, want ErrAlreadyExists", err) + } + got, err := s.GetGlobalAccessPolicy(ctx) + if err != nil || !cmp.Equal(got, created, protocmp.Transform()) { + t.Fatalf("GetGlobalAccessPolicy = %v, %v; want %v", got, err, created) + } + + noop := func(*ateapipb.AccessPolicy) error { return nil } + if _, err := s.UpdateGlobalAccessPolicy(ctx, store.Precondition{}, noop); !errors.Is(err, store.ErrPreconditionRequired) { + t.Fatalf("UpdateGlobalAccessPolicy without precondition = %v, want ErrPreconditionRequired", err) + } + if _, err := s.UpdateGlobalAccessPolicy(ctx, store.Precondition{UID: created.GetMetadata().GetUid(), Version: 99}, noop); !errors.Is(err, store.ErrVersionConflict) { + t.Fatalf("UpdateGlobalAccessPolicy wrong version = %v, want ErrVersionConflict", err) + } + if _, err := s.UpdateGlobalAccessPolicy(ctx, store.Precondition{UID: foreignUID, Version: 1}, noop); !errors.Is(err, store.ErrUIDConflict) { + t.Fatalf("UpdateGlobalAccessPolicy wrong uid = %v, want ErrUIDConflict", err) + } + + updated, err := s.UpdateGlobalAccessPolicy(ctx, store.PreconditionFrom(created), func(toUpdate *ateapipb.AccessPolicy) error { + toUpdate.Bindings = []*ateapipb.Binding{ + {Role: authz.RoleOwner, Members: []string{"user:alice", "user:carol"}}, + } + return nil + }) + if err != nil || updated.GetMetadata().GetVersion() != 2 { + t.Fatalf("UpdateGlobalAccessPolicy = %v, %v; want version 2", updated, err) + } + }) + + t.Run("AtespaceAccessPolicy_LifecycleAndCascade", func(t *testing.T) { + s := setup(t) + ctx := context.Background() + + policy := &ateapipb.AccessPolicy{ + Bindings: []*ateapipb.Binding{ + {Role: authz.RoleEditor, Members: []string{"user:bob"}}, + }, + } + if _, err := s.CreateAtespaceAccessPolicy(ctx, "team-missing", policy); !errors.Is(err, store.ErrFailedPrecondition) { + t.Fatalf("CreateAtespaceAccessPolicy on missing atespace = %v, want ErrFailedPrecondition", err) + } + + mustCreateAtespace(t, s, "team-a") + if _, err := s.GetAtespaceAccessPolicy(ctx, "team-a"); !errors.Is(err, store.ErrNotFound) { + t.Fatalf("GetAtespaceAccessPolicy before create = %v, want ErrNotFound", err) + } + created, err := s.CreateAtespaceAccessPolicy(ctx, "team-a", policy) + if err != nil { + t.Fatalf("CreateAtespaceAccessPolicy failed: %v", err) + } + if _, err := s.CreateAtespaceAccessPolicy(ctx, "team-a", policy); !errors.Is(err, store.ErrAlreadyExists) { + t.Fatalf("second CreateAtespaceAccessPolicy = %v, want ErrAlreadyExists", err) + } + + updated, err := s.UpdateAtespaceAccessPolicy(ctx, "team-a", store.PreconditionFrom(created), func(toUpdate *ateapipb.AccessPolicy) error { + toUpdate.Bindings = []*ateapipb.Binding{ + {Role: authz.RoleViewer, Members: []string{"user:dave"}}, + } + return nil + }) + if err != nil || updated.GetMetadata().GetVersion() != 2 { + t.Fatalf("UpdateAtespaceAccessPolicy = %v, %v; want version 2", updated, err) + } + + if _, err := s.DeleteAtespaceAccessPolicy(ctx, "team-a", store.DeletePreconditions{Version: 99}); !errors.Is(err, store.ErrVersionConflict) { + t.Fatalf("DeleteAtespaceAccessPolicy wrong version = %v, want ErrVersionConflict", err) + } + if _, err := s.DeleteAtespaceAccessPolicy(ctx, "team-a", store.DeletePreconditions{UID: foreignUID}); !errors.Is(err, store.ErrUIDConflict) { + t.Fatalf("DeleteAtespaceAccessPolicy wrong uid = %v, want ErrUIDConflict", err) + } + deleted, err := s.DeleteAtespaceAccessPolicy(ctx, "team-a", store.DeletePreconditions{ + UID: updated.GetMetadata().GetUid(), + Version: updated.GetMetadata().GetVersion(), + }) + if err != nil || !cmp.Equal(deleted, updated, protocmp.Transform()) { + t.Fatalf("DeleteAtespaceAccessPolicy = %v, %v; want %v", deleted, err, updated) + } + if _, err := s.GetAtespaceAccessPolicy(ctx, "team-a"); !errors.Is(err, store.ErrNotFound) { + t.Fatalf("GetAtespaceAccessPolicy after delete = %v, want ErrNotFound", err) + } + + // An access policy does not block its atespace's deletion; it goes with it. + if _, err := s.CreateAtespaceAccessPolicy(ctx, "team-a", policy); err != nil { + t.Fatalf("re-creating access policy failed: %v", err) + } + if _, err := s.DeleteAtespace(ctx, "team-a", store.DeletePreconditions{}); err != nil { + t.Fatalf("DeleteAtespace failed: %v", err) + } + if _, err := s.GetAtespaceAccessPolicy(ctx, "team-a"); !errors.Is(err, store.ErrNotFound) { + t.Fatalf("GetAtespaceAccessPolicy after DeleteAtespace = %v, want ErrNotFound", err) + } + }) +} diff --git a/cmd/ateapi/internal/store/storesql/lease.go b/cmd/ateapi/internal/store/storesql/lease.go new file mode 100644 index 0000000000..e028bbbf3d --- /dev/null +++ b/cmd/ateapi/internal/store/storesql/lease.go @@ -0,0 +1,164 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package storesql + +import ( + "context" + "errors" + "log/slog" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/google/uuid" +) + +// DefaultLeaseTTL is how long a lease may go unrenewed before another client +// can reclaim it. +const DefaultLeaseTTL = 30 * time.Second + +// LeaseCleanupBatch bounds one DELETE of a cleanup pass, so a backlog of +// expired rows drains over several short statements rather than one long +// one holding many row locks. +const LeaseCleanupBatch = 1000 + +// LeaseSQL is the database half of a distributed lease. Each func is atomic +// and decides expiry by the database clock; AcquireLease supplies the token +// and drives renewal and release. +type LeaseSQL struct { + // Database names the backend in log lines. + Database string + // Acquire takes key for token unless an unexpired holder has it. + Acquire func(ctx context.Context, key, token string, ttl time.Duration) (bool, error) + // Renew extends key while token still holds it unexpired. + Renew func(ctx context.Context, key, token string, ttl time.Duration) (bool, error) + // Release drops key if token still holds it. + Release func(ctx context.Context, key, token string) error +} + +// AcquireLease takes key for ttl and renews it until the returned Lease is +// closed. Returns store.ErrLeaseConflict if another client holds it. +func (l LeaseSQL) AcquireLease(ctx context.Context, key string, ttl time.Duration) (*store.Lease, error) { + token := uuid.NewString() + // Acquisition runs before any workflow step span opens, so log the + // query's duration to make this window attributable. + t := time.Now() + acquired, err := l.Acquire(ctx, key, token, ttl) + dAcquire := time.Since(t) + slog.InfoContext(ctx, "lease acquisition finished", + slog.String("database", l.Database), + slog.String("key", key), + slog.Bool("acquired", acquired && err == nil), + slog.Duration("acquire", dAcquire)) + if err != nil { + return nil, err + } + if !acquired { + return nil, store.ErrLeaseConflict + } + + leaseCtx, cancel := context.WithCancel(ctx) + renewalDone := make(chan struct{}) + go func() { + defer close(renewalDone) + defer cancel() + l.renewLoop(leaseCtx, key, token, ttl) + }() + + closeFn := func() { + // Close runs after the last workflow step span ends but inside the + // operation, so log its two waits: the renewal goroutine may be + // mid-query when cancelled, and the release DELETE may wait on a + // concurrent write to the same row. + t := time.Now() + cancel() + <-renewalDone + dRenewalStop := time.Since(t) + + releaseCtx, releaseCancel := context.WithTimeout(context.Background(), 5*time.Second) + defer releaseCancel() + t = time.Now() + if err := l.Release(releaseCtx, key, token); err != nil { + slog.WarnContext(releaseCtx, "failed to release lease, relying on TTL to reclaim it", "database", l.Database, "key", key, "error", err) + } + slog.InfoContext(releaseCtx, "lease released", + slog.String("database", l.Database), + slog.String("key", key), + slog.Duration("renewal_stop", dRenewalStop), + slog.Duration("release", time.Since(t))) + } + return store.NewLease(leaseCtx, closeFn), nil +} + +const ( + renewIntervalDivisor = 3 + renewRetryPeriodDivisor = 10 + renewDeadlineFraction = 2.0 / 3.0 +) + +func (l LeaseSQL) renewLoop(ctx context.Context, key, token string, ttl time.Duration) { + interval := ttl / renewIntervalDivisor + renewDeadline := time.Duration(float64(ttl) * renewDeadlineFraction) + + lastRenewed := time.Now() + timer := time.NewTimer(interval) + defer timer.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-timer.C: + renewCtx, cancel := context.WithDeadline(ctx, lastRenewed.Add(renewDeadline)) + renewed := l.tryRenew(renewCtx, key, token, ttl) + cancel() + if !renewed { + return + } + lastRenewed = time.Now() + timer.Reset(interval) + } + } +} + +func (l LeaseSQL) tryRenew(ctx context.Context, key, token string, ttl time.Duration) bool { + retryPeriod := ttl / renewRetryPeriodDivisor + retry := time.NewTimer(0) + defer retry.Stop() + + for { + select { + case <-ctx.Done(): + if errors.Is(ctx.Err(), context.DeadlineExceeded) { + slog.WarnContext(ctx, "failed to renew lease before its deadline", "database", l.Database, "key", key) + } + return false + case <-retry.C: + renewed, err := l.Renew(ctx, key, token, ttl) + if ctx.Err() != nil { + return false + } + switch { + case err == nil && renewed: + return true + case err == nil: + slog.WarnContext(ctx, "lease renewal found lease no longer owned", "database", l.Database, "key", key) + return false + default: + slog.WarnContext(ctx, "failed to renew lease, retrying", "database", l.Database, "key", key, "error", err) + retry.Reset(retryPeriod) + } + } + } +} diff --git a/cmd/ateapi/internal/store/storesql/migrate.go b/cmd/ateapi/internal/store/storesql/migrate.go new file mode 100644 index 0000000000..cc8cbab509 --- /dev/null +++ b/cmd/ateapi/internal/store/storesql/migrate.go @@ -0,0 +1,75 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package storesql + +import ( + "context" + "errors" + "fmt" + "log/slog" + "time" + + "github.com/pressly/goose/v3" +) + +// MigrateToLatest applies provider's pending migrations and logs the versions +// before and after. database names the backend in errors and logs. +func MigrateToLatest(ctx context.Context, provider *goose.Provider, database string) (migrationErr error) { + started := time.Now() + current, latest, err := provider.GetVersions(ctx) + if err != nil { + return fmt.Errorf("get %s migration versions: %w", database, err) + } + starting := current + + applied := 0 + defer func() { + attributes := []any{ + slog.String("database", database), + slog.Int64("starting_version", starting), + slog.Int64("current_version", current), + slog.Int64("latest_version", latest), + slog.Int("applied_migrations", applied), + slog.Duration("duration", time.Since(started)), + } + if migrationErr != nil { + attributes = append(attributes, slog.Any("err", migrationErr)) + slog.ErrorContext(ctx, "migrations failed", attributes...) + return + } + slog.InfoContext(ctx, "migrations ready", attributes...) + }() + + results, err := provider.Up(ctx) + if err != nil { + var partial *goose.PartialError + if errors.As(err, &partial) { + applied = len(partial.Applied) + } + applyErr := fmt.Errorf("apply %s migrations: %w", database, err) + failedCurrent, _, versionErr := provider.GetVersions(ctx) + if versionErr != nil { + return errors.Join(applyErr, fmt.Errorf("get %s migration versions after a failure: %w", database, versionErr)) + } + current = failedCurrent + return applyErr + } + applied = len(results) + current, _, err = provider.GetVersions(ctx) + if err != nil { + return fmt.Errorf("get %s migration versions after migration: %w", database, err) + } + return nil +} diff --git a/cmd/ateapi/internal/store/atepg/pagetoken.go b/cmd/ateapi/internal/store/storesql/pagetoken.go similarity index 51% rename from cmd/ateapi/internal/store/atepg/pagetoken.go rename to cmd/ateapi/internal/store/storesql/pagetoken.go index ebf56b5ae2..a5e27daf3d 100644 --- a/cmd/ateapi/internal/store/atepg/pagetoken.go +++ b/cmd/ateapi/internal/store/storesql/pagetoken.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package atepg +package storesql import ( "encoding/base64" @@ -26,59 +26,60 @@ import ( // future token format. const pageTokenVersion = 1 -// resourceKind identifies which List method a page token belongs to, so a +// Kind identifies which List method a page token belongs to, so a // token can't be replayed against the wrong method. -type resourceKind string +type Kind string const ( - kindAtespace resourceKind = "atespace" - kindActor resourceKind = "actor" - kindActorTemplate resourceKind = "actor-template" - kindTag resourceKind = "tag" - kindWorker resourceKind = "worker" - kindWorkerAssign resourceKind = "worker-assignment" + KindAtespace Kind = "atespace" + KindActor Kind = "actor" + KindActorTemplate Kind = "actor-template" + KindTag Kind = "tag" + KindWorker Kind = "worker" + KindWorkerAssign Kind = "worker-assignment" ) -// pageToken is PostgreSQL's opaque keyset page token. It carries no database -// topology: just enough to resume an ORDER BY ... WHERE (cols) > (last) scan. -type pageToken struct { - Version int `json:"v"` - Kind resourceKind `json:"kind"` - Scope string `json:"scope"` // atespace name for scoped actor listing; empty otherwise. - Last []string `json:"last"` // last row's ordering-column values, in order. +// PageToken is the opaque keyset page token of the SQL backends. It carries no +// database topology, only what resumes an ORDER BY ... WHERE (cols) > (last) scan. +type PageToken struct { + Version int `json:"v"` + Kind Kind `json:"kind"` + Scope string `json:"scope"` // atespace name for scoped actor listing; empty otherwise. + Last []string `json:"last"` // last row's ordering-column values, in order. } -func encodePageToken(kind resourceKind, scope string, last []string) string { - b, _ := json.Marshal(pageToken{Version: pageTokenVersion, Kind: kind, Scope: scope, Last: last}) +// EncodePageToken builds the token that resumes a listing after last. +func EncodePageToken(kind Kind, scope string, last []string) string { + b, _ := json.Marshal(PageToken{Version: pageTokenVersion, Kind: kind, Scope: scope, Last: last}) return base64.StdEncoding.EncodeToString(b) } -// decodePageToken decodes tokenStr and validates it was issued for the same +// DecodePageToken decodes tokenStr and validates it was issued for the same // method/scope it's now being presented to. An empty tokenStr decodes to a // zero-value (start of list) token. -func decodePageToken(tokenStr string, wantKind resourceKind, wantScope string, wantKeyParts int) (pageToken, error) { +func DecodePageToken(tokenStr string, wantKind Kind, wantScope string, wantKeyParts int) (PageToken, error) { if tokenStr == "" { - return pageToken{Version: pageTokenVersion, Kind: wantKind, Scope: wantScope}, nil + return PageToken{Version: pageTokenVersion, Kind: wantKind, Scope: wantScope}, nil } b, err := base64.StdEncoding.DecodeString(tokenStr) if err != nil { - return pageToken{}, fmt.Errorf("%w: %v", store.ErrInvalidPageToken, err) + return PageToken{}, fmt.Errorf("%w: %v", store.ErrInvalidPageToken, err) } - var token pageToken + var token PageToken if err := json.Unmarshal(b, &token); err != nil { - return pageToken{}, fmt.Errorf("%w: %v", store.ErrInvalidPageToken, err) + return PageToken{}, fmt.Errorf("%w: %v", store.ErrInvalidPageToken, err) } if token.Version != pageTokenVersion { - return pageToken{}, fmt.Errorf("%w: unsupported version %d", store.ErrInvalidPageToken, token.Version) + return PageToken{}, fmt.Errorf("%w: unsupported version %d", store.ErrInvalidPageToken, token.Version) } if token.Kind != wantKind { - return pageToken{}, fmt.Errorf("%w: for %q, used with %q", store.ErrInvalidPageToken, token.Kind, wantKind) + return PageToken{}, fmt.Errorf("%w: for %q, used with %q", store.ErrInvalidPageToken, token.Kind, wantKind) } if token.Scope != wantScope { - return pageToken{}, fmt.Errorf("%w: for scope %q, used with scope %q", store.ErrInvalidPageToken, token.Scope, wantScope) + return PageToken{}, fmt.Errorf("%w: for scope %q, used with scope %q", store.ErrInvalidPageToken, token.Scope, wantScope) } if len(token.Last) != wantKeyParts { - return pageToken{}, fmt.Errorf("%w: got %d key parts, want %d", store.ErrInvalidPageToken, len(token.Last), wantKeyParts) + return PageToken{}, fmt.Errorf("%w: got %d key parts, want %d", store.ErrInvalidPageToken, len(token.Last), wantKeyParts) } return token, nil } diff --git a/cmd/ateapi/internal/store/storesql/pagetoken_test.go b/cmd/ateapi/internal/store/storesql/pagetoken_test.go new file mode 100644 index 0000000000..dd0b15c949 --- /dev/null +++ b/cmd/ateapi/internal/store/storesql/pagetoken_test.go @@ -0,0 +1,69 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package storesql + +import ( + "encoding/base64" + "encoding/json" + "errors" + "testing" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" +) + +func TestDecodePageToken(t *testing.T) { + encodeRaw := func(token PageToken) string { + b, err := json.Marshal(token) + if err != nil { + t.Fatalf("marshaling token: %v", err) + } + return base64.StdEncoding.EncodeToString(b) + } + teamAActors := EncodePageToken(KindActor, "team-a", []string{"a1"}) + + tests := []struct { + name string + token string + kind Kind + scope string + keyParts int + wantError bool + }{ + {name: "matching kind and scope", token: teamAActors, kind: KindActor, scope: "team-a", keyParts: 1}, + {name: "empty token starts the list", token: "", kind: KindActor, scope: "team-a", keyParts: 1}, + {name: "other scope", token: teamAActors, kind: KindActor, scope: "team-b", keyParts: 1, wantError: true}, + {name: "other kind", token: teamAActors, kind: KindActorTemplate, scope: "team-a", keyParts: 1, wantError: true}, + {name: "wrong key shape", token: EncodePageToken(KindActor, "", []string{"only-an-atespace"}), kind: KindActor, scope: "", keyParts: 2, wantError: true}, + {name: "unsupported version", token: encodeRaw(PageToken{Version: pageTokenVersion + 1, Kind: KindActor, Scope: "team-a", Last: []string{"a1"}}), kind: KindActor, scope: "team-a", keyParts: 1, wantError: true}, + {name: "malformed json", token: base64.StdEncoding.EncodeToString([]byte("{")), kind: KindActor, scope: "", keyParts: 2, wantError: true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := DecodePageToken(tt.token, tt.kind, tt.scope, tt.keyParts) + if tt.wantError { + if !errors.Is(err, store.ErrInvalidPageToken) { + t.Errorf("DecodePageToken() error = %v, want ErrInvalidPageToken", err) + } + return + } + if err != nil { + t.Fatalf("DecodePageToken() error = %v", err) + } + if got.Kind != tt.kind || got.Scope != tt.scope { + t.Errorf("DecodePageToken() = %+v, want kind %q scope %q", got, tt.kind, tt.scope) + } + }) + } +} diff --git a/cmd/ateapi/internal/store/storesql/pool.go b/cmd/ateapi/internal/store/storesql/pool.go new file mode 100644 index 0000000000..59dd36854d --- /dev/null +++ b/cmd/ateapi/internal/store/storesql/pool.go @@ -0,0 +1,34 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package storesql + +import ( + "runtime" + "time" +) + +// Connection pool defaults for backends built on database/sql. They are the +// pgxpool defaults atepg runs with, so both backends hold and recycle +// connections alike. +const ( + ConnMaxLifetime = time.Hour + ConnMaxIdleTime = 30 * time.Minute +) + +// DefaultMaxConns is the pool size when none is configured: the pgxpool +// default of the larger of 4 and the CPU count. +func DefaultMaxConns() int { + return max(4, runtime.NumCPU()) +} diff --git a/cmd/ateapi/internal/store/storesql/record.go b/cmd/ateapi/internal/store/storesql/record.go new file mode 100644 index 0000000000..6c4b2d8b7a --- /dev/null +++ b/cmd/ateapi/internal/store/storesql/record.go @@ -0,0 +1,113 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package storesql holds the backend-neutral pieces shared by the SQL store +// backends, atepg (PostgreSQL) and atemy (MySQL): proto record handling, +// keyset page tokens, the worker event codec and in-process fan-out, the +// lease renewal loop, and the migration runner. The backends supply every SQL +// statement. +package storesql + +import ( + "fmt" + "strings" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/defaults" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "github.com/google/uuid" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/known/timestamppb" +) + +// UnmarshalStored decodes a stored proto, dropping fields this binary has no +// descriptor for. This means a newer replica can have written such a field. +// It also backfills defaults, including for fields added after the resource +// was stored. +func UnmarshalStored(b []byte, m proto.Message) error { + if err := (proto.UnmarshalOptions{DiscardUnknown: true}).Unmarshal(b, m); err != nil { + return err + } + defaults.Apply(m) + return nil +} + +// UnmarshalRow is UnmarshalStored for a row in a listing. A listing fails as a +// whole on one bad row, so the error names the row. +func UnmarshalRow(b []byte, m proto.Message, kind string, id ...string) error { + if err := UnmarshalStored(b, m); err != nil { + return fmt.Errorf("unmarshaling %s %s: %w", kind, strings.Join(id, "/"), err) + } + return nil +} + +// SetCreateMetadata assigns the identity and first revision of a new resource. +func SetCreateMetadata(metadata *ateapipb.ResourceMetadata) { + metadata.Uid = uuid.NewString() + metadata.Version = 1 + metadata.CreateTime = timestamppb.Now() + metadata.UpdateTime = metadata.CreateTime +} + +// ValidateProtoMetadataMatchesColumns verifies that the metadata in the database +// matches the metadata in the proto. +func ValidateProtoMetadataMatchesColumns(resource string, metadata *ateapipb.ResourceMetadata, uid string, version int64) error { + if metadata.GetUid() != uid { + return fmt.Errorf("%s uid projection %q does not match proto metadata uid %q", resource, uid, metadata.GetUid()) + } + if metadata.GetVersion() != version { + return fmt.Errorf("%s version projection %d does not match proto metadata version %d", resource, version, metadata.GetVersion()) + } + return nil +} + +// SetUpdateMetadata derives the next revision of newMeta from the stored oldMeta. +func SetUpdateMetadata(newMeta, oldMeta *ateapipb.ResourceMetadata) { + newMeta.Uid = oldMeta.Uid + newMeta.Version = oldMeta.Version + 1 + newMeta.CreateTime = oldMeta.CreateTime + newMeta.UpdateTime = timestamppb.Now() +} + +// ValidateUpdateActorTemplateMutation rejects a mutation that changes an +// actor template's identity. +func ValidateUpdateActorTemplateMutation(storedTemplate, mutatedTemplate *ateapipb.ActorTemplate) error { + if stored, mutated := storedTemplate.GetMetadata().GetAtespace(), mutatedTemplate.GetMetadata().GetAtespace(); stored != mutated { + return fmt.Errorf("metadata.atespace is immutable: mutation changed it from %q to %q", stored, mutated) + } + if stored, mutated := storedTemplate.GetMetadata().GetName(), mutatedTemplate.GetMetadata().GetName(); stored != mutated { + return fmt.Errorf("metadata.name is immutable: mutation changed it from %q to %q", stored, mutated) + } + return nil +} + +// ValidateUpdateTagMutation reports a mutation that changed a field of a tag +// that is immutable for its lifetime. +func ValidateUpdateTagMutation(storedTag, mutatedTag *ateapipb.Tag) error { + if stored, mutated := storedTag.GetMetadata().GetAtespace(), mutatedTag.GetMetadata().GetAtespace(); stored != mutated { + return fmt.Errorf("metadata.atespace is immutable: mutation changed it from %q to %q", stored, mutated) + } + if stored, mutated := storedTag.GetMetadata().GetName(), mutatedTag.GetMetadata().GetName(); stored != mutated { + return fmt.Errorf("metadata.name is immutable: mutation changed it from %q to %q", stored, mutated) + } + if stored, mutated := storedTag.GetStatus().GetSnapshot(), mutatedTag.GetStatus().GetSnapshot(); stored != nil && !proto.Equal(stored, mutated) { + return fmt.Errorf("status.snapshot is immutable once set: mutation changed it from %s to %s", stored, mutated) + } + if stored, mutated := storedTag.GetStatus().GetStorageLocation(), mutatedTag.GetStatus().GetStorageLocation(); stored != mutated { + return fmt.Errorf("status.storage_location is immutable: mutation changed it from %q to %q", stored, mutated) + } + if stored, mutated := storedTag.GetStatus().GetActorTemplateUid(), mutatedTag.GetStatus().GetActorTemplateUid(); stored != mutated { + return fmt.Errorf("status.actor_template_uid is immutable: mutation changed it from %q to %q", stored, mutated) + } + return nil +} diff --git a/cmd/ateapi/internal/store/storesql/workerevent.go b/cmd/ateapi/internal/store/storesql/workerevent.go new file mode 100644 index 0000000000..c57d686f9f --- /dev/null +++ b/cmd/ateapi/internal/store/storesql/workerevent.go @@ -0,0 +1,109 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package storesql + +import ( + "context" + "fmt" + "log/slog" + "sync" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" + "google.golang.org/protobuf/proto" +) + +// MarshalWorkerEvent encodes an outbox payload: one event-type byte followed by +// the binary Worker proto. The tag byte is read by other replicas during +// rolling deploys, so store.WorkerEventType values must stay append-only +// stable and fit a byte. +func MarshalWorkerEvent(eventType store.WorkerEventType, worker *ateapipb.Worker) ([]byte, error) { + b, err := proto.Marshal(worker) + if err != nil { + return nil, fmt.Errorf("in proto.Marshal: %w", err) + } + return append([]byte{byte(eventType)}, b...), nil +} + +// UnmarshalWorkerEvent decodes a payload written by MarshalWorkerEvent. +func UnmarshalWorkerEvent(payload []byte) (store.WorkerEvent, error) { + if len(payload) == 0 { + return store.WorkerEvent{}, fmt.Errorf("empty worker event payload") + } + // Assert invariants at the boundary. Corrupted payloads or unknown types + // must fail here to trigger a loud resync, rather than falling through + // downstream as silent no-ops. + eventType := store.WorkerEventType(payload[0]) + switch eventType { + case store.WorkerEventCreated, store.WorkerEventUpdated, store.WorkerEventDeleted: + default: + return store.WorkerEvent{}, fmt.Errorf("unknown worker event type byte %d", payload[0]) + } + worker := &ateapipb.Worker{} + if err := UnmarshalStored(payload[1:], worker); err != nil { + return store.WorkerEvent{}, fmt.Errorf("unmarshaling worker event payload: %w", err) + } + if worker.GetMetadata().GetName() == "" { + return store.WorkerEvent{}, fmt.Errorf("worker event payload has no worker name") + } + return store.WorkerEvent{Type: eventType, Worker: worker}, nil +} + +// Watchers fans committed worker events out to this process's WatchWorkers +// channels. The zero value is ready to use. +type Watchers struct { + mu sync.Mutex + channels map[chan store.WorkerEvent]struct{} +} + +// Add enrolls a WatchWorkers channel to receive locally published events. +func (w *Watchers) Add(ch chan store.WorkerEvent) { + w.mu.Lock() + defer w.mu.Unlock() + if w.channels == nil { + w.channels = make(map[chan store.WorkerEvent]struct{}) + } + w.channels[ch] = struct{}{} +} + +// Remove unenrolls a channel. The caller must call it before closing the +// channel: once it returns, Publish can no longer send on it. +func (w *Watchers) Remove(ch chan store.WorkerEvent) { + w.mu.Lock() + defer w.mu.Unlock() + delete(w.channels, ch) +} + +// Publish hands a committed event to this process's watchers a poll interval +// ahead of the outbox, one copy each. Sends are non-blocking: a watcher with a +// full buffer is skipped and gets the event from the outbox. +func (w *Watchers) Publish(ctx context.Context, payload []byte) { + w.mu.Lock() + defer w.mu.Unlock() + if len(w.channels) == 0 { + return + } + event, err := UnmarshalWorkerEvent(payload) + if err != nil { + slog.ErrorContext(ctx, "decoding locally published worker event failed", slog.Any("err", err)) + return + } + for ch := range w.channels { + select { + case ch <- store.WorkerEvent{Type: event.Type, Worker: proto.Clone(event.Worker).(*ateapipb.Worker)}: + default: + } + } +} diff --git a/cmd/ateapi/internal/store/storesql/workerevent_test.go b/cmd/ateapi/internal/store/storesql/workerevent_test.go new file mode 100644 index 0000000000..c0de42eee1 --- /dev/null +++ b/cmd/ateapi/internal/store/storesql/workerevent_test.go @@ -0,0 +1,55 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package storesql + +import ( + "testing" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/pkg/proto/ateapipb" +) + +// TestUnmarshalWorkerEvent_BoundaryAssertions pins the write-side invariants +// asserted at the read boundary: known event-type byte and a keyable worker. +func TestUnmarshalWorkerEvent_BoundaryAssertions(t *testing.T) { + valid, err := MarshalWorkerEvent(store.WorkerEventUpdated, &ateapipb.Worker{ + Metadata: &ateapipb.ResourceMetadata{Name: "w1"}, + }) + if err != nil { + t.Fatalf("marshal: %v", err) + } + nameless, err := MarshalWorkerEvent(store.WorkerEventDeleted, &ateapipb.Worker{}) + if err != nil { + t.Fatalf("marshal: %v", err) + } + for name, tc := range map[string]struct { + payload []byte + wantErr bool + }{ + "valid": {valid, false}, + "empty": {nil, true}, + "unknown type byte": {[]byte{0xff, 0x00}, true}, + "type byte only": {[]byte{byte(store.WorkerEventCreated)}, true}, // empty proto = nameless + "garbage proto": {append([]byte{byte(store.WorkerEventCreated)}, 0xde, 0xad, 0xbe), true}, + "nameless worker": {nameless, true}, + } { + t.Run(name, func(t *testing.T) { + _, err := UnmarshalWorkerEvent(tc.payload) + if (err != nil) != tc.wantErr { + t.Fatalf("UnmarshalWorkerEvent() err = %v, wantErr = %v", err, tc.wantErr) + } + }) + } +} diff --git a/cmd/ateapi/internal/store/storetest/mysql.go b/cmd/ateapi/internal/store/storetest/mysql.go new file mode 100644 index 0000000000..f96e03053e --- /dev/null +++ b/cmd/ateapi/internal/store/storetest/mysql.go @@ -0,0 +1,134 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package storetest + +import ( + "context" + "database/sql" + "fmt" + "os" + "sync" + "testing" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/atemy" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/dockerenv" + gomysql "github.com/go-sql-driver/mysql" + "github.com/testcontainers/testcontainers-go/modules/mysql" +) + +var ( + mysqlOnce sync.Once + mysqlAdmin *sql.DB + mysqlAdminDSN string + containerMySQL *mysql.MySQLContainer + mysqlErr error +) + +// SetupMySQLPersistence returns an isolated atemy persistence instance. +func SetupMySQLPersistence(t *testing.T) *atemy.Persistence { + t.Helper() + ctx := context.Background() + admin := requireMySQLAdmin(t) + databaseName := nextDatabaseName() + if _, err := admin.ExecContext(ctx, "CREATE DATABASE "+databaseName); err != nil { + t.Fatalf("creating MySQL test database: %v", err) + } + + cfg, err := gomysql.ParseDSN(mysqlAdminDSN) + if err != nil { + t.Fatalf("parsing MySQL test DSN: %v", err) + } + cfg.DBName = databaseName + db, err := atemy.Open(cfg.FormatDSN()) + if err != nil { + t.Fatalf("connecting to MySQL test database: %v", err) + } + persistence, err := atemy.NewPersistence(ctx, db) + if err != nil { + db.Close() + t.Fatalf("creating MySQL persistence: %v", err) + } + t.Cleanup(func() { + persistence.Close() + db.Close() + if _, err := admin.ExecContext(context.Background(), "DROP DATABASE "+databaseName); err != nil { + t.Errorf("dropping MySQL test database: %v", err) + } + }) + wirePolicyManager(t, ctx, persistence, authz.MySQLBackend(db)) + return persistence +} + +func shutdownMySQL() { + if mysqlAdmin != nil { + mysqlAdmin.Close() + mysqlAdmin = nil + } + if containerMySQL != nil { + if err := containerMySQL.Terminate(context.Background()); err != nil { + fmt.Fprintf(os.Stderr, "terminating MySQL testcontainer: %v\n", err) + } + containerMySQL = nil + } +} + +// requireMySQLAdmin starts the shared MySQL container as root, which test +// setup needs to create a database per test. Production atemy never creates +// databases. +func requireMySQLAdmin(t *testing.T) *sql.DB { + t.Helper() + mysqlOnce.Do(func() { + ctx := context.Background() + if err := dockerenv.Configure(ctx); err != nil { + mysqlErr = err + return + } + containerMySQL, mysqlErr = mysql.Run(ctx, "mysql:8.4", + mysql.WithDatabase("storetest"), + mysql.WithUsername("root"), + mysql.WithPassword("root"), + ) + if mysqlErr != nil { + return + } + mysqlAdminDSN, mysqlErr = containerMySQL.ConnectionString(ctx) + if mysqlErr != nil { + return + } + mysqlAdmin, mysqlErr = atemy.Open(mysqlAdminDSN) + if mysqlErr != nil { + return + } + var pingErr error + for range 30 { + if pingErr = mysqlAdmin.PingContext(ctx); pingErr == nil { + return + } + time.Sleep(500 * time.Millisecond) + } + mysqlAdmin.Close() + mysqlAdmin = nil + mysqlErr = fmt.Errorf("pinging MySQL testcontainer after retries: %w", pingErr) + }) + if mysqlErr != nil { + if dockerenv.Required() { + t.Fatalf("MySQL testcontainer unavailable and required (CI or REQUIRE_DOCKER is set): %v", mysqlErr) + } + t.Skipf("MySQL testcontainer unavailable (requires Docker): %v", mysqlErr) + } + return mysqlAdmin +} diff --git a/cmd/ateapi/internal/store/storetest/postgres.go b/cmd/ateapi/internal/store/storetest/postgres.go new file mode 100644 index 0000000000..7ea5a68a1e --- /dev/null +++ b/cmd/ateapi/internal/store/storetest/postgres.go @@ -0,0 +1,128 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package storetest + +import ( + "context" + "fmt" + "os" + "sync" + "testing" + "time" + + "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/atepg" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/dockerenv" + "github.com/jackc/pgx/v5/pgxpool" + "github.com/testcontainers/testcontainers-go/modules/postgres" +) + +var ( + pgOnce sync.Once + adminPool *pgxpool.Pool + containerPG *postgres.PostgresContainer + pgErr error +) + +// SetupPostgresPersistence returns an isolated atepg persistence instance. +func SetupPostgresPersistence(t *testing.T) *atepg.Persistence { + t.Helper() + ctx := context.Background() + admin := requireAdminPool(t) + databaseName := nextDatabaseName() + if _, err := admin.Exec(ctx, "CREATE DATABASE "+databaseName); err != nil { + t.Fatalf("creating PostgreSQL test database: %v", err) + } + + config := admin.Config().Copy() + config.ConnConfig.Database = databaseName + pool, err := pgxpool.NewWithConfig(ctx, config) + if err != nil { + t.Fatalf("connecting to PostgreSQL test database: %v", err) + } + persistence, err := atepg.NewPersistence(ctx, pool) + if err != nil { + pool.Close() + t.Fatalf("creating PostgreSQL persistence: %v", err) + } + t.Cleanup(func() { + persistence.Close() + pool.Close() + if _, err := admin.Exec(context.Background(), "DROP DATABASE "+databaseName); err != nil { + t.Errorf("dropping PostgreSQL test database: %v", err) + } + }) + wirePolicyManager(t, ctx, persistence, authz.PostgresBackend(pool)) + return persistence +} + +func shutdownPostgres() { + if adminPool != nil { + adminPool.Close() + adminPool = nil + } + if containerPG != nil { + if err := containerPG.Terminate(context.Background()); err != nil { + fmt.Fprintf(os.Stderr, "terminating PostgreSQL testcontainer: %v\n", err) + } + containerPG = nil + } +} + +func requireAdminPool(t *testing.T) *pgxpool.Pool { + t.Helper() + pgOnce.Do(func() { + ctx := context.Background() + if err := dockerenv.Configure(ctx); err != nil { + pgErr = err + return + } + containerPG, pgErr = postgres.Run(ctx, "postgres:18-alpine", + postgres.WithDatabase("postgres"), + postgres.WithUsername("postgres"), + postgres.WithPassword("postgres"), + ) + if pgErr != nil { + return + } + dsn, err := containerPG.ConnectionString(ctx, "sslmode=disable") + if err != nil { + pgErr = err + return + } + adminPool, pgErr = pgxpool.New(ctx, dsn) + if pgErr != nil { + return + } + var pingErr error + for i := 0; i < 30; i++ { + pingErr = adminPool.Ping(ctx) + if pingErr == nil { + return + } + time.Sleep(500 * time.Millisecond) + } + adminPool.Close() + adminPool = nil + pgErr = fmt.Errorf("pinging PostgreSQL testcontainer after retries: %w", pingErr) + }) + if pgErr != nil { + if dockerenv.Required() { + t.Fatalf("PostgreSQL testcontainer unavailable and required (CI or REQUIRE_DOCKER is set): %v", pgErr) + } + t.Skipf("PostgreSQL testcontainer unavailable (requires Docker): %v", pgErr) + } + return adminPool +} diff --git a/cmd/ateapi/internal/store/storetest/storetest.go b/cmd/ateapi/internal/store/storetest/storetest.go index ed8e457d7b..0455ef6a6b 100644 --- a/cmd/ateapi/internal/store/storetest/storetest.go +++ b/cmd/ateapi/internal/store/storetest/storetest.go @@ -12,11 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package storetest provides isolated PostgreSQL-backed stores for tests. +// Package storetest provides isolated database-backed stores for tests. // -// One PostgreSQL container is shared by every test in a package; each test gets -// its own database. Nothing stops that container when the test binary exits, so -// packages using this package must call [Shutdown] from their TestMain. +// SetupTestStore uses PostgreSQL unless ATE_TEST_STORE_BACKEND is "mysql". One +// container per backend is shared by every test in a package; each test gets +// its own database. Nothing stops those containers when the test binary exits, +// so packages using this package must call [Shutdown] from their TestMain. package storetest import ( @@ -24,78 +25,74 @@ import ( "errors" "fmt" "os" - "sync" "sync/atomic" "testing" - "time" "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" - "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/atepg" - "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/dockerenv" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" - "github.com/jackc/pgx/v5/pgxpool" - "github.com/testcontainers/testcontainers-go/modules/postgres" ) -var ( - containerOnce sync.Once - adminPool *pgxpool.Pool - containerPG *postgres.PostgresContainer - containerErr error - databaseCount atomic.Uint64 -) +// BackendEnv selects the database SetupTestStore uses: "postgres" (the +// default) or "mysql". +const BackendEnv = "ATE_TEST_STORE_BACKEND" + +// databaseCount numbers the per-test databases across both backends. +var databaseCount atomic.Uint64 -// SetupTestStore returns a real PostgreSQL-backed store with a database unique +// SetupTestStore returns a real database-backed store with a database unique // to this test. A shared container keeps this suitable for packages that run // subtests in parallel; databases are dropped during cleanup. func SetupTestStore(t *testing.T) (store.Interface, func()) { t.Helper() - return SetupPostgresPersistence(t), func() {} + s, _ := SetupAuthzTestStore(t) + return s, func() {} } -// SetupPostgresPersistence returns an isolated atepg persistence instance. -func SetupPostgresPersistence(t *testing.T) *atepg.Persistence { +// AuthzStore is a test store that also takes an authz policy manager. +type AuthzStore interface { + store.Interface + SetPolicyManager(*authz.PolicyManager) +} + +// SetupAuthzTestStore is SetupTestStore for tests that wire their own OpenFGA +// authorizer: it also returns the authz backend over the store's pool. +func SetupAuthzTestStore(t *testing.T) (AuthzStore, authz.Backend) { t.Helper() - ctx := context.Background() - admin := requireAdminPool(t) - databaseName := fmt.Sprintf("ateapi_test_%d", databaseCount.Add(1)) - if _, err := admin.Exec(ctx, "CREATE DATABASE "+databaseName); err != nil { - t.Fatalf("creating PostgreSQL test database: %v", err) + switch backend := os.Getenv(BackendEnv); backend { + case "", "postgres": + p := SetupPostgresPersistence(t) + return p, authz.PostgresBackend(p.Pool()) + case "mysql": + p := SetupMySQLPersistence(t) + return p, authz.MySQLBackend(p.DB()) + default: + t.Fatalf("%s must be postgres or mysql, got %q", BackendEnv, backend) + return nil, authz.Backend{} } +} - config := admin.Config().Copy() - config.ConnConfig.Database = databaseName - pool, err := pgxpool.NewWithConfig(ctx, config) - if err != nil { - t.Fatalf("connecting to PostgreSQL test database: %v", err) - } - persistence, err := atepg.NewPersistence(ctx, pool) - if err != nil { - pool.Close() - t.Fatalf("creating PostgreSQL persistence: %v", err) - } - t.Cleanup(func() { - pool.Close() - if _, err := admin.Exec(context.Background(), "DROP DATABASE "+databaseName); err != nil { - t.Errorf("dropping PostgreSQL test database: %v", err) - } - }) - // The server always wires a PolicyManager, so test stores do too. - fgaServer, err := authz.NewOpenFGAServer(pool) +func nextDatabaseName() string { + return fmt.Sprintf("ateapi_test_%d", databaseCount.Add(1)) +} + +// wirePolicyManager gives s an OpenFGA PolicyManager on backend. The server +// always wires one, so test stores do too. +func wirePolicyManager(t *testing.T, ctx context.Context, s AuthzStore, backend authz.Backend) { + t.Helper() + fgaServer, err := authz.NewOpenFGAServer(backend) if err != nil { t.Fatalf("creating OpenFGA server: %v", err) } t.Cleanup(fgaServer.Close) - _, policyManager, err := authz.New(ctx, pool, fgaServer, nil) + _, policyManager, err := authz.New(ctx, backend, fgaServer, nil) if err != nil { t.Fatalf("initializing OpenFGA authz: %v", err) } - persistence.SetPolicyManager(policyManager) - return persistence + s.SetPolicyManager(policyManager) } -// MustCreateAtespace creates name unless it already exists. PostgreSQL enforces +// MustCreateAtespace creates name unless it already exists. Both backends enforce // the parent relationship for actor and snapshot records, so test fixtures use // this before seeding those resources. func MustCreateAtespace(t *testing.T, ctx context.Context, s store.Interface, name string) { @@ -136,7 +133,7 @@ func MustCreateTag(t *testing.T, ctx context.Context, s store.Interface, tag *at return created } -// RunTests runs m and terminates the shared PostgreSQL container afterwards. +// RunTests runs m and terminates the shared database containers afterwards. // Packages with no other TestMain work should use it as their whole TestMain; // the rest must call [Shutdown] themselves. func RunTests(m *testing.M) { @@ -145,62 +142,8 @@ func RunTests(m *testing.M) { os.Exit(code) } -// Shutdown terminates the shared PostgreSQL container, if one was started. +// Shutdown terminates the shared database containers, if any were started. func Shutdown() { - if adminPool != nil { - adminPool.Close() - adminPool = nil - } - if containerPG != nil { - if err := containerPG.Terminate(context.Background()); err != nil { - fmt.Fprintf(os.Stderr, "terminating PostgreSQL testcontainer: %v\n", err) - } - containerPG = nil - } -} - -func requireAdminPool(t *testing.T) *pgxpool.Pool { - t.Helper() - containerOnce.Do(func() { - ctx := context.Background() - if err := dockerenv.Configure(ctx); err != nil { - containerErr = err - return - } - containerPG, containerErr = postgres.Run(ctx, "postgres:18-alpine", - postgres.WithDatabase("postgres"), - postgres.WithUsername("postgres"), - postgres.WithPassword("postgres"), - ) - if containerErr != nil { - return - } - dsn, err := containerPG.ConnectionString(ctx, "sslmode=disable") - if err != nil { - containerErr = err - return - } - adminPool, containerErr = pgxpool.New(ctx, dsn) - if containerErr != nil { - return - } - var pingErr error - for i := 0; i < 30; i++ { - pingErr = adminPool.Ping(ctx) - if pingErr == nil { - return - } - time.Sleep(500 * time.Millisecond) - } - adminPool.Close() - adminPool = nil - containerErr = fmt.Errorf("pinging PostgreSQL testcontainer after retries: %w", pingErr) - }) - if containerErr != nil { - if dockerenv.Required() { - t.Fatalf("PostgreSQL testcontainer unavailable and required (CI or REQUIRE_DOCKER is set): %v", containerErr) - } - t.Skipf("PostgreSQL testcontainer unavailable (requires Docker): %v", containerErr) - } - return adminPool + shutdownPostgres() + shutdownMySQL() } diff --git a/cmd/ateapi/main.go b/cmd/ateapi/main.go index 3d2c7b8a91..c0fa840b51 100644 --- a/cmd/ateapi/main.go +++ b/cmd/ateapi/main.go @@ -33,6 +33,8 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/authz" "github.com/agent-substrate/substrate/cmd/ateapi/internal/controlapi" "github.com/agent-substrate/substrate/cmd/ateapi/internal/oidcjwt" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/atemy" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/atepg" "github.com/agent-substrate/substrate/cmd/ateapi/internal/workercache" "github.com/agent-substrate/substrate/cmd/ateapi/internal/workerservice" @@ -51,6 +53,7 @@ import ( "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/s3" + "github.com/go-sql-driver/mysql" "github.com/jackc/pgx/v5/pgconn" "github.com/spf13/pflag" "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" @@ -75,12 +78,18 @@ var ( grpcServerCredBundle = pflag.String("grpc-server-cred-bundle", "", "File with the server TLS credential bundle.") authenticationConfigFile = pflag.String("authentication-config", "", "YAML file configuring trusted JWT providers.") + storeBackend = pflag.String("store-backend", storeBackendPostgres, "Database backend for Substrate state: postgres or mysql.") + storePoolMaxConns = pflag.Int32("store-pool-max-conns", 0, "Maximum connections in the shared Substrate and OpenFGA read/write pool of the selected backend. Does not affect the owner or watch pools. When unset, PostgreSQL uses the DSN or pgxpool default and MySQL uses the larger of 4 and the CPU count.") postgresReadWriteConnectionString = pflag.String("postgres-read-write-connection-string", "", "PostgreSQL connection string (libpq DSN or URI).") postgresOwnerConnectionString = pflag.String("postgres-owner-connection-string", "", "PostgreSQL owner connection string (libpq DSN or URI).") postgresReadWriteRole = pflag.String("postgres-read-write-role", "", "Required PostgreSQL role assumed by read/write connections.") postgresOwnerRole = pflag.String("postgres-owner-role", "", "Required PostgreSQL role assumed by owner connections.") postgresSchema = pflag.String("postgres-schema", "substrate", "PostgreSQL schema for Substrate tables. This overrides a search_path connection parameter.") - postgresPoolMaxConns = pflag.Int32("postgres-pool-max-conns", 0, "Maximum connections in the shared Substrate and OpenFGA read/write PostgreSQL pool. Does not affect the owner or watch pools. The DSN or pgx default is used when unset.") + mysqlReadWriteConnectionString = pflag.String("mysql-read-write-connection-string", "", "MySQL connection string (go-sql-driver DSN) naming the Substrate database.") + mysqlOwnerConnectionString = pflag.String("mysql-owner-connection-string", "", "MySQL owner connection string (go-sql-driver DSN).") + mysqlTLSCAFile = pflag.String("mysql-tls-ca-file", "", "PEM file with the CA that verifies the MySQL server certificate. Empty uses the system roots when TLS is enabled.") + mysqlTLSCertFile = pflag.String("mysql-tls-cert-file", "", "PEM file with the client certificate presented to MySQL.") + mysqlTLSKeyFile = pflag.String("mysql-tls-key-file", "", "PEM file with the private key for --mysql-tls-cert-file.") experimentalEnableAuthz = pflag.Bool("experimental-enable-authz", false, "Enforce OpenFGA authorization checks on all registered RPCs (experimental). AccessPolicy RPCs are always checked.") // TODO: Move the authz settings into the hot-reloadable config proto // (agent-substrate/substrate#2021) once it lands, so bootstrap owner @@ -117,7 +126,7 @@ func main() { serverboot.Fatal(ctx, "Invalid --log-level", err) } if err := loadFlagsFromEnv(); err != nil { - serverboot.Fatal(ctx, "Invalid PostgreSQL configuration", err) + serverboot.Fatal(ctx, "Invalid database configuration", err) } slog.InfoContext(ctx, "ateapi starting", slog.String("version", version.Version)) if *templateResyncInterval < minResyncInterval { @@ -172,14 +181,14 @@ func main() { serverboot.Fatal(ctx, "Failed to initialize JWT providers", err) } - persistence, err := connectStore(shutdownCtx) + backend, err := connectStore(shutdownCtx) if err != nil { serverboot.Fatal(ctx, "Failed to set up persistence backend", err) } - pool := persistence.Pool() - defer pool.Close() - // Backends may run background maintenance rooted in their own context - // (atepg's outbox maintenance loop); stop it on shutdown before closing pool. + persistence := backend.persistence + defer backend.closePool() + // Backends run background maintenance rooted in their own context (the + // outbox maintenance loop); stop it on shutdown before closing the pool. defer persistence.Close() // The authz stack is always wired so policy tuples stay in sync with the @@ -190,12 +199,12 @@ func main() { // AccessPolicy, so the enforced API would be unusable. serverboot.Fatal(ctx, "Invalid flags", fmt.Errorf("--authz-bootstrap-owners must list at least one principal when --experimental-enable-authz is set")) } - fgaServer, err := authz.NewOpenFGAServer(pool) + fgaServer, err := authz.NewOpenFGAServer(backend.authz) if err != nil { serverboot.Fatal(ctx, "Failed to create OpenFGA server", err) } defer fgaServer.Close() - authorizer, policyManager, err := authz.New(shutdownCtx, pool, fgaServer, *authzBootstrapOwners) + authorizer, policyManager, err := authz.New(shutdownCtx, backend.authz, fgaServer, *authzBootstrapOwners) if err != nil { serverboot.Fatal(ctx, "Failed to initialize OpenFGA authz", err) } @@ -393,20 +402,29 @@ func loadFlagsFromEnv() error { {postgresReadWriteRole, "ATE_API_POSTGRES_READ_WRITE_ROLE"}, {postgresOwnerRole, "ATE_API_POSTGRES_OWNER_ROLE"}, {postgresSchema, "ATE_API_POSTGRES_SCHEMA"}, + {mysqlReadWriteConnectionString, "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING"}, + {mysqlOwnerConnectionString, "ATE_API_MYSQL_OWNER_CONNECTION_STRING"}, + {mysqlTLSCAFile, "ATE_API_MYSQL_TLS_CA_FILE"}, + {mysqlTLSCertFile, "ATE_API_MYSQL_TLS_CERT_FILE"}, + {mysqlTLSKeyFile, "ATE_API_MYSQL_TLS_KEY_FILE"}, } for _, o := range overrides { if *o.flag == "@env" { *o.flag = os.Getenv(o.env) } } - if !pflag.CommandLine.Changed("postgres-pool-max-conns") { - if raw, ok := os.LookupEnv("ATE_API_POSTGRES_POOL_MAX_CONNS"); ok && raw != "" { - value, err := strconv.ParseInt(raw, 10, 32) - if err != nil || value <= 0 { - return fmt.Errorf("ATE_API_POSTGRES_POOL_MAX_CONNS must be a positive integer") - } - *postgresPoolMaxConns = int32(value) + if v := os.Getenv("ATE_API_STORE_BACKEND"); v != "" && !pflag.CommandLine.Changed("store-backend") { + *storeBackend = v + } + if *storeBackend != storeBackendPostgres && *storeBackend != storeBackendMySQL { + return fmt.Errorf("--store-backend must be %q or %q, got %q", storeBackendPostgres, storeBackendMySQL, *storeBackend) + } + if raw := os.Getenv("ATE_API_STORE_POOL_MAX_CONNS"); raw != "" && !pflag.CommandLine.Changed("store-pool-max-conns") { + value, err := strconv.ParseInt(raw, 10, 32) + if err != nil || value <= 0 { + return fmt.Errorf("ATE_API_STORE_POOL_MAX_CONNS must be a positive integer") } + *storePoolMaxConns = int32(value) } if v := os.Getenv("ATE_API_EXPERIMENTAL_ENABLE_AUTHZ"); v != "" && !pflag.CommandLine.Changed("experimental-enable-authz") { *experimentalEnableAuthz = (v == "true" || v == "1") @@ -419,12 +437,18 @@ func logFlagValues(ctx context.Context) { slog.String("grpc-listen-addr", *listenAddr), slog.String("grpc-server-cred-bundle", *grpcServerCredBundle), slog.String("authentication-config", *authenticationConfigFile), + slog.String("store-backend", *storeBackend), + slog.Int("store-pool-max-conns", int(*storePoolMaxConns)), postgresConnectionAttr("postgres-read-write-connection-string", *postgresReadWriteConnectionString), postgresConnectionAttr("postgres-owner-connection-string", *postgresOwnerConnectionString), slog.String("postgres-read-write-role", *postgresReadWriteRole), slog.String("postgres-owner-role", *postgresOwnerRole), slog.String("postgres-schema", *postgresSchema), - slog.Int("postgres-pool-max-conns", int(*postgresPoolMaxConns)), + mysqlConnectionAttr("mysql-read-write-connection-string", *mysqlReadWriteConnectionString), + mysqlConnectionAttr("mysql-owner-connection-string", *mysqlOwnerConnectionString), + slog.String("mysql-tls-ca-file", *mysqlTLSCAFile), + slog.String("mysql-tls-cert-file", *mysqlTLSCertFile), + slog.String("mysql-tls-key-file", *mysqlTLSKeyFile), slog.Bool("experimental-enable-authz", *experimentalEnableAuthz), slog.Any("authz-bootstrap-owners", *authzBootstrapOwners), slog.String("actor-id-jwt-pool", *actorIDJWTPoolFile), @@ -490,56 +514,139 @@ func postgresConnectionAttr(key, connString string) slog.Attr { ) } -// connectStore builds the PostgreSQL-backed *atepg.Persistence. Startup fails if -// its configuration is missing or the database can't be reached. -func connectStore(ctx context.Context) (*atepg.Persistence, error) { - if *postgresReadWriteConnectionString == "" { - return nil, fmt.Errorf("--postgres-read-write-connection-string is required") +// mysqlConnectionAttr is the MySQL counterpart of postgresConnectionAttr. TLS +// is on when the DSN asks for it or any --mysql-tls-*-file flag is set. +func mysqlConnectionAttr(key, connString string) slog.Attr { + if connString == "" { + return slog.String(key, "") } - if *postgresPoolMaxConns < 0 { - return nil, fmt.Errorf("--postgres-pool-max-conns must not be negative") + cfg, err := mysql.ParseDSN(connString) + if err != nil { + return slog.String(key, "") } - persistence, err := connectPostgresWithRetries(ctx) + host, port, err := net.SplitHostPort(cfg.Addr) if err != nil { - return nil, fmt.Errorf("setting up PostgreSQL: %w", err) + host = cfg.Addr } - return persistence, nil + portNumber, _ := strconv.Atoi(port) + tlsFiles := *mysqlTLSCAFile != "" || *mysqlTLSCertFile != "" || *mysqlTLSKeyFile != "" + return slog.Group(key, + slog.String("host", host), + slog.Int("port", portNumber), + slog.String("database", cfg.DBName), + slog.String("user", cfg.User), + slog.Bool("password-set", cfg.Passwd != ""), + slog.Bool("tls", tlsFiles || (cfg.TLSConfig != "" && cfg.TLSConfig != "false")), + ) } -var ( - postgresConnectTries = 30 - postgresConnectPeriod = 2 * time.Second +const ( + storeBackendPostgres = "postgres" + storeBackendMySQL = "mysql" ) -func connectPostgresWithRetries(ctx context.Context) (*atepg.Persistence, error) { - var connectErr error - for attempt := 1; attempt <= postgresConnectTries; attempt++ { - persistence, err := atepg.Connect(ctx, atepg.ConnectConfig{ +// persistenceBackend is the part of a store backend that main uses beyond +// store.Interface. +type persistenceBackend interface { + store.Interface + Close() + SetPolicyManager(*authz.PolicyManager) +} + +// connectedStore is a connected store backend and the shared read/write pool +// it runs on. The caller owns the pool and closes it after the persistence. +type connectedStore struct { + persistence persistenceBackend + authz authz.Backend + closePool func() +} + +// connectStore connects the backend selected by --store-backend. Startup fails +// if its configuration is missing or the database can't be reached. +func connectStore(ctx context.Context) (connectedStore, error) { + if *storePoolMaxConns < 0 { + return connectedStore{}, fmt.Errorf("--store-pool-max-conns must not be negative") + } + if *storeBackend == storeBackendMySQL { + return connectMySQL(ctx) + } + return connectPostgres(ctx) +} + +func connectPostgres(ctx context.Context) (connectedStore, error) { + if *postgresReadWriteConnectionString == "" { + return connectedStore{}, fmt.Errorf("--postgres-read-write-connection-string is required") + } + persistence, err := connectWithRetries(ctx, "PostgreSQL", atepg.ErrUnavailable, func() (*atepg.Persistence, error) { + return atepg.Connect(ctx, atepg.ConnectConfig{ ReadWriteDSN: *postgresReadWriteConnectionString, OwnerDSN: *postgresOwnerConnectionString, ReadWriteRole: *postgresReadWriteRole, OwnerRole: *postgresOwnerRole, Schema: *postgresSchema, - PoolMaxConns: *postgresPoolMaxConns, + PoolMaxConns: *storePoolMaxConns, + }) + }) + if err != nil { + return connectedStore{}, fmt.Errorf("setting up PostgreSQL: %w", err) + } + pool := persistence.Pool() + return connectedStore{persistence: persistence, authz: authz.PostgresBackend(pool), closePool: pool.Close}, nil +} + +func connectMySQL(ctx context.Context) (connectedStore, error) { + if *mysqlReadWriteConnectionString == "" { + return connectedStore{}, fmt.Errorf("--mysql-read-write-connection-string is required") + } + persistence, err := connectWithRetries(ctx, "MySQL", atemy.ErrUnavailable, func() (*atemy.Persistence, error) { + return atemy.Connect(ctx, atemy.ConnectConfig{ + ReadWriteDSN: *mysqlReadWriteConnectionString, + OwnerDSN: *mysqlOwnerConnectionString, + TLS: atemy.TLSFiles{ + CAFile: *mysqlTLSCAFile, + CertFile: *mysqlTLSCertFile, + KeyFile: *mysqlTLSKeyFile, + }, + PoolMaxConns: *storePoolMaxConns, }) + }) + if err != nil { + return connectedStore{}, fmt.Errorf("setting up MySQL: %w", err) + } + db := persistence.DB() + return connectedStore{persistence: persistence, authz: authz.MySQLBackend(db), closePool: func() { _ = db.Close() }}, nil +} + +var ( + storeConnectTries = 30 + storeConnectPeriod = 2 * time.Second +) + +// connectWithRetries retries connect while it fails with unavailable, so +// ateapi can start before its database is reachable. +func connectWithRetries[T any](ctx context.Context, database string, unavailable error, connect func() (T, error)) (T, error) { + var zero T + var connectErr error + for attempt := 1; attempt <= storeConnectTries; attempt++ { + persistence, err := connect() if err == nil { return persistence, nil } - if !errors.Is(err, atepg.ErrUnavailable) { - return nil, err + if !errors.Is(err, unavailable) { + return zero, err } connectErr = err - slog.WarnContext(ctx, "Failed to connect to PostgreSQL, retrying...", slog.Int("attempt", attempt), slog.Any("err", err)) - if attempt == postgresConnectTries { + slog.WarnContext(ctx, "Failed to connect to database, retrying...", slog.String("database", database), slog.Int("attempt", attempt), slog.Any("err", err)) + if attempt == storeConnectTries { break } select { case <-ctx.Done(): - return nil, ctx.Err() - case <-time.After(postgresConnectPeriod): + return zero, ctx.Err() + case <-time.After(storeConnectPeriod): } } - return nil, fmt.Errorf("connect to PostgreSQL after %d attempts: %w", postgresConnectTries, connectErr) + return zero, fmt.Errorf("connect to %s after %d attempts: %w", database, storeConnectTries, connectErr) } // newKubeClients builds the standard Kubernetes clientset and the ate diff --git a/cmd/ateapi/main_test.go b/cmd/ateapi/main_test.go index 4aa3b5922d..e68355c9e3 100644 --- a/cmd/ateapi/main_test.go +++ b/cmd/ateapi/main_test.go @@ -17,80 +17,209 @@ package main import ( "bytes" "context" + "errors" + "fmt" "log/slog" "net/url" "strings" "testing" + "time" + + "github.com/spf13/pflag" ) -func TestConnectStoreRequiresPostgresReadWriteConnectionString(t *testing.T) { - oldDSN := *postgresReadWriteConnectionString - t.Cleanup(func() { - *postgresReadWriteConnectionString = oldDSN - }) - *postgresReadWriteConnectionString = "" +// saveFlag restores a flag variable when the test ends. +func saveFlag[T any](t *testing.T, p *T) { + t.Helper() + old := *p + t.Cleanup(func() { *p = old }) +} + +// markFlagChanged makes a flag look as if it was set on the command line. +func markFlagChanged(t *testing.T, name string) { + t.Helper() + f := pflag.CommandLine.Lookup(name) + old := f.Changed + t.Cleanup(func() { f.Changed = old }) + f.Changed = true +} + +func TestConnectStoreRequiresReadWriteConnectionString(t *testing.T) { + for _, tc := range []struct { + backend string + flag *string + wantErr string + }{ + {storeBackendPostgres, postgresReadWriteConnectionString, "--postgres-read-write-connection-string is required"}, + {storeBackendMySQL, mysqlReadWriteConnectionString, "--mysql-read-write-connection-string is required"}, + } { + t.Run(tc.backend, func(t *testing.T) { + saveFlag(t, storeBackend) + saveFlag(t, tc.flag) + *storeBackend = tc.backend + *tc.flag = "" - _, err := connectStore(context.Background()) - if err == nil || !strings.Contains(err.Error(), "--postgres-read-write-connection-string is required") { - t.Fatalf("connectStore() error = %v, want missing-connection-string error", err) + _, err := connectStore(context.Background()) + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("connectStore() error = %v, want %q", err, tc.wantErr) + } + }) } } -func TestLoadFlagsFromEnvResolvesPostgresSourcesOnce(t *testing.T) { - oldRuntime, oldDDL := *postgresReadWriteConnectionString, *postgresOwnerConnectionString - oldRuntimeRole, oldDDLRole := *postgresReadWriteRole, *postgresOwnerRole - oldAuthz := *experimentalEnableAuthz - t.Cleanup(func() { - *postgresReadWriteConnectionString = oldRuntime - *postgresOwnerConnectionString = oldDDL - *postgresReadWriteRole = oldRuntimeRole - *postgresOwnerRole = oldDDLRole - *experimentalEnableAuthz = oldAuthz - }) - *postgresReadWriteConnectionString = "@env" - *postgresOwnerConnectionString = "@env" - *postgresReadWriteRole = "@env" - *postgresOwnerRole = "@env" +func TestConnectWithRetries(t *testing.T) { + saveFlag(t, &storeConnectTries) + saveFlag(t, &storeConnectPeriod) + storeConnectTries = 3 + storeConnectPeriod = time.Millisecond + unavailable := errors.New("unavailable") + permanent := errors.New("bad credentials") + retryable := fmt.Errorf("dial: %w", unavailable) + + tests := []struct { + name string + errs []error + wantCalls int + wantErr error + }{ + {name: "succeeds after unavailable", errs: []error{retryable, retryable, nil}, wantCalls: 3}, + {name: "other error returns at once", errs: []error{permanent}, wantCalls: 1, wantErr: permanent}, + {name: "unavailable on every try", errs: []error{retryable, retryable, retryable}, wantCalls: 3, wantErr: unavailable}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + calls := 0 + got, err := connectWithRetries(t.Context(), "test", unavailable, func() (int, error) { + if calls == len(tc.errs) { + t.Fatalf("connect called more than %d times", len(tc.errs)) + } + err := tc.errs[calls] + calls++ + if err != nil { + return 0, err + } + return 1, nil + }) + if calls != tc.wantCalls { + t.Errorf("connect called %d times, want %d", calls, tc.wantCalls) + } + if tc.wantErr == nil { + if err != nil || got != 1 { + t.Fatalf("connectWithRetries() = %d, %v, want 1, nil", got, err) + } + return + } + if !errors.Is(err, tc.wantErr) { + t.Fatalf("connectWithRetries() error = %v, want %v", err, tc.wantErr) + } + }) + } +} + +func TestLoadFlagsFromEnvStoreBackend(t *testing.T) { + tests := []struct { + name string + flagValue string + flagChanged bool + env string + want string + wantErr bool + }{ + {name: "env applies when flag unset", flagValue: storeBackendPostgres, env: storeBackendMySQL, want: storeBackendMySQL}, + {name: "flag wins over env", flagValue: storeBackendPostgres, flagChanged: true, env: storeBackendMySQL, want: storeBackendPostgres}, + {name: "invalid env", flagValue: storeBackendPostgres, env: "sqlite", wantErr: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + saveFlag(t, storeBackend) + *storeBackend = tc.flagValue + if tc.flagChanged { + markFlagChanged(t, "store-backend") + } + t.Setenv("ATE_API_STORE_BACKEND", tc.env) + + err := loadFlagsFromEnv() + if tc.wantErr { + if err == nil || !strings.Contains(err.Error(), "--store-backend must be") { + t.Fatalf("loadFlagsFromEnv() error = %v, want backend validation", err) + } + return + } + if err != nil { + t.Fatal(err) + } + if *storeBackend != tc.want { + t.Fatalf("store backend = %q, want %q", *storeBackend, tc.want) + } + }) + } +} + +func TestLoadFlagsFromEnvResolvesSourcesOnce(t *testing.T) { + tests := []struct { + flag *string + env string + }{ + {flag: postgresReadWriteConnectionString, env: "ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING"}, + {flag: postgresOwnerConnectionString, env: "ATE_API_POSTGRES_OWNER_CONNECTION_STRING"}, + {flag: postgresReadWriteRole, env: "ATE_API_POSTGRES_READ_WRITE_ROLE"}, + {flag: postgresOwnerRole, env: "ATE_API_POSTGRES_OWNER_ROLE"}, + {flag: postgresSchema, env: "ATE_API_POSTGRES_SCHEMA"}, + {flag: mysqlReadWriteConnectionString, env: "ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING"}, + {flag: mysqlOwnerConnectionString, env: "ATE_API_MYSQL_OWNER_CONNECTION_STRING"}, + {flag: mysqlTLSCAFile, env: "ATE_API_MYSQL_TLS_CA_FILE"}, + {flag: mysqlTLSCertFile, env: "ATE_API_MYSQL_TLS_CERT_FILE"}, + {flag: mysqlTLSKeyFile, env: "ATE_API_MYSQL_TLS_KEY_FILE"}, + } + for _, tc := range tests { + saveFlag(t, tc.flag) + *tc.flag = "@env" + t.Setenv(tc.env, tc.env+"-a") + } + saveFlag(t, experimentalEnableAuthz) *experimentalEnableAuthz = false - t.Setenv("ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING", "runtime-a") - t.Setenv("ATE_API_POSTGRES_OWNER_CONNECTION_STRING", "ddl-a") - t.Setenv("ATE_API_POSTGRES_READ_WRITE_ROLE", "runtime-role") - t.Setenv("ATE_API_POSTGRES_OWNER_ROLE", "ddl-role") t.Setenv("ATE_API_EXPERIMENTAL_ENABLE_AUTHZ", "true") + check := func() { + t.Helper() + for _, tc := range tests { + if want := tc.env + "-a"; *tc.flag != want { + t.Errorf("%s resolved to %q, want %q", tc.env, *tc.flag, want) + } + } + } if err := loadFlagsFromEnv(); err != nil { t.Fatal(err) } - if *postgresReadWriteConnectionString != "runtime-a" || *postgresOwnerConnectionString != "ddl-a" || - *postgresReadWriteRole != "runtime-role" || *postgresOwnerRole != "ddl-role" { - t.Fatalf("resolved values = %q, %q, %q, %q", *postgresReadWriteConnectionString, *postgresOwnerConnectionString, *postgresReadWriteRole, *postgresOwnerRole) - } + check() if !*experimentalEnableAuthz { - t.Fatal("authorization environment flag was not resolved alongside PostgreSQL settings") + t.Error("authorization environment flag was not resolved") + } + + for _, tc := range tests { + t.Setenv(tc.env, tc.env+"-b") } - t.Setenv("ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING", "runtime-b") - t.Setenv("ATE_API_POSTGRES_OWNER_CONNECTION_STRING", "ddl-b") if err := loadFlagsFromEnv(); err != nil { t.Fatal(err) } - if *postgresReadWriteConnectionString != "runtime-a" || *postgresOwnerConnectionString != "ddl-a" { - t.Fatal("environment-backed connection strings changed after startup resolution") - } + check() } func TestLoadFlagsFromEnvPoolMaxConns(t *testing.T) { - old := *postgresPoolMaxConns - t.Cleanup(func() { *postgresPoolMaxConns = old }) - t.Setenv("ATE_API_POSTGRES_POOL_MAX_CONNS", "20") + saveFlag(t, storePoolMaxConns) + *storePoolMaxConns = 0 + t.Setenv("ATE_API_STORE_POOL_MAX_CONNS", "20") if err := loadFlagsFromEnv(); err != nil { t.Fatal(err) } - if *postgresPoolMaxConns != 20 { - t.Fatalf("pool max connections = %d, want 20", *postgresPoolMaxConns) + if *storePoolMaxConns != 20 { + t.Fatalf("pool max connections = %d, want 20", *storePoolMaxConns) } - t.Setenv("ATE_API_POSTGRES_POOL_MAX_CONNS", "invalid") - if err := loadFlagsFromEnv(); err == nil || !strings.Contains(err.Error(), "ATE_API_POSTGRES_POOL_MAX_CONNS must be a positive integer") { - t.Fatalf("loadFlagsFromEnv() error = %v, want pool-size validation", err) + for _, raw := range []string{"invalid", "0"} { + t.Setenv("ATE_API_STORE_POOL_MAX_CONNS", raw) + if err := loadFlagsFromEnv(); err == nil || !strings.Contains(err.Error(), "ATE_API_STORE_POOL_MAX_CONNS must be a positive integer") { + t.Fatalf("loadFlagsFromEnv() with %q error = %v, want pool-size validation", raw, err) + } } } @@ -187,18 +316,90 @@ func TestPostgresConnectionAttrNeverLogsThePassword(t *testing.T) { }) } -// TestLogFlagValuesDoesNotLogThePostgresPassword goes through the real startup +func TestMySQLConnectionAttrNeverLogsThePassword(t *testing.T) { + const password = "hunter2-very-secret" + render := func(connString string) string { + var buf bytes.Buffer + slog.New(slog.NewJSONHandler(&buf, nil)).LogAttrs(context.Background(), slog.LevelInfo, "Final flag values", mysqlConnectionAttr("mysql-connection-string", connString)) + return buf.String() + } + + for name, tc := range map[string]struct { + connString string + want []string + }{ + "tcp": { + connString: "ateapi:" + password + "@tcp(db.example.internal:3307)/substrate?parseTime=true", + want: []string{`"host":"db.example.internal"`, `"port":3307`, `"database":"substrate"`, `"user":"ateapi"`, `"password-set":true`, `"tls":false`}, + }, + "reserved_characters": { + connString: "ateapi:" + password + "@:x@tcp(db.example.internal:3306)/substrate", + want: []string{`"host":"db.example.internal"`, `"port":3306`, `"password-set":true`}, + }, + "tls": { + connString: "ateapi:" + password + "@tcp(db.example.internal:3306)/substrate?tls=true", + want: []string{`"tls":true`}, + }, + "tls_false": { + connString: "ateapi:" + password + "@tcp(db.example.internal:3306)/substrate?tls=false", + want: []string{`"tls":false`}, + }, + "passwordless": { + connString: "ateapi@tcp(mysql.ate-system.svc:3306)/substrate", + want: []string{`"user":"ateapi"`, `"password-set":false`}, + }, + } { + t.Run(name, func(t *testing.T) { + got := render(tc.connString) + if strings.Contains(got, password) { + t.Fatalf("log line contains the password: %s", got) + } + for _, want := range tc.want { + if !strings.Contains(got, want) { + t.Errorf("log line missing %s: %s", want, got) + } + } + }) + } + + t.Run("tls_files", func(t *testing.T) { + saveFlag(t, mysqlTLSCAFile) + *mysqlTLSCAFile = "/run/mysql-server-ca/server-ca.pem" + if got := render("ateapi@tcp(db.example.internal:3306)/substrate"); !strings.Contains(got, `"tls":true`) { + t.Errorf("log line reports TLS off with a CA file set: %s", got) + } + }) + + t.Run("missing_database_separator", func(t *testing.T) { + raw := "ateapi:" + password + "@tcp(db.example.internal:3306)" + got := render(raw) + if strings.Contains(got, password) || strings.Contains(got, raw) { + t.Fatalf("log line echoes an unparseable connection string: %s", got) + } + if !strings.Contains(got, `"mysql-connection-string":""`) { + t.Errorf("expected the unparseable marker: %s", got) + } + }) + + t.Run("empty", func(t *testing.T) { + if got := render(""); !strings.Contains(got, `"mysql-connection-string":""`) { + t.Errorf("expected empty marker: %s", got) + } + }) +} + +// TestLogFlagValuesDoesNotLogADatabasePassword goes through the real startup // log call, so a change at the call site (logging the flag directly again) // fails here even if the helper stays correct. -func TestLogFlagValuesDoesNotLogThePostgresPassword(t *testing.T) { +func TestLogFlagValuesDoesNotLogADatabasePassword(t *testing.T) { const password = "hunter2-very-secret" - origReadWrite, origOwner := *postgresReadWriteConnectionString, *postgresOwnerConnectionString - t.Cleanup(func() { - *postgresReadWriteConnectionString = origReadWrite - *postgresOwnerConnectionString = origOwner - }) + for _, p := range []*string{postgresReadWriteConnectionString, postgresOwnerConnectionString, mysqlReadWriteConnectionString, mysqlOwnerConnectionString} { + saveFlag(t, p) + } *postgresReadWriteConnectionString = "postgresql://runtime:" + password + "@db.example.internal:5432/atepg?sslmode=disable" *postgresOwnerConnectionString = "postgresql://owner:" + password + "@db.example.internal:5432/atepg?sslmode=disable" + *mysqlReadWriteConnectionString = "runtime:" + password + "@tcp(db.example.internal:3306)/substrate" + *mysqlOwnerConnectionString = "owner:" + password + "@tcp(db.example.internal:3306)/substrate" var buf bytes.Buffer origLogger := slog.Default() @@ -219,4 +420,9 @@ func TestLogFlagValuesDoesNotLogThePostgresPassword(t *testing.T) { t.Errorf("startup line missing the structured %s summary: %s", key, got) } } + for _, key := range []string{"mysql-read-write-connection-string", "mysql-owner-connection-string"} { + if !strings.Contains(got, `"`+key+`":{"host":"db.example.internal","port":3306`) { + t.Errorf("startup line missing the structured %s summary: %s", key, got) + } + } } diff --git a/docs/architecture.md b/docs/architecture.md index effb2c23b0..9225600d0d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -229,7 +229,7 @@ administrative operations and actor environment definitions. These resources represent the high-frequency, ephemeral state of individual actors and workers. They are stored in a high-performance, low-latency state -store (PostgreSQL) to support real-time operations. +store (PostgreSQL or MySQL) to support real-time operations. * **Actor**: A specific instance of an ActorTemplate. An Actor record tracks its globally unique identifier, physical location (Worker IP), current @@ -310,7 +310,7 @@ The brain of the system. It exposes a gRPC API for the data plane and CLI to manage actor lifecycles. * **State Store**: Tracks the mapping of Actors to Workers in a - PostgreSQL store. + PostgreSQL or MySQL store. * **Scheduler**: Selects a ready worker for a resumption request. diff --git a/docs/code-style-guide.md b/docs/code-style-guide.md index 16f597930b..a315c0a39b 100644 --- a/docs/code-style-guide.md +++ b/docs/code-style-guide.md @@ -37,8 +37,9 @@ far from its cause. - Standard library `testing` only — no assertion or mocking frameworks. - Table-driven tests with `t.Run` subtests are the default shape. -- Prefer a real test implementation when one exists: the PostgreSQL test fixture for the store, `envtest` for - the Kubernetes API. Release resources with `t.Cleanup`. +- Prefer a real test implementation when one exists: the `storetest` fixture for the store (PostgreSQL, or + MySQL with `ATE_TEST_STORE_BACKEND=mysql`), `envtest` for the Kubernetes API. Release resources with + `t.Cleanup`. - A test that skips on a missing precondition must fail on it in CI instead — a skipped test and a passing one are the same exit code. Resolve strictness through a named predicate (`dockerenv.Required()`), and prove the strict diff --git a/docs/dev/best-practices/ci-fail-closed.md b/docs/dev/best-practices/ci-fail-closed.md index d6b2bba341..a4b4785361 100644 --- a/docs/dev/best-practices/ci-fail-closed.md +++ b/docs/dev/best-practices/ci-fail-closed.md @@ -33,7 +33,9 @@ func Required() bool { } ``` -Callers branch on it (`storetest.go:187-192`, `atepg/main_test.go:105-110`): +Callers branch on it (`storetest/postgres.go:133-138`, +`storetest/mysql.go:138-143`, `atepg/main_test.go:105-110`, +`atemy/main_test.go:100-105`): ```go if containerErr != nil { @@ -77,6 +79,7 @@ merging: # Docker precondition, without Docker sudo systemctl stop docker.socket docker.service CI=true go test ./cmd/ateapi/internal/store/... # expect FAIL, not SKIP +CI=true ATE_TEST_STORE_BACKEND=mysql go test ./cmd/ateapi/internal/controlapi/... # same # JUnit precondition, unset CI=true hack/run-e2e.sh ./internal/e2e/suites/example # expect exit 1 diff --git a/docs/dev/cloud-sql-scaling-guide.md b/docs/dev/cloud-sql-scaling-guide.md index 048c3a33be..f76781350a 100644 --- a/docs/dev/cloud-sql-scaling-guide.md +++ b/docs/dev/cloud-sql-scaling-guide.md @@ -35,7 +35,7 @@ Set with `--storage-size` at create time; it only grows afterwards. ## Connection sizing for a target throughput -Set the shared read/write pool limit with `ATE_API_POSTGRES_POOL_MAX_CONNS` at +Set the shared read/write pool limit with `ATE_API_STORE_POOL_MAX_CONNS` at deploy time. Substrate and the embedded OpenFGA server use the same pool. Target active connections equal throughput multiplied by average query latency: @@ -48,7 +48,7 @@ connections ≈ QPS × mean latency in seconds Provision ~2× headroom for bursts. This setting does not affect the owner and watch pools; their fixed limits add up to 5 connections per replica beyond the read/write pool limit (e.g. 4 replicas with -`ATE_API_POSTGRES_POOL_MAX_CONNS=32` can open up to +`ATE_API_STORE_POOL_MAX_CONNS=32` can open up to 4 × (32 + 2 + 3) = 148 connections). An undersized pool causes client-side queuing inside `pgx` rather than database errors. Ensure total connections across all replicas stay within Cloud SQL's limit: diff --git a/docs/dev/postgresql-schema-evolution.md b/docs/dev/schema-evolution.md similarity index 50% rename from docs/dev/postgresql-schema-evolution.md rename to docs/dev/schema-evolution.md index b41afcadc0..a313826662 100644 --- a/docs/dev/postgresql-schema-evolution.md +++ b/docs/dev/schema-evolution.md @@ -1,8 +1,10 @@ -# PostgreSQL schema evolution +# Schema evolution -`ateapi` applies PostgreSQL migrations before it becomes ready. During a rolling update, the previous binary continues to serve requests while the new binary changes the schema. +`ateapi` applies migrations for its store backend, PostgreSQL or MySQL, before it becomes ready. During a rolling update, the previous binary continues to serve requests while the new binary changes the schema. -Goose commits one migration at a time. A failed migration run can leave any completed migration prefix in place. These rules keep the previous binary safe with every migration prefix. +Goose commits one migration at a time. A failed migration run can leave any completed migration prefix in place. These rules keep the previous binary safe with every migration prefix on both backends. + +MySQL adds the constraints in [MySQL migrations](#mysql-migrations). ## Compatibility contract @@ -29,6 +31,8 @@ Do not run a large data backfill during startup. Propose a separate migration pr ## Keep the actors table partitionable +These rules apply to the PostgreSQL schema and queries. The MySQL backend does not partition tables. + To preserve the option to partition the `actors` table by `atespace` or by `name`, and every other table with an `atespace` column by `atespace`, do not add a schema change or a query that introduces: - A unique index or constraint on one of these tables that omits `atespace`, or on `actors` that omits `name`. @@ -47,9 +51,11 @@ This sequence keeps the previous binary compatible during a rollout and a tempor ## Migration file rules -Store migration files in `cmd/ateapi/internal/store/atepg/migrations`. +Store PostgreSQL migration files in `cmd/ateapi/internal/store/atepg/migrations` and MySQL migration files in `cmd/ateapi/internal/store/atemy/migrations`. Make each schema change for both backends in the same change. Give the two files the same number, name, and intent, so the directories stay in step. -Standalone Substrate defaults to the `substrate` schema, including under Kagent's umbrella chart. Set `postgres.schema` in Helm or `ATE_API_POSTGRES_SCHEMA` in local setup to use another schema. +`hack/verify/sql-migrations.sh` applies the rules below to both directories. + +On PostgreSQL, standalone Substrate defaults to the `substrate` schema, including under Kagent's umbrella chart. Set `postgres.schema` in Helm or `ATE_API_POSTGRES_SCHEMA` in local setup to use another schema. - Use the next sequential `NNNNNN_name.sql` filename. - Add exactly one `-- +goose Up` annotation. @@ -57,9 +63,9 @@ Standalone Substrate defaults to the `substrate` schema, including under Kagent' - Do not add down migrations. - Do not use `NO TRANSACTION` or `ENVSUB`. - Do not add SQL transaction control statements. -- Do not use `IF NOT EXISTS` for a schema change. +- Do not use `IF NOT EXISTS` for a PostgreSQL schema change. MySQL needs it; see [MySQL migrations](#mysql-migrations). - Keep each startup migration short. -- A database administrator must configure the owner role's default table and sequence privileges for the read/write role before migrations. These grants do not go to `PUBLIC` or Kagent's roles. In this first pass, migration ledgers receive the same table privileges as other objects. +- On PostgreSQL, a database administrator must configure the owner role's default table and sequence privileges for the read/write role before migrations. These grants do not go to `PUBLIC` or Kagent's roles. In this first pass, migration ledgers receive the same table privileges as other objects. Before the first stable v1 release, developers can change or squash migration files. Recreate a development database after its migration history changes. @@ -67,15 +73,31 @@ After the first stable v1 release, do not change or delete a released migration Do not edit the `schema_migrations` ledger manually. +## MySQL migrations + +MySQL commits each DDL statement on its own, even inside the Goose transaction. A failed MySQL migration file can therefore leave some of its statements applied without a ledger entry, and the next startup runs the whole file again. Each file boundary is still a durable state, but a file is not atomic. + +- Make every MySQL statement safe to run again. Use `CREATE TABLE IF NOT EXISTS`, which the MySQL verifier allows, and seed rows with `INSERT ... ON DUPLICATE KEY UPDATE`. +- MySQL has no `IF NOT EXISTS` for `ALTER TABLE` or `CREATE INDEX`. Put each such statement in its own migration file, so a failure cannot leave a file half applied. Split the PostgreSQL file the same way, so the numbering stays in step. +- A crash after such a statement commits but before Goose records the file still needs a manual repair. +- Order the statements so that every statement prefix is compatible with the previous binary. +- Use only features PlanetScale's Vitess supports. Do not add foreign keys, stored routines, triggers, partitioning, or `CREATE DATABASE`. +- Declare `DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_bin` on new tables, so key columns compare byte for byte. Equality then matches PostgreSQL. Listings sort by byte, which matches PostgreSQL only under the `C` collation. +- Enforce parent and child relationships in the transactions that write them, as the helpers in `cmd/ateapi/internal/store/atemy/relations.go` do. + +See the [MySQL configuration guide](../mysql.md) for the operator side. + ## Before you submit 1. Identify the previous binary reads and writes that use the changed objects. 2. Check those operations against every new migration prefix. 3. Use expand and contract when one prefix would break an operation. 4. Add or update a test for the schema behavior. -5. Run the migration verifier and PostgreSQL store tests. +5. Run the migration verifier and both store test suites. `ATE_TEST_STORE_BACKEND=mysql` reruns the store-backed control plane tests on MySQL, and the `vitess` build tag runs the MySQL store tests through Vitess, as CI does. ```sh -hack/verify/postgresql-migrations.sh -go test ./cmd/ateapi/internal/store/atepg +hack/verify/sql-migrations.sh +go test ./cmd/ateapi/internal/store/atepg ./cmd/ateapi/internal/store/atemy +ATE_TEST_STORE_BACKEND=mysql go test ./cmd/ateapi/internal/controlapi/... ./cmd/ateapi/internal/workerservice/... +go test -tags vitess -run TestVitess ./cmd/ateapi/internal/store/atemy ``` diff --git a/docs/mysql.md b/docs/mysql.md new file mode 100644 index 0000000000..a0261cd0bf --- /dev/null +++ b/docs/mysql.md @@ -0,0 +1,126 @@ +# MySQL Configuration + +## Overview +Substrate can keep its state in MySQL 8.0 or later instead of PostgreSQL. Set `ATE_API_STORE_BACKEND=mysql` (or `--store-backend=mysql` when running `ateapi` directly); the default is `postgres`. The MySQL backend uses only features that PlanetScale's Vitess supports, so it also runs on a PlanetScale database. + +There is no bundled MySQL. The operator provides the server and creates the database. `ateapi` creates its tables at startup but never creates the database. + +`ateapi` checks the server at startup and refuses to start in these cases: + +- `VERSION()` is older than 8.0 or names MariaDB. Vitess reports a version such as `8.0.40-Vitess`, which passes. +- `auto_increment_increment` is not 1, as on multi-primary replication. +- The session `sql_mode` includes neither `STRICT_TRANS_TABLES` nor `STRICT_ALL_TABLES`. +- Substrate tables exist without a `schema_migrations` ledger, as with PostgreSQL. + +## Connection strings +Connection strings use the [go-sql-driver/mysql DSN format](https://github.com/go-sql-driver/mysql#dsn-data-source-name) and must name a database: + +``` +substrate_rw:@tcp(mysql.example.com:3306)/substrate?tls=true +``` + +`ateapi` always sets `parseTime=true`, `loc=UTC`, `clientFoundRows=true`, and `interpolateParams=true`, turns off `multiStatements`, and sets the session's `transaction_isolation` to `READ-COMMITTED`. These values replace any set in the DSN. + +OpenFGA writes `TIMESTAMP` columns with `NOW()`, which follows the session time zone. Run a self-hosted server with a UTC time zone. PlanetScale fixes it to UTC. + +`ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING` (or `--mysql-read-write-connection-string`) and `ATE_API_MYSQL_OWNER_CONNECTION_STRING` (or `--mysql-owner-connection-string`) are both required by `ateapi`, as with PostgreSQL. The installer sets the owner DSN to the read/write DSN when you leave it unset. When running `ateapi` directly, pass the DSNs as flags or use `@env` flags to read these environment variables; the installer manifest already uses `@env`. + +Startup logs the host, port, database, user, and whether TLS is on for each DSN. It never logs the password. + +### Owner and read/write connections +Migrations use the owner connection. Application queries, OpenFGA authorization queries, worker watch polling, and background cleanup use the read/write connection. Both DSNs must name the same database. + +The owner user needs DDL privileges on the database (such as `CREATE`, `ALTER`, `INDEX`, and `DROP`) and `SELECT`, `INSERT`, `UPDATE`, and `DELETE`. The read/write user needs only `SELECT`, `INSERT`, `UPDATE`, and `DELETE`. `ateapi` does not switch roles on MySQL, so each connection runs with the privileges of its DSN user. + +When the installer defaults the owner DSN to the read/write DSN, the read/write user runs migrations and needs the owner privileges. + +For a self-managed server: + +```sql +CREATE DATABASE substrate; +CREATE USER 'substrate_owner'@'%' IDENTIFIED BY ''; +CREATE USER 'substrate_rw'@'%' IDENTIFIED BY ''; +GRANT ALL PRIVILEGES ON substrate.* TO 'substrate_owner'@'%'; +GRANT SELECT, INSERT, UPDATE, DELETE ON substrate.* TO 'substrate_rw'@'%'; +``` + +Every table with text columns declares `utf8mb4`, and Substrate's key columns compare byte for byte. The database default character set does not affect Substrate. + +## TLS +Without TLS settings, connections are unencrypted. There are two ways to turn TLS on. + +- Add `tls=true` to the DSN. The driver verifies the server certificate against the system roots and the DSN host name. +- Set `ATE_API_MYSQL_TLS_CA_FILE`, the client certificate and key (`ATE_API_MYSQL_TLS_CERT_FILE` and `ATE_API_MYSQL_TLS_KEY_FILE`), or all three. The matching `--mysql-tls-*-file` flags work too. These settings turn on TLS with that material and replace the DSN's `tls` setting. + +When the CA file is unset, the system roots verify the server certificate. The server certificate must always match the DSN host name, even with a CA file. That is stricter than PostgreSQL's `sslmode=verify-ca`, which checks only the certificate chain. + +The client certificate and key must be set together, and `ateapi` refuses to start with only one. Both may name the same file. + +`ateapi` reads the files again for every new connection, so rotated certificates apply without a restart. Open connections keep the certificate they started with. + +The TLS files always require TLS. A `tls=preferred` setting in the DSN does not let the driver fall back to an unencrypted connection when the files are set. + +## Pool sizing +`ATE_API_STORE_POOL_MAX_CONNS` (or `--store-pool-max-conns`) caps the read/write pool, which Substrate and OpenFGA share. The same setting sizes the PostgreSQL pool. It must be a positive integer. + +When unset, the limit is the larger of 4 and the CPU count. Connections are replaced after 1 hour, or after 30 minutes idle, as with PostgreSQL. + +The owner pool is capped at 2 connections and keeps none idle. The watch pool is capped at 3 connections and uses the read/write DSN. + +Each `ateapi` replica can therefore open at most the read/write limit plus 5 connections. Set the limit so that every replica fits within the server's connection limit. + +## Startup and migrations +`ateapi` applies pending migrations before it becomes ready. Replicas serialize migration runs with a named lock (`GET_LOCK`) scoped to the database and wait up to 5 minutes for another replica's run. `ateapi` retries the initial connection while the database is unreachable. + +The next startup reruns a migration file that failed partway through. See [Schema evolution](dev/schema-evolution.md#mysql-migrations) for the migration rules. + +## PlanetScale +Use a PlanetScale database on Vitess. Create the database in PlanetScale first, and name it in both DSNs. + +**Keyspace.** Substrate targets an unsharded keyspace. It defines no VSchema or vindexes, and its transactions write several tables that must commit together. Vitess commits multi-shard transactions on a best-effort basis by default. + +**Passwords.** PlanetScale grants access through password roles. The owner DSN needs an `admin` password, since only that role can change the schema. The read/write DSN can use a `readwriter` password. + +**Safe migrations.** A branch with safe migrations enabled rejects direct DDL. `ateapi` runs DDL at startup whenever a migration is pending, including the first install. Keep safe migrations off on the branch that `ateapi` uses. + +Do not apply Substrate migrations through a deploy request. The ledger would not record them, and `ateapi` would try to apply them again. + +**Locks and limits.** The migration lock and the OpenFGA setup lock use `GET_LOCK` at startup. Vitess keeps a session that took a lock on a reserved connection until it disconnects, so `ateapi` closes each lock connection after unlocking. The OpenFGA setup lock waits until it is free, as PostgreSQL's advisory lock does. + +PlanetScale ends a transaction after 20 seconds, which `ateapi` reports as an internal error rather than a version conflict. Size the pool against the database's connection limit as described in [Pool sizing](#pool-sizing). + +**Foreign keys.** Substrate declares no foreign keys, so the PlanetScale foreign key setting does not affect it. + +**TLS.** PlanetScale server certificates are signed by a common system root. Use `tls=true` in the DSN and leave the CA file unset. + +See PlanetScale's documentation on [safe migrations](https://planetscale.com/docs/vitess/schema-changes/safe-migrations), [password roles](https://planetscale.com/docs/vitess/security/password-roles), [secure connections](https://planetscale.com/docs/vitess/connecting/secure-connections), and [system limits](https://planetscale.com/docs/vitess/planetscale-system-limits), and the Vitess [locking functions](https://vitess.io/docs/reference/query-serving/locking-functions) reference. + +## Installer configuration +`ate-setup` and `hack/install-ate.sh` read these variables: + +```sh +export ATE_API_STORE_BACKEND=mysql +export ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING='substrate_rw:@tcp(mysql.example.com:3306)/substrate?tls=true' +export ATE_API_MYSQL_OWNER_CONNECTION_STRING='substrate_owner:@tcp(mysql.example.com:3306)/substrate?tls=true' +export ATE_API_STORE_POOL_MAX_CONNS=20 +./hack/install-ate.sh --deploy-ate-system +``` + +`hack/install-ate.sh --help` lists the TLS variables. See [`cmd/ate-setup/differences.md`](../cmd/ate-setup/differences.md#known-differences-worth-flagging) for how the installer handles the server CA, the backend conflict check, and a redeploy with `ATE_API_STORE_BACKEND` unset. The Cloud SQL helpers in `tools/setup-gcp` support PostgreSQL only. + +## Differences from PostgreSQL +These differences follow from MySQL and Vitess features, and they affect operation. + +- **Worker outbox.** Each worker write takes an `AUTO_INCREMENT` seq in its last statement, so worker writes do not wait on each other. A seq is assigned before its write commits, so a watch can see a later seq first. +- **Pending seqs.** The watch delivers rows past a gap and keeps each skipped seq pending until it commits. A seq pending for more than 2 seconds is probed with a `FOR SHARE NOWAIT` read. The watch drops it only when the probe finds no row and no writer holding it, which means the write rolled back. +- **Event order.** Events for one worker arrive in order, because writes to one worker take its row lock in turn. Events for different workers can arrive out of seq order. +- **Subscribe window.** A new watch tracks seqs still committing among rows written in the last 5 minutes. A write that took its seq earlier than that and commits after the watch starts is not delivered to that watch. +- **Outbox durability.** The MySQL outbox table is durable, so a restart of the same server loses no events. This assumes `innodb_flush_log_at_trx_commit=1`, the default. PostgreSQL uses unlogged partitions and resyncs watchers after a restart. +- **Outbox retention.** Background cleanup deletes the outbox rows older than 15 minutes that it read, in batches. The replica that locks the trim row with `SKIP LOCKED` runs the pass and the others skip it, as with PostgreSQL's elected retention. PostgreSQL drops whole partitions. +- **Parent and child rows.** There are no foreign keys. A transaction that writes a child row takes a shared lock on its parent, and a parent delete takes an exclusive lock first. +- **Leases.** Acquiring a lease takes over an expired row with an `UPDATE`, then inserts a new row if none exists. PostgreSQL uses one conditional upsert. Releasing a lease expires its row instead of deleting it, because inserters racing for a deleted key deadlock in InnoDB. +- **Transactions.** Transactions run at `READ COMMITTED`. InnoDB can pick a transaction as a deadlock victim even when two writes only race to insert the same key, so the store runs it again, up to 5 more times. +- **Deadlock retries.** The retries back off exponentially from 10ms with jitter, about 0.3 to 0.6 seconds in all, which gives the winning transaction time to commit. A deadlock that persists returns a version conflict, which callers may retry. +- **Lock wait timeouts.** A lock wait timeout returns the database error as is, as a lock wait that hits a deadline does on PostgreSQL. +- **Watch resync.** A watch closes so its consumer resyncs from the primary tables when the server changes (`@@server_uuid`, as after a failover), or when the outbox moves behind its cursor (lost writes). It also closes when it falls behind retention, when retention deletes a row it was waiting for, or when it has more than 4096 pending seqs. +- **Policy members.** OpenFGA's MySQL schema holds a tuple user in 256 characters, so the MySQL store rejects an access policy member longer than that once encoded, as an invalid argument. PostgreSQL accepts members up to the API limit of 512 characters. diff --git a/docs/postgres.md b/docs/postgres.md index b13b94f547..52621c3b85 100644 --- a/docs/postgres.md +++ b/docs/postgres.md @@ -1,6 +1,8 @@ # Database Configuration ## Overview +PostgreSQL is Substrate's default store backend. `ATE_API_STORE_BACKEND=mysql` (or `--store-backend=mysql`) selects MySQL instead; see the [MySQL configuration guide](mysql.md). + Substrate supports separate connections for schema ownership (DDL) and normal reads and writes (DML). Migrations and partition maintenance use the owner connection; application queries use the read/write connection. Each connection assumes its configured PostgreSQL role. The bundled development database uses its existing `postgres` account and role for both pools. For restricted runtime access, provision separate owner and read/write roles in an external database. A single login can also be used if it has membership in both roles. @@ -19,7 +21,7 @@ Set `ATE_API_POSTGRES_OWNER_ROLE` and `ATE_API_POSTGRES_READ_WRITE_ROLE` to the User credentials are passed as connection strings through `ATE_API_POSTGRES_OWNER_CONNECTION_STRING` and `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING`. When running `ateapi` directly, pass the DSNs as flags or use `@env` flags to read these environment variables; the installer manifest already uses `@env`. -`ATE_API_POSTGRES_POOL_MAX_CONNS` (or `--postgres-pool-max-conns` when running `ateapi` directly) sets the read/write pool limit after the connection string is loaded. When unset, a `pool_max_conns` value in the DSN or the pgxpool default applies. This setting does not affect the owner or watch pools; they are capped at 2 and 3 connections, respectively. +`ATE_API_STORE_POOL_MAX_CONNS` (or `--store-pool-max-conns` when running `ateapi` directly) sets the read/write pool limit after the connection string is loaded. When unset, a `pool_max_conns` value in the DSN or the pgxpool default applies. This setting does not affect the owner or watch pools; they are capped at 2 and 3 connections, respectively. `ateapi` uses the same `@env` flag pattern for role names and schema. diff --git a/docs/threat-model.md b/docs/threat-model.md index a674f2c8fd..60be0f75e9 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -43,7 +43,7 @@ Substrate is an early, fast moving product. It is full of debate and subject to * **atenet-router:** Substrate runs Envoy with an `ext_proc` external processor to handle Actor ingress. The ext_proc reads and validates `ate-target-actor`, calls the Substrate API to resume the Actor and obtain its current worker assignment, and selects that worker as a dynamic backend. It overwrites the routing header before forwarding. The router then connects with mTLS to `atunnel`; `atunnel` validates the router identity and authorizes the header against the Actor currently assigned to that worker. * **Object Storage:** Used to store actor snapshots. * **Filesystem support:** Container local filesystem is saved in snapshots, future integrations likely to include networked storage. -* **Substrate Database:** PostgreSQL. On GCP, PostgreSQL may be Cloud SQL reached through the Cloud SQL Auth Proxy sidecar (see [tools/setup-gcp/cloud-sql.md](../tools/setup-gcp/cloud-sql.md)); that path adds two egress flows from ate-api-server: HTTPS to `sqladmin.googleapis.com:443` and the proxy tunnel to the instance IP on port 3307, both authenticated via IAM and encrypted. +* **Substrate Database:** PostgreSQL, or an external MySQL 8.0+ or PlanetScale database (see [mysql.md](mysql.md)). On GCP, PostgreSQL may be Cloud SQL reached through the Cloud SQL Auth Proxy sidecar (see [tools/setup-gcp/cloud-sql.md](../tools/setup-gcp/cloud-sql.md)); that path adds two egress flows from ate-api-server: HTTPS to `sqladmin.googleapis.com:443` and the proxy tunnel to the instance IP on port 3307, both authenticated via IAM and encrypted. * **Kubernetes:** The underlying infrastructure that Substrate runs on is expected to be Kubernetes. # Threats and Mitigations diff --git a/go.mod b/go.mod index d5699f40dd..bcb6281bc1 100644 --- a/go.mod +++ b/go.mod @@ -24,6 +24,7 @@ require ( github.com/envoyproxy/go-control-plane/envoy v1.37.1-0.20260812071801-353463cc7248 github.com/fsnotify/fsnotify v1.9.0 github.com/go-logr/logr v1.4.4 + github.com/go-sql-driver/mysql v1.10.1 github.com/google/go-cmp v0.7.0 github.com/google/go-containerregistry v0.21.7 github.com/google/nftables v0.3.0 @@ -43,6 +44,7 @@ require ( github.com/spf13/pflag v1.0.10 github.com/spiffe/go-spiffe/v2 v2.7.0 github.com/testcontainers/testcontainers-go v0.44.0 + github.com/testcontainers/testcontainers-go/modules/mysql v0.44.0 github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0 github.com/vishvananda/netlink v1.3.1 github.com/vishvananda/netns v0.0.5 @@ -88,6 +90,7 @@ require ( cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect cloud.google.com/go/longrunning v0.9.0 // indirect dario.cat/mergo v1.0.2 // indirect + filippo.io/edwards25519 v1.2.0 // indirect github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.33.0 // indirect github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.55.0 // indirect diff --git a/go.sum b/go.sum index ed34379a7c..3da8ff566a 100644 --- a/go.sum +++ b/go.sum @@ -231,8 +231,8 @@ github.com/go-openapi/testify/enable/yaml/v2 v2.6.0 h1:gGHwAJ0R/5jU8BEGDbfRNR3hL github.com/go-openapi/testify/enable/yaml/v2 v2.6.0/go.mod h1:tY+St1SGq4NFl0QIqdTY4aEdbChAHxhyB77XQi9iJCo= github.com/go-openapi/testify/v2 v2.6.0 h1:5PKH2HE7YJ/LuRPQGvSxBRlFXNQhSetBLlGAgUEu3ug= github.com/go-openapi/testify/v2 v2.6.0/go.mod h1:SgsVHtfooshd0tublTtJ50FPKhujf47YRqauXXOUxfw= -github.com/go-sql-driver/mysql v1.10.0 h1:Q+1LV8DkHJvSYAdR83XzuhDaTykuDx0l6fkXxoWCWfw= -github.com/go-sql-driver/mysql v1.10.0/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk= +github.com/go-sql-driver/mysql v1.10.1 h1:arlSnNLq6a5yxGxV7qg9lF4j0C+KwD6NbQyKr9QL6ME= +github.com/go-sql-driver/mysql v1.10.1/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk= github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 h1:p104kn46Q8WdvHunIJ9dAyjPVtrBPhSr3KT2yUst43I= github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= @@ -482,6 +482,8 @@ github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8 github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU= github.com/testcontainers/testcontainers-go v0.44.0 h1:/Fwh6HY1mIikhnm9e7HwoxGycx0lzRAE0f5VQpjFxzI= github.com/testcontainers/testcontainers-go v0.44.0/go.mod h1:IcnwQrYTO86xHXu5bvMaBH7ATlbS3Qn1M1QWW3c66rE= +github.com/testcontainers/testcontainers-go/modules/mysql v0.44.0 h1:oJPJPxNE6YQ0zlq6mZKh06JOlyimCky4ruQUimdDet4= +github.com/testcontainers/testcontainers-go/modules/mysql v0.44.0/go.mod h1:MSOAU6ukCpehJVHQDN1k9JgOZXZuqHD+2pT20M3JkIg= github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0 h1:8fdv/9y3JMxjQ+ULAcOG8RtgeNu5t9XF9LolSXDuTwM= github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0/go.mod h1:CFr2LncGYokw+OKjXcr8ARCKG1SaC2UEnGxFBovE86g= github.com/tklauser/go-sysconf v0.4.0 h1:7H0uAN+7RkwWRaxhYXDLqa5V3LPrJeV8wmD9dRUgPQU= diff --git a/hack/install-ate.sh b/hack/install-ate.sh index a43b218b60..bace6572a4 100755 --- a/hack/install-ate.sh +++ b/hack/install-ate.sh @@ -27,8 +27,9 @@ # # The environment variables the installer reads -- BUCKET_NAME, KO_DOCKER_REPO, # PROJECT_ID, CLUSTER_NAME, CLUSTER_LOCATION, KUBECTL_CONTEXT, NO_DEV_ENV, the -# ATE_API_POSTGRES_* set, and the rest -- are read by ate-setup directly, as is -# .ate-dev-env.sh. They need no translation and are not repeated here. +# ATE_API_POSTGRES_* and ATE_API_MYSQL_* sets, and the rest -- are read by +# ate-setup directly, as is .ate-dev-env.sh. They need no translation and are +# not repeated here. set -o errexit -o nounset -o pipefail @@ -142,10 +143,24 @@ usage() { echo " ATE_API_POSTGRES_CLOUDSQL_GSA GSA email backing Workload Identity + the IAM database user" echo " ATE_API_POSTGRES_CLOUDSQL_IP_TYPE private (default) | public | psc" echo " ATE_API_POSTGRES_CLOUDSQL_IAM_AUTH true (default) | false (password-over-proxy escape hatch)" - echo " ATE_API_POSTGRES_POOL_MAX_CONNS pgxpool max connections per ateapi replica (default: max(4, NumCPU))" + echo " ATE_API_STORE_POOL_MAX_CONNS Read/write pool max connections per ateapi replica, for either backend" + echo " (default: max(4, NumCPU), or pool_max_conns in a PostgreSQL DSN)" echo " ATE_API_POSTGRES_SERVER_CA_FILE PEM file to mount for verify-ca DSNs (non-Cloud-SQL databases)" echo " ATE_API_POSTGRES_SCHEMA Select the Substrate schema (default: substrate)" echo "" + echo "MySQL configuration (MySQL 8.0+ or PlanetScale). The database is external and must" + echo "already exist. ATE_API_POSTGRES_* variables must be unset or empty:" + echo "" + echo " ATE_API_STORE_BACKEND postgres (default) | mysql. mysql skips the bundled PostgreSQL" + echo " ATE_API_MYSQL_READ_WRITE_CONNECTION_STRING" + echo " Required go-sql-driver DSN, e.g." + echo " user:pass@tcp(host:3306)/substrate?tls=true (stored in a Secret)" + echo " ATE_API_MYSQL_OWNER_CONNECTION_STRING Optional owner DSN for migrations; defaults to the read/write DSN" + echo " ATE_API_MYSQL_SERVER_CA_FILE PEM file with the server CA; turns on TLS verified against it" + echo " ATE_API_MYSQL_TLS_CA_FILE Server CA path inside the pod (instead of ATE_API_MYSQL_SERVER_CA_FILE)" + echo " ATE_API_MYSQL_TLS_CERT_FILE Client certificate path inside the pod, e.g. the pod identity bundle" + echo " ATE_API_MYSQL_TLS_KEY_FILE Client key path inside the pod; set with ATE_API_MYSQL_TLS_CERT_FILE" + echo "" echo "Authentication configuration:" echo "" echo " EXPECTED_JWT_ISSUER Issuer URL ate-api-server requires in service account tokens, verbatim." diff --git a/hack/verify/postgresql-migrations.sh b/hack/verify/postgresql-migrations.sh deleted file mode 100755 index 4894cf0e61..0000000000 --- a/hack/verify/postgresql-migrations.sh +++ /dev/null @@ -1,94 +0,0 @@ -#!/usr/bin/env bash - -# Copyright 2026 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -o errexit -o nounset -o pipefail - -ROOT="$(git rev-parse --show-toplevel)" -cd "${ROOT}" - -export LC_ALL=C - -MIGRATIONS_DIR="cmd/ateapi/internal/store/atepg/migrations" - -if (( $# > 1 )); then - echo "Usage: $0 [released-ref]" >&2 - exit 2 -fi - -shopt -s nullglob -migrations=("${MIGRATIONS_DIR}"/*.sql) -if (( ${#migrations[@]} == 0 )); then - echo "Add at least one PostgreSQL migration." >&2 - exit 1 -fi - -expected=1 -for migration in "${migrations[@]}"; do - name="${migration##*/}" - if [[ ! "${name}" =~ ^([0-9]{6})_[a-z0-9_]+\.sql$ ]]; then - echo "Migration file ${name} must match NNNNNN_name.sql." >&2 - exit 1 - fi - version=$((10#${BASH_REMATCH[1]})) - if (( version != expected )); then - printf 'Expected PostgreSQL migration version %06d, but found %s.\n' "${expected}" "${name}" >&2 - exit 1 - fi - if ! grep -Fxq -- '-- +goose Up' "${migration}"; then - echo "Migration file ${name} must contain a Goose Up annotation." >&2 - exit 1 - fi - if grep -Fiq -- '-- +goose Down' "${migration}"; then - echo "Migration file ${name} must not contain a Goose Down migration." >&2 - exit 1 - fi - if grep -Fiq -- '-- +goose no transaction' "${migration}"; then - echo "Migration file ${name} must run in a PostgreSQL transaction." >&2 - exit 1 - fi - if grep -Fiq -- 'IF NOT EXISTS' "${migration}"; then - echo "Migration file ${name} must not contain an IF NOT EXISTS guard." >&2 - exit 1 - fi - if grep -Fiq -- '-- +goose ENVSUB' "${migration}"; then - echo "Migration file ${name} must not use Goose environment substitution." >&2 - exit 1 - fi - if grep -Eiq -- '^[[:space:]]*(BEGIN|START[[:space:]]+TRANSACTION|COMMIT|ROLLBACK)[[:space:]]*;' "${migration}"; then - echo "Migration file ${name} must let Goose control the PostgreSQL transaction." >&2 - exit 1 - fi - expected=$((expected + 1)) -done - -released_ref="${1:-}" -if [[ -z "${released_ref}" ]]; then - while read -r tag; do - if [[ ! "${tag}" =~ ^v[1-9][0-9]*\.[0-9]+\.[0-9]+$ ]]; then - continue - fi - if [[ "$(git ls-tree -r --name-only "${tag}" -- "${MIGRATIONS_DIR}")" == *".sql"* ]]; then - released_ref="${tag}" - break - fi - done < <(git tag --merged HEAD --sort=-version:refname) -fi - -if [[ -n "${released_ref}" ]] && ! git diff --quiet --no-renames --diff-filter=MD "${released_ref}" HEAD -- "${MIGRATIONS_DIR}"; then - echo "Do not change or delete released PostgreSQL migrations." >&2 - git diff --name-only --no-renames --diff-filter=MD "${released_ref}" HEAD -- "${MIGRATIONS_DIR}" >&2 - exit 1 -fi diff --git a/hack/verify/sql-migrations.sh b/hack/verify/sql-migrations.sh new file mode 100755 index 0000000000..df12604908 --- /dev/null +++ b/hack/verify/sql-migrations.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash + +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Checks the PostgreSQL and MySQL Goose migration directories against the +# migration file rules in docs/dev/schema-evolution.md and refuses changes to +# migrations from the newest release tag that has any (or from released-ref). + +set -o errexit -o nounset -o pipefail + +ROOT="$(git rev-parse --show-toplevel)" +cd "${ROOT}" + +export LC_ALL=C + +if (( $# > 1 )); then + echo "Usage: $0 [released-ref]" >&2 + exit 2 +fi + +shopt -s nullglob + +# if_not_exists is "deny" or "allow". MySQL commits DDL statement by statement, +# so its migrations use IF NOT EXISTS to stay rerunnable after a partial run. +verify_migrations() { + local migrations_dir="$1" + local database="$2" + local if_not_exists="$3" + local released_ref="$4" + local migrations migration name version expected tag + + migrations=("${migrations_dir}"/*.sql) + if (( ${#migrations[@]} == 0 )); then + echo "Add at least one ${database} migration." >&2 + exit 1 + fi + + expected=1 + for migration in "${migrations[@]}"; do + name="${migration##*/}" + if [[ ! "${name}" =~ ^([0-9]{6})_[a-z0-9_]+\.sql$ ]]; then + echo "Migration file ${name} must match NNNNNN_name.sql." >&2 + exit 1 + fi + version=$((10#${BASH_REMATCH[1]})) + if (( version != expected )); then + printf 'Expected %s migration version %06d, but found %s.\n' "${database}" "${expected}" "${name}" >&2 + exit 1 + fi + if ! grep -Fxq -- '-- +goose Up' "${migration}"; then + echo "Migration file ${name} must contain a Goose Up annotation." >&2 + exit 1 + fi + if grep -Fiq -- '-- +goose Down' "${migration}"; then + echo "Migration file ${name} must not contain a Goose Down migration." >&2 + exit 1 + fi + if grep -Fiq -- '-- +goose no transaction' "${migration}"; then + echo "Migration file ${name} must not opt out of the Goose transaction." >&2 + exit 1 + fi + if [[ "${if_not_exists}" == "deny" ]] && grep -Fiq -- 'IF NOT EXISTS' "${migration}"; then + echo "Migration file ${name} must not contain an IF NOT EXISTS guard." >&2 + exit 1 + fi + if grep -Fiq -- '-- +goose ENVSUB' "${migration}"; then + echo "Migration file ${name} must not use Goose environment substitution." >&2 + exit 1 + fi + if grep -Eiq -- '^[[:space:]]*(BEGIN|START[[:space:]]+TRANSACTION|COMMIT|ROLLBACK)[[:space:]]*;' "${migration}"; then + echo "Migration file ${name} must let Goose control the ${database} transaction." >&2 + exit 1 + fi + expected=$((expected + 1)) + done + + if [[ -z "${released_ref}" ]]; then + while read -r tag; do + if [[ ! "${tag}" =~ ^v[1-9][0-9]*\.[0-9]+\.[0-9]+$ ]]; then + continue + fi + if [[ "$(git ls-tree -r --name-only "${tag}" -- "${migrations_dir}")" == *".sql"* ]]; then + released_ref="${tag}" + break + fi + done < <(git tag --merged HEAD --sort=-version:refname) + fi + + if [[ -n "${released_ref}" ]] && ! git diff --quiet --no-renames --diff-filter=MD "${released_ref}" HEAD -- "${migrations_dir}"; then + echo "Do not change or delete released ${database} migrations." >&2 + git diff --name-only --no-renames --diff-filter=MD "${released_ref}" HEAD -- "${migrations_dir}" >&2 + exit 1 + fi +} + +verify_migrations "cmd/ateapi/internal/store/atepg/migrations" "PostgreSQL" deny "${1:-}" +verify_migrations "cmd/ateapi/internal/store/atemy/migrations" "MySQL" allow "${1:-}" diff --git a/manifests/ate-install/ate-api-server.yaml b/manifests/ate-install/ate-api-server.yaml index 955d67b849..d05b00f216 100644 --- a/manifests/ate-install/ate-api-server.yaml +++ b/manifests/ate-install/ate-api-server.yaml @@ -94,6 +94,11 @@ spec: - --postgres-read-write-role=@env - --postgres-owner-role=@env - --postgres-schema=@env + - --mysql-read-write-connection-string=@env + - --mysql-owner-connection-string=@env + - --mysql-tls-ca-file=@env + - --mysql-tls-cert-file=@env + - --mysql-tls-key-file=@env - --actor-id-jwt-pool=/run/actor-id-jwt-pool/pool.json - --actor-id-ca-pool=/run/actor-id-ca-pool/pool.json - --atelet-client-cred-bundle=/run/podidentity.podcert.ate.dev/credential-bundle.pem @@ -134,7 +139,7 @@ spec: - configMapRef: name: ate-otel-config # Inject env vars from a ConfigMap created by each developer. This lets - # each developer customize their own PostgreSQL connection settings without having + # each developer customize their own store connection settings without having # to edit this manifest, which can remain constant across all # developers. - configMapRef: @@ -172,6 +177,12 @@ spec: - name: "postgres-server-ca" mountPath: "/run/postgres-server-ca" readOnly: true + # mysql-server-ca: server CA for an external MySQL, read through + # ATE_API_MYSQL_TLS_CA_FILE. Empty unless the mysql-server-ca Secret + # exists. + - name: "mysql-server-ca" + mountPath: "/run/mysql-server-ca" + readOnly: true ports: - containerPort: 443 - name: prometheus @@ -257,6 +268,10 @@ spec: secret: secretName: "postgres-server-ca" optional: true + - name: "mysql-server-ca" + secret: + secretName: "mysql-server-ca" + optional: true --- # Allow voluntary disruptions (node drains, cluster upgrades) to take # at most one API server pod at a time, at any replica count. diff --git a/tools/setup-gcp/cloud-sql.md b/tools/setup-gcp/cloud-sql.md index 578500fdab..462dbd4a27 100644 --- a/tools/setup-gcp/cloud-sql.md +++ b/tools/setup-gcp/cloud-sql.md @@ -158,7 +158,7 @@ Optional environment variables: - `ATE_API_POSTGRES_SCHEMA` — the schema holding the store's tables (default `substrate`). If you override it, create the named schema with the owner role as owner and target it in the grants in section 2. -- `ATE_API_POSTGRES_POOL_MAX_CONNS` — connections in the read/write pool +- `ATE_API_STORE_POOL_MAX_CONNS` — connections in the read/write pool (default: `max(4, NumCPU)`). It does not affect the owner and watch pools, which are capped at 2 and 3 connections. diff --git a/vendor/filippo.io/edwards25519/LICENSE b/vendor/filippo.io/edwards25519/LICENSE new file mode 100644 index 0000000000..6a66aea5ea --- /dev/null +++ b/vendor/filippo.io/edwards25519/LICENSE @@ -0,0 +1,27 @@ +Copyright (c) 2009 The Go Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/vendor/filippo.io/edwards25519/README.md b/vendor/filippo.io/edwards25519/README.md new file mode 100644 index 0000000000..dcdd8d85fc --- /dev/null +++ b/vendor/filippo.io/edwards25519/README.md @@ -0,0 +1,16 @@ +# filippo.io/edwards25519 + +``` +import "filippo.io/edwards25519" +``` + +This library implements the edwards25519 elliptic curve, exposing the necessary APIs to build a wide array of higher-level primitives. +Read the docs at [pkg.go.dev/filippo.io/edwards25519](https://pkg.go.dev/filippo.io/edwards25519). + +The package tracks the upstream standard library package `crypto/internal/fips140/edwards25519` and extends it with additional functionality. + +The code is originally derived from Adam Langley's internal implementation in the Go standard library, and includes George Tankersley's [performance improvements](https://golang.org/cl/71950). It was then further developed by Henry de Valence for use in ristretto255, and was finally [merged back into the Go standard library](https://golang.org/cl/276272) as of Go 1.17. + +Most users don't need this package, and should instead use `crypto/ed25519` for signatures, `crypto/ecdh` for Diffie-Hellman, or `github.com/gtank/ristretto255` for prime order group logic. However, for anyone currently using a fork of the internal `edwards25519` package or of `github.com/agl/edwards25519`, this package should be a safer, faster, and more powerful alternative. + +Since this package is meant to curb proliferation of edwards25519 implementations in the Go ecosystem, it welcomes requests for new APIs or reviewable performance improvements. diff --git a/vendor/filippo.io/edwards25519/doc.go b/vendor/filippo.io/edwards25519/doc.go new file mode 100644 index 0000000000..dd2deb6449 --- /dev/null +++ b/vendor/filippo.io/edwards25519/doc.go @@ -0,0 +1,20 @@ +// Copyright (c) 2021 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// Package edwards25519 implements group logic for the twisted Edwards curve +// +// -x^2 + y^2 = 1 + -(121665/121666)*x^2*y^2 +// +// This is better known as the Edwards curve equivalent to Curve25519, and is +// the curve used by the Ed25519 signature scheme. +// +// Most users don't need this package, and should instead use crypto/ed25519 for +// signatures, crypto/ecdh for Diffie-Hellman, or github.com/gtank/ristretto255 +// for prime order group logic. +// +// However, developers who do need to interact with low-level edwards25519 +// operations can use this package, which is an extended version of +// crypto/internal/fips140/edwards25519 from the standard library repackaged as +// an importable module. +package edwards25519 diff --git a/vendor/filippo.io/edwards25519/edwards25519.go b/vendor/filippo.io/edwards25519/edwards25519.go new file mode 100644 index 0000000000..a744da2c6d --- /dev/null +++ b/vendor/filippo.io/edwards25519/edwards25519.go @@ -0,0 +1,427 @@ +// Copyright (c) 2017 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package edwards25519 + +import ( + "errors" + + "filippo.io/edwards25519/field" +) + +// Point types. + +type projP1xP1 struct { + X, Y, Z, T field.Element +} + +type projP2 struct { + X, Y, Z field.Element +} + +// Point represents a point on the edwards25519 curve. +// +// This type works similarly to math/big.Int, and all arguments and receivers +// are allowed to alias. +// +// The zero value is NOT valid, and it may be used only as a receiver. +type Point struct { + // Make the type not comparable (i.e. used with == or as a map key), as + // equivalent points can be represented by different Go values. + _ incomparable + + // The point is internally represented in extended coordinates (X, Y, Z, T) + // where x = X/Z, y = Y/Z, and xy = T/Z per https://eprint.iacr.org/2008/522. + x, y, z, t field.Element +} + +type incomparable [0]func() + +func checkInitialized(points ...*Point) { + for _, p := range points { + if p.x == (field.Element{}) && p.y == (field.Element{}) { + panic("edwards25519: use of uninitialized Point") + } + } +} + +type projCached struct { + YplusX, YminusX, Z, T2d field.Element +} + +type affineCached struct { + YplusX, YminusX, T2d field.Element +} + +// Constructors. + +func (v *projP2) Zero() *projP2 { + v.X.Zero() + v.Y.One() + v.Z.One() + return v +} + +// identity is the point at infinity. +var identity, _ = new(Point).SetBytes([]byte{ + 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}) + +// NewIdentityPoint returns a new Point set to the identity. +func NewIdentityPoint() *Point { + return new(Point).Set(identity) +} + +// generator is the canonical curve basepoint. See TestGenerator for the +// correspondence of this encoding with the values in RFC 8032. +var generator, _ = new(Point).SetBytes([]byte{ + 0x58, 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, + 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, + 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, + 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, 0x66, 0x66}) + +// NewGeneratorPoint returns a new Point set to the canonical generator. +func NewGeneratorPoint() *Point { + return new(Point).Set(generator) +} + +func (v *projCached) Zero() *projCached { + v.YplusX.One() + v.YminusX.One() + v.Z.One() + v.T2d.Zero() + return v +} + +func (v *affineCached) Zero() *affineCached { + v.YplusX.One() + v.YminusX.One() + v.T2d.Zero() + return v +} + +// Assignments. + +// Set sets v = u, and returns v. +func (v *Point) Set(u *Point) *Point { + *v = *u + return v +} + +// Encoding. + +// Bytes returns the canonical 32-byte encoding of v, according to RFC 8032, +// Section 5.1.2. +func (v *Point) Bytes() []byte { + // This function is outlined to make the allocations inline in the caller + // rather than happen on the heap. + var buf [32]byte + return v.bytes(&buf) +} + +func (v *Point) bytes(buf *[32]byte) []byte { + checkInitialized(v) + + var zInv, x, y field.Element + zInv.Invert(&v.z) // zInv = 1 / Z + x.Multiply(&v.x, &zInv) // x = X / Z + y.Multiply(&v.y, &zInv) // y = Y / Z + + out := copyFieldElement(buf, &y) + out[31] |= byte(x.IsNegative() << 7) + return out +} + +var feOne = new(field.Element).One() + +// SetBytes sets v = x, where x is a 32-byte encoding of v. If x does not +// represent a valid point on the curve, SetBytes returns nil and an error and +// the receiver is unchanged. Otherwise, SetBytes returns v. +// +// Note that SetBytes accepts all non-canonical encodings of valid points. +// That is, it follows decoding rules that match most implementations in +// the ecosystem rather than RFC 8032. +func (v *Point) SetBytes(x []byte) (*Point, error) { + // Specifically, the non-canonical encodings that are accepted are + // 1) the ones where the field element is not reduced (see the + // (*field.Element).SetBytes docs) and + // 2) the ones where the x-coordinate is zero and the sign bit is set. + // + // Read more at https://hdevalence.ca/blog/2020-10-04-its-25519am, + // specifically the "Canonical A, R" section. + + y, err := new(field.Element).SetBytes(x) + if err != nil { + return nil, errors.New("edwards25519: invalid point encoding length") + } + + // -x² + y² = 1 + dx²y² + // x² + dx²y² = x²(dy² + 1) = y² - 1 + // x² = (y² - 1) / (dy² + 1) + + // u = y² - 1 + y2 := new(field.Element).Square(y) + u := new(field.Element).Subtract(y2, feOne) + + // v = dy² + 1 + vv := new(field.Element).Multiply(y2, d) + vv = vv.Add(vv, feOne) + + // x = +√(u/v) + xx, wasSquare := new(field.Element).SqrtRatio(u, vv) + if wasSquare == 0 { + return nil, errors.New("edwards25519: invalid point encoding") + } + + // Select the negative square root if the sign bit is set. + xxNeg := new(field.Element).Negate(xx) + xx = xx.Select(xxNeg, xx, int(x[31]>>7)) + + v.x.Set(xx) + v.y.Set(y) + v.z.One() + v.t.Multiply(xx, y) // xy = T / Z + + return v, nil +} + +func copyFieldElement(buf *[32]byte, v *field.Element) []byte { + copy(buf[:], v.Bytes()) + return buf[:] +} + +// Conversions. + +func (v *projP2) FromP1xP1(p *projP1xP1) *projP2 { + v.X.Multiply(&p.X, &p.T) + v.Y.Multiply(&p.Y, &p.Z) + v.Z.Multiply(&p.Z, &p.T) + return v +} + +func (v *projP2) FromP3(p *Point) *projP2 { + v.X.Set(&p.x) + v.Y.Set(&p.y) + v.Z.Set(&p.z) + return v +} + +func (v *Point) fromP1xP1(p *projP1xP1) *Point { + v.x.Multiply(&p.X, &p.T) + v.y.Multiply(&p.Y, &p.Z) + v.z.Multiply(&p.Z, &p.T) + v.t.Multiply(&p.X, &p.Y) + return v +} + +func (v *Point) fromP2(p *projP2) *Point { + v.x.Multiply(&p.X, &p.Z) + v.y.Multiply(&p.Y, &p.Z) + v.z.Square(&p.Z) + v.t.Multiply(&p.X, &p.Y) + return v +} + +// d is a constant in the curve equation. +var d, _ = new(field.Element).SetBytes([]byte{ + 0xa3, 0x78, 0x59, 0x13, 0xca, 0x4d, 0xeb, 0x75, + 0xab, 0xd8, 0x41, 0x41, 0x4d, 0x0a, 0x70, 0x00, + 0x98, 0xe8, 0x79, 0x77, 0x79, 0x40, 0xc7, 0x8c, + 0x73, 0xfe, 0x6f, 0x2b, 0xee, 0x6c, 0x03, 0x52}) +var d2 = new(field.Element).Add(d, d) + +func (v *projCached) FromP3(p *Point) *projCached { + v.YplusX.Add(&p.y, &p.x) + v.YminusX.Subtract(&p.y, &p.x) + v.Z.Set(&p.z) + v.T2d.Multiply(&p.t, d2) + return v +} + +func (v *affineCached) FromP3(p *Point) *affineCached { + v.YplusX.Add(&p.y, &p.x) + v.YminusX.Subtract(&p.y, &p.x) + v.T2d.Multiply(&p.t, d2) + + var invZ field.Element + invZ.Invert(&p.z) + v.YplusX.Multiply(&v.YplusX, &invZ) + v.YminusX.Multiply(&v.YminusX, &invZ) + v.T2d.Multiply(&v.T2d, &invZ) + return v +} + +// (Re)addition and subtraction. + +// Add sets v = p + q, and returns v. +func (v *Point) Add(p, q *Point) *Point { + checkInitialized(p, q) + qCached := new(projCached).FromP3(q) + result := new(projP1xP1).Add(p, qCached) + return v.fromP1xP1(result) +} + +// Subtract sets v = p - q, and returns v. +func (v *Point) Subtract(p, q *Point) *Point { + checkInitialized(p, q) + qCached := new(projCached).FromP3(q) + result := new(projP1xP1).Sub(p, qCached) + return v.fromP1xP1(result) +} + +func (v *projP1xP1) Add(p *Point, q *projCached) *projP1xP1 { + var YplusX, YminusX, PP, MM, TT2d, ZZ2 field.Element + + YplusX.Add(&p.y, &p.x) + YminusX.Subtract(&p.y, &p.x) + + PP.Multiply(&YplusX, &q.YplusX) + MM.Multiply(&YminusX, &q.YminusX) + TT2d.Multiply(&p.t, &q.T2d) + ZZ2.Multiply(&p.z, &q.Z) + + ZZ2.Add(&ZZ2, &ZZ2) + + v.X.Subtract(&PP, &MM) + v.Y.Add(&PP, &MM) + v.Z.Add(&ZZ2, &TT2d) + v.T.Subtract(&ZZ2, &TT2d) + return v +} + +func (v *projP1xP1) Sub(p *Point, q *projCached) *projP1xP1 { + var YplusX, YminusX, PP, MM, TT2d, ZZ2 field.Element + + YplusX.Add(&p.y, &p.x) + YminusX.Subtract(&p.y, &p.x) + + PP.Multiply(&YplusX, &q.YminusX) // flipped sign + MM.Multiply(&YminusX, &q.YplusX) // flipped sign + TT2d.Multiply(&p.t, &q.T2d) + ZZ2.Multiply(&p.z, &q.Z) + + ZZ2.Add(&ZZ2, &ZZ2) + + v.X.Subtract(&PP, &MM) + v.Y.Add(&PP, &MM) + v.Z.Subtract(&ZZ2, &TT2d) // flipped sign + v.T.Add(&ZZ2, &TT2d) // flipped sign + return v +} + +func (v *projP1xP1) AddAffine(p *Point, q *affineCached) *projP1xP1 { + var YplusX, YminusX, PP, MM, TT2d, Z2 field.Element + + YplusX.Add(&p.y, &p.x) + YminusX.Subtract(&p.y, &p.x) + + PP.Multiply(&YplusX, &q.YplusX) + MM.Multiply(&YminusX, &q.YminusX) + TT2d.Multiply(&p.t, &q.T2d) + + Z2.Add(&p.z, &p.z) + + v.X.Subtract(&PP, &MM) + v.Y.Add(&PP, &MM) + v.Z.Add(&Z2, &TT2d) + v.T.Subtract(&Z2, &TT2d) + return v +} + +func (v *projP1xP1) SubAffine(p *Point, q *affineCached) *projP1xP1 { + var YplusX, YminusX, PP, MM, TT2d, Z2 field.Element + + YplusX.Add(&p.y, &p.x) + YminusX.Subtract(&p.y, &p.x) + + PP.Multiply(&YplusX, &q.YminusX) // flipped sign + MM.Multiply(&YminusX, &q.YplusX) // flipped sign + TT2d.Multiply(&p.t, &q.T2d) + + Z2.Add(&p.z, &p.z) + + v.X.Subtract(&PP, &MM) + v.Y.Add(&PP, &MM) + v.Z.Subtract(&Z2, &TT2d) // flipped sign + v.T.Add(&Z2, &TT2d) // flipped sign + return v +} + +// Doubling. + +func (v *projP1xP1) Double(p *projP2) *projP1xP1 { + var XX, YY, ZZ2, XplusYsq field.Element + + XX.Square(&p.X) + YY.Square(&p.Y) + ZZ2.Square(&p.Z) + ZZ2.Add(&ZZ2, &ZZ2) + XplusYsq.Add(&p.X, &p.Y) + XplusYsq.Square(&XplusYsq) + + v.Y.Add(&YY, &XX) + v.Z.Subtract(&YY, &XX) + + v.X.Subtract(&XplusYsq, &v.Y) + v.T.Subtract(&ZZ2, &v.Z) + return v +} + +// Negation. + +// Negate sets v = -p, and returns v. +func (v *Point) Negate(p *Point) *Point { + checkInitialized(p) + v.x.Negate(&p.x) + v.y.Set(&p.y) + v.z.Set(&p.z) + v.t.Negate(&p.t) + return v +} + +// Equal returns 1 if v is equivalent to u, and 0 otherwise. +func (v *Point) Equal(u *Point) int { + checkInitialized(v, u) + + var t1, t2, t3, t4 field.Element + t1.Multiply(&v.x, &u.z) + t2.Multiply(&u.x, &v.z) + t3.Multiply(&v.y, &u.z) + t4.Multiply(&u.y, &v.z) + + return t1.Equal(&t2) & t3.Equal(&t4) +} + +// Constant-time operations + +// Select sets v to a if cond == 1 and to b if cond == 0. +func (v *projCached) Select(a, b *projCached, cond int) *projCached { + v.YplusX.Select(&a.YplusX, &b.YplusX, cond) + v.YminusX.Select(&a.YminusX, &b.YminusX, cond) + v.Z.Select(&a.Z, &b.Z, cond) + v.T2d.Select(&a.T2d, &b.T2d, cond) + return v +} + +// Select sets v to a if cond == 1 and to b if cond == 0. +func (v *affineCached) Select(a, b *affineCached, cond int) *affineCached { + v.YplusX.Select(&a.YplusX, &b.YplusX, cond) + v.YminusX.Select(&a.YminusX, &b.YminusX, cond) + v.T2d.Select(&a.T2d, &b.T2d, cond) + return v +} + +// CondNeg negates v if cond == 1 and leaves it unchanged if cond == 0. +func (v *projCached) CondNeg(cond int) *projCached { + v.YplusX.Swap(&v.YminusX, cond) + v.T2d.Select(new(field.Element).Negate(&v.T2d), &v.T2d, cond) + return v +} + +// CondNeg negates v if cond == 1 and leaves it unchanged if cond == 0. +func (v *affineCached) CondNeg(cond int) *affineCached { + v.YplusX.Swap(&v.YminusX, cond) + v.T2d.Select(new(field.Element).Negate(&v.T2d), &v.T2d, cond) + return v +} diff --git a/vendor/filippo.io/edwards25519/extra.go b/vendor/filippo.io/edwards25519/extra.go new file mode 100644 index 0000000000..ee9b5ca5bf --- /dev/null +++ b/vendor/filippo.io/edwards25519/extra.go @@ -0,0 +1,401 @@ +// Copyright (c) 2021 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package edwards25519 + +// This file contains additional functionality that is not included in the +// upstream crypto/internal/edwards25519 package. + +import ( + "errors" + "slices" + + "filippo.io/edwards25519/field" +) + +// ExtendedCoordinates returns v in extended coordinates (X:Y:Z:T) where +// x = X/Z, y = Y/Z, and xy = T/Z as in https://eprint.iacr.org/2008/522. +func (v *Point) ExtendedCoordinates() (X, Y, Z, T *field.Element) { + // This function is outlined to make the allocations inline in the caller + // rather than happen on the heap. Don't change the style without making + // sure it doesn't increase the inliner cost. + var e [4]field.Element + X, Y, Z, T = v.extendedCoordinates(&e) + return +} + +func (v *Point) extendedCoordinates(e *[4]field.Element) (X, Y, Z, T *field.Element) { + checkInitialized(v) + X = e[0].Set(&v.x) + Y = e[1].Set(&v.y) + Z = e[2].Set(&v.z) + T = e[3].Set(&v.t) + return +} + +// SetExtendedCoordinates sets v = (X:Y:Z:T) in extended coordinates where +// x = X/Z, y = Y/Z, and xy = T/Z as in https://eprint.iacr.org/2008/522. +// +// If the coordinates are invalid or don't represent a valid point on the curve, +// SetExtendedCoordinates returns nil and an error and the receiver is +// unchanged. Otherwise, SetExtendedCoordinates returns v. +func (v *Point) SetExtendedCoordinates(X, Y, Z, T *field.Element) (*Point, error) { + if !isOnCurve(X, Y, Z, T) { + return nil, errors.New("edwards25519: invalid point coordinates") + } + v.x.Set(X) + v.y.Set(Y) + v.z.Set(Z) + v.t.Set(T) + return v, nil +} + +func isOnCurve(X, Y, Z, T *field.Element) bool { + var lhs, rhs field.Element + XX := new(field.Element).Square(X) + YY := new(field.Element).Square(Y) + ZZ := new(field.Element).Square(Z) + TT := new(field.Element).Square(T) + // -x² + y² = 1 + dx²y² + // -(X/Z)² + (Y/Z)² = 1 + d(T/Z)² + // -X² + Y² = Z² + dT² + lhs.Subtract(YY, XX) + rhs.Multiply(d, TT).Add(&rhs, ZZ) + if lhs.Equal(&rhs) != 1 { + return false + } + // xy = T/Z + // XY/Z² = T/Z + // XY = TZ + lhs.Multiply(X, Y) + rhs.Multiply(T, Z) + return lhs.Equal(&rhs) == 1 +} + +// BytesMontgomery converts v to a point on the birationally-equivalent +// Curve25519 Montgomery curve, and returns its canonical 32 bytes encoding +// according to RFC 7748. +// +// Note that BytesMontgomery only encodes the u-coordinate, so v and -v encode +// to the same value. If v is the identity point, BytesMontgomery returns 32 +// zero bytes, analogously to the X25519 function. +// +// The lack of an inverse operation (such as SetMontgomeryBytes) is deliberate: +// while every valid edwards25519 point has a unique u-coordinate Montgomery +// encoding, X25519 accepts inputs on the quadratic twist, which don't correspond +// to any edwards25519 point, and every other X25519 input corresponds to two +// edwards25519 points. +func (v *Point) BytesMontgomery() []byte { + // This function is outlined to make the allocations inline in the caller + // rather than happen on the heap. + var buf [32]byte + return v.bytesMontgomery(&buf) +} + +func (v *Point) bytesMontgomery(buf *[32]byte) []byte { + checkInitialized(v) + + // RFC 7748, Section 4.1 provides the bilinear map to calculate the + // Montgomery u-coordinate + // + // u = (1 + y) / (1 - y) + // + // where y = Y / Z and therefore + // + // u = (Z + Y) / (Z - Y) + + var n, r, u field.Element + + n.Add(&v.z, &v.y) // n = Z + Y + r.Invert(r.Subtract(&v.z, &v.y)) // r = 1 / (Z - Y) + u.Multiply(&n, &r) // u = n * r + + return copyFieldElement(buf, &u) +} + +// MultByCofactor sets v = 8 * p, and returns v. +func (v *Point) MultByCofactor(p *Point) *Point { + checkInitialized(p) + result := projP1xP1{} + pp := (&projP2{}).FromP3(p) + result.Double(pp) + pp.FromP1xP1(&result) + result.Double(pp) + pp.FromP1xP1(&result) + result.Double(pp) + return v.fromP1xP1(&result) +} + +// Given k > 0, set s = s**(2*k). +func (s *Scalar) pow2k(k int) { + for i := 0; i < k; i++ { + s.Multiply(s, s) + } +} + +// Invert sets s to the inverse of a nonzero scalar v, and returns s. +// +// If t is zero, Invert returns zero. +func (s *Scalar) Invert(t *Scalar) *Scalar { + // Uses a hardcoded sliding window of width 4. + var table [8]Scalar + var tt Scalar + tt.Multiply(t, t) + table[0] = *t + for i := 0; i < 7; i++ { + table[i+1].Multiply(&table[i], &tt) + } + // Now table = [t**1, t**3, t**5, t**7, t**9, t**11, t**13, t**15] + // so t**k = t[k/2] for odd k + + // To compute the sliding window digits, use the following Sage script: + + // sage: import itertools + // sage: def sliding_window(w,k): + // ....: digits = [] + // ....: while k > 0: + // ....: if k % 2 == 1: + // ....: kmod = k % (2**w) + // ....: digits.append(kmod) + // ....: k = k - kmod + // ....: else: + // ....: digits.append(0) + // ....: k = k // 2 + // ....: return digits + + // Now we can compute s roughly as follows: + + // sage: s = 1 + // sage: for coeff in reversed(sliding_window(4,l-2)): + // ....: s = s*s + // ....: if coeff > 0 : + // ....: s = s*t**coeff + + // This works on one bit at a time, with many runs of zeros. + // The digits can be collapsed into [(count, coeff)] as follows: + + // sage: [(len(list(group)),d) for d,group in itertools.groupby(sliding_window(4,l-2))] + + // Entries of the form (k, 0) turn into pow2k(k) + // Entries of the form (1, coeff) turn into a squaring and then a table lookup. + // We can fold the squaring into the previous pow2k(k) as pow2k(k+1). + + *s = table[1/2] + s.pow2k(127 + 1) + s.Multiply(s, &table[1/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[9/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[11/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[13/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[15/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[7/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[15/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[5/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[1/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[15/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[15/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[7/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[3/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[11/2]) + s.pow2k(5 + 1) + s.Multiply(s, &table[11/2]) + s.pow2k(9 + 1) + s.Multiply(s, &table[9/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[3/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[3/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[3/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[9/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[7/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[3/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[13/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[7/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[9/2]) + s.pow2k(3 + 1) + s.Multiply(s, &table[15/2]) + s.pow2k(4 + 1) + s.Multiply(s, &table[11/2]) + + return s +} + +// MultiScalarMult sets v = sum(scalars[i] * points[i]), and returns v. +// +// Execution time depends only on the lengths of the two slices, which must match. +func (v *Point) MultiScalarMult(scalars []*Scalar, points []*Point) *Point { + if len(scalars) != len(points) { + panic("edwards25519: called MultiScalarMult with different size inputs") + } + checkInitialized(points...) + + // Proceed as in the single-base case, but share doublings + // between each point in the multiscalar equation. + + // Build lookup tables for each point + tables := make([]projLookupTable, 0, 2) // avoid allocation for small sizes + tables = slices.Grow(tables, len(points))[:len(points)] + for i := range tables { + tables[i].FromP3(points[i]) + } + // Compute signed radix-16 digits for each scalar + digits := make([][64]int8, 0, 2) // avoid allocation for small sizes + digits = slices.Grow(digits, len(scalars))[:len(scalars)] + for i := range digits { + digits[i] = scalars[i].signedRadix16() + } + + // Unwrap first loop iteration to save computing 16*identity + multiple := &projCached{} + tmp1 := &projP1xP1{} + tmp2 := &projP2{} + // Lookup-and-add the appropriate multiple of each input point + v.Set(NewIdentityPoint()) + for j := range tables { + tables[j].SelectInto(multiple, digits[j][63]) + tmp1.Add(v, multiple) // tmp1 = v + x_(j,63)*Q in P1xP1 coords + v.fromP1xP1(tmp1) // update v + } + tmp2.FromP3(v) // set up tmp2 = v in P2 coords for next iteration + for i := 62; i >= 0; i-- { + tmp1.Double(tmp2) // tmp1 = 2*(prev) in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 2*(prev) in P2 coords + tmp1.Double(tmp2) // tmp1 = 4*(prev) in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 4*(prev) in P2 coords + tmp1.Double(tmp2) // tmp1 = 8*(prev) in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 8*(prev) in P2 coords + tmp1.Double(tmp2) // tmp1 = 16*(prev) in P1xP1 coords + v.fromP1xP1(tmp1) // v = 16*(prev) in P3 coords + // Lookup-and-add the appropriate multiple of each input point + for j := range tables { + tables[j].SelectInto(multiple, digits[j][i]) + tmp1.Add(v, multiple) // tmp1 = v + x_(j,i)*Q in P1xP1 coords + v.fromP1xP1(tmp1) // update v + } + tmp2.FromP3(v) // set up tmp2 = v in P2 coords for next iteration + } + return v +} + +// VarTimeMultiScalarMult sets v = sum(scalars[i] * points[i]), and returns v. +// +// Execution time depends on the inputs. +func (v *Point) VarTimeMultiScalarMult(scalars []*Scalar, points []*Point) *Point { + if len(scalars) != len(points) { + panic("edwards25519: called VarTimeMultiScalarMult with different size inputs") + } + checkInitialized(points...) + + // Generalize double-base NAF computation to arbitrary sizes. + // Here all the points are dynamic, so we only use the smaller + // tables. + + // Build lookup tables for each point + tables := make([]nafLookupTable5, len(points)) + for i := range tables { + tables[i].FromP3(points[i]) + } + // Compute a NAF for each scalar + nafs := make([][256]int8, len(scalars)) + for i := range nafs { + nafs[i] = scalars[i].nonAdjacentForm(5) + } + + multiple := &projCached{} + tmp1 := &projP1xP1{} + tmp2 := &projP2{} + tmp2.Zero() + + // Move from high to low bits, doubling the accumulator + // at each iteration and checking whether there is a nonzero + // coefficient to look up a multiple of. + // + // Skip trying to find the first nonzero coefficent, because + // searching might be more work than a few extra doublings. + for i := 255; i >= 0; i-- { + tmp1.Double(tmp2) + + for j := range nafs { + if nafs[j][i] > 0 { + v.fromP1xP1(tmp1) + tables[j].SelectInto(multiple, nafs[j][i]) + tmp1.Add(v, multiple) + } else if nafs[j][i] < 0 { + v.fromP1xP1(tmp1) + tables[j].SelectInto(multiple, -nafs[j][i]) + tmp1.Sub(v, multiple) + } + } + + tmp2.FromP1xP1(tmp1) + } + + v.fromP2(tmp2) + return v +} + +// Select sets v to a if cond == 1 and to b if cond == 0. +func (v *Point) Select(a, b *Point, cond int) *Point { + checkInitialized(a, b) + v.x.Select(&a.x, &b.x, cond) + v.y.Select(&a.y, &b.y, cond) + v.z.Select(&a.z, &b.z, cond) + v.t.Select(&a.t, &b.t, cond) + return v +} + +// Double sets v = p + p, and returns v. +func (v *Point) Double(p *Point) *Point { + checkInitialized(p) + + pp := new(projP2).FromP3(p) + p1 := new(projP1xP1).Double(pp) + return v.fromP1xP1(p1) +} + +func (v *Point) addCached(p *Point, qCached *projCached) *Point { + result := new(projP1xP1).Add(p, qCached) + return v.fromP1xP1(result) +} + +// ScalarMultSlow sets v = x * q, and returns v. It doesn't precompute a large +// table, so it is considerably slower, but requires less memory. +// +// The scalar multiplication is done in constant time. +func (v *Point) ScalarMultSlow(x *Scalar, q *Point) *Point { + checkInitialized(q) + + s := x.Bytes() + qCached := new(projCached).FromP3(q) + v.Set(NewIdentityPoint()) + t := new(Point) + + for i := 255; i >= 0; i-- { + v.Double(v) + t.addCached(v, qCached) + cond := (s[i/8] >> (i % 8)) & 1 + v.Select(t, v, int(cond)) + } + + return v +} diff --git a/vendor/filippo.io/edwards25519/field/fe.go b/vendor/filippo.io/edwards25519/field/fe.go new file mode 100644 index 0000000000..4d52cc10d1 --- /dev/null +++ b/vendor/filippo.io/edwards25519/field/fe.go @@ -0,0 +1,420 @@ +// Copyright (c) 2017 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// Package field implements fast arithmetic modulo 2^255-19. +package field + +import ( + "crypto/subtle" + "encoding/binary" + "errors" + "math/bits" +) + +// Element represents an element of the field GF(2^255-19). Note that this +// is not a cryptographically secure group, and should only be used to interact +// with edwards25519.Point coordinates. +// +// This type works similarly to math/big.Int, and all arguments and receivers +// are allowed to alias. +// +// The zero value is a valid zero element. +type Element struct { + // An element t represents the integer + // t.l0 + t.l1*2^51 + t.l2*2^102 + t.l3*2^153 + t.l4*2^204 + // + // Between operations, all limbs are expected to be lower than 2^52. + l0 uint64 + l1 uint64 + l2 uint64 + l3 uint64 + l4 uint64 +} + +const maskLow51Bits uint64 = (1 << 51) - 1 + +var feZero = &Element{0, 0, 0, 0, 0} + +// Zero sets v = 0, and returns v. +func (v *Element) Zero() *Element { + *v = *feZero + return v +} + +var feOne = &Element{1, 0, 0, 0, 0} + +// One sets v = 1, and returns v. +func (v *Element) One() *Element { + *v = *feOne + return v +} + +// reduce reduces v modulo 2^255 - 19 and returns it. +func (v *Element) reduce() *Element { + v.carryPropagate() + + // After the light reduction we now have a field element representation + // v < 2^255 + 2^13 * 19, but need v < 2^255 - 19. + + // If v >= 2^255 - 19, then v + 19 >= 2^255, which would overflow 2^255 - 1, + // generating a carry. That is, c will be 0 if v < 2^255 - 19, and 1 otherwise. + c := (v.l0 + 19) >> 51 + c = (v.l1 + c) >> 51 + c = (v.l2 + c) >> 51 + c = (v.l3 + c) >> 51 + c = (v.l4 + c) >> 51 + + // If v < 2^255 - 19 and c = 0, this will be a no-op. Otherwise, it's + // effectively applying the reduction identity to the carry. + v.l0 += 19 * c + + v.l1 += v.l0 >> 51 + v.l0 = v.l0 & maskLow51Bits + v.l2 += v.l1 >> 51 + v.l1 = v.l1 & maskLow51Bits + v.l3 += v.l2 >> 51 + v.l2 = v.l2 & maskLow51Bits + v.l4 += v.l3 >> 51 + v.l3 = v.l3 & maskLow51Bits + // no additional carry + v.l4 = v.l4 & maskLow51Bits + + return v +} + +// Add sets v = a + b, and returns v. +func (v *Element) Add(a, b *Element) *Element { + v.l0 = a.l0 + b.l0 + v.l1 = a.l1 + b.l1 + v.l2 = a.l2 + b.l2 + v.l3 = a.l3 + b.l3 + v.l4 = a.l4 + b.l4 + return v.carryPropagate() +} + +// Subtract sets v = a - b, and returns v. +func (v *Element) Subtract(a, b *Element) *Element { + // We first add 2 * p, to guarantee the subtraction won't underflow, and + // then subtract b (which can be up to 2^255 + 2^13 * 19). + v.l0 = (a.l0 + 0xFFFFFFFFFFFDA) - b.l0 + v.l1 = (a.l1 + 0xFFFFFFFFFFFFE) - b.l1 + v.l2 = (a.l2 + 0xFFFFFFFFFFFFE) - b.l2 + v.l3 = (a.l3 + 0xFFFFFFFFFFFFE) - b.l3 + v.l4 = (a.l4 + 0xFFFFFFFFFFFFE) - b.l4 + return v.carryPropagate() +} + +// Negate sets v = -a, and returns v. +func (v *Element) Negate(a *Element) *Element { + return v.Subtract(feZero, a) +} + +// Invert sets v = 1/z mod p, and returns v. +// +// If z == 0, Invert returns v = 0. +func (v *Element) Invert(z *Element) *Element { + // Inversion is implemented as exponentiation with exponent p − 2. It uses the + // same sequence of 255 squarings and 11 multiplications as [Curve25519]. + var z2, z9, z11, z2_5_0, z2_10_0, z2_20_0, z2_50_0, z2_100_0, t Element + + z2.Square(z) // 2 + t.Square(&z2) // 4 + t.Square(&t) // 8 + z9.Multiply(&t, z) // 9 + z11.Multiply(&z9, &z2) // 11 + t.Square(&z11) // 22 + z2_5_0.Multiply(&t, &z9) // 31 = 2^5 - 2^0 + + t.Square(&z2_5_0) // 2^6 - 2^1 + for i := 0; i < 4; i++ { + t.Square(&t) // 2^10 - 2^5 + } + z2_10_0.Multiply(&t, &z2_5_0) // 2^10 - 2^0 + + t.Square(&z2_10_0) // 2^11 - 2^1 + for i := 0; i < 9; i++ { + t.Square(&t) // 2^20 - 2^10 + } + z2_20_0.Multiply(&t, &z2_10_0) // 2^20 - 2^0 + + t.Square(&z2_20_0) // 2^21 - 2^1 + for i := 0; i < 19; i++ { + t.Square(&t) // 2^40 - 2^20 + } + t.Multiply(&t, &z2_20_0) // 2^40 - 2^0 + + t.Square(&t) // 2^41 - 2^1 + for i := 0; i < 9; i++ { + t.Square(&t) // 2^50 - 2^10 + } + z2_50_0.Multiply(&t, &z2_10_0) // 2^50 - 2^0 + + t.Square(&z2_50_0) // 2^51 - 2^1 + for i := 0; i < 49; i++ { + t.Square(&t) // 2^100 - 2^50 + } + z2_100_0.Multiply(&t, &z2_50_0) // 2^100 - 2^0 + + t.Square(&z2_100_0) // 2^101 - 2^1 + for i := 0; i < 99; i++ { + t.Square(&t) // 2^200 - 2^100 + } + t.Multiply(&t, &z2_100_0) // 2^200 - 2^0 + + t.Square(&t) // 2^201 - 2^1 + for i := 0; i < 49; i++ { + t.Square(&t) // 2^250 - 2^50 + } + t.Multiply(&t, &z2_50_0) // 2^250 - 2^0 + + t.Square(&t) // 2^251 - 2^1 + t.Square(&t) // 2^252 - 2^2 + t.Square(&t) // 2^253 - 2^3 + t.Square(&t) // 2^254 - 2^4 + t.Square(&t) // 2^255 - 2^5 + + return v.Multiply(&t, &z11) // 2^255 - 21 +} + +// Set sets v = a, and returns v. +func (v *Element) Set(a *Element) *Element { + *v = *a + return v +} + +// SetBytes sets v to x, where x is a 32-byte little-endian encoding. If x is +// not of the right length, SetBytes returns nil and an error, and the +// receiver is unchanged. +// +// Consistent with RFC 7748, the most significant bit (the high bit of the +// last byte) is ignored, and non-canonical values (2^255-19 through 2^255-1) +// are accepted. Note that this is laxer than specified by RFC 8032, but +// consistent with most Ed25519 implementations. +func (v *Element) SetBytes(x []byte) (*Element, error) { + if len(x) != 32 { + return nil, errors.New("edwards25519: invalid field element input size") + } + + // Bits 0:51 (bytes 0:8, bits 0:64, shift 0, mask 51). + v.l0 = binary.LittleEndian.Uint64(x[0:8]) + v.l0 &= maskLow51Bits + // Bits 51:102 (bytes 6:14, bits 48:112, shift 3, mask 51). + v.l1 = binary.LittleEndian.Uint64(x[6:14]) >> 3 + v.l1 &= maskLow51Bits + // Bits 102:153 (bytes 12:20, bits 96:160, shift 6, mask 51). + v.l2 = binary.LittleEndian.Uint64(x[12:20]) >> 6 + v.l2 &= maskLow51Bits + // Bits 153:204 (bytes 19:27, bits 152:216, shift 1, mask 51). + v.l3 = binary.LittleEndian.Uint64(x[19:27]) >> 1 + v.l3 &= maskLow51Bits + // Bits 204:255 (bytes 24:32, bits 192:256, shift 12, mask 51). + // Note: not bytes 25:33, shift 4, to avoid overread. + v.l4 = binary.LittleEndian.Uint64(x[24:32]) >> 12 + v.l4 &= maskLow51Bits + + return v, nil +} + +// Bytes returns the canonical 32-byte little-endian encoding of v. +func (v *Element) Bytes() []byte { + // This function is outlined to make the allocations inline in the caller + // rather than happen on the heap. + var out [32]byte + return v.bytes(&out) +} + +func (v *Element) bytes(out *[32]byte) []byte { + t := *v + t.reduce() + + // Pack five 51-bit limbs into four 64-bit words: + // + // 255 204 153 102 51 0 + // ├──l4──┼──l3──┼──l2──┼──l1──┼──l0──┤ + // ├───u3───┼───u2───┼───u1───┼───u0───┤ + // 256 192 128 64 0 + + u0 := t.l1<<51 | t.l0 + u1 := t.l2<<(102-64) | t.l1>>(64-51) + u2 := t.l3<<(153-128) | t.l2>>(128-102) + u3 := t.l4<<(204-192) | t.l3>>(192-153) + + binary.LittleEndian.PutUint64(out[0*8:], u0) + binary.LittleEndian.PutUint64(out[1*8:], u1) + binary.LittleEndian.PutUint64(out[2*8:], u2) + binary.LittleEndian.PutUint64(out[3*8:], u3) + + return out[:] +} + +// Equal returns 1 if v and u are equal, and 0 otherwise. +func (v *Element) Equal(u *Element) int { + sa, sv := u.Bytes(), v.Bytes() + return subtle.ConstantTimeCompare(sa, sv) +} + +// mask64Bits returns 0xffffffff if cond is 1, and 0 otherwise. +func mask64Bits(cond int) uint64 { return ^(uint64(cond) - 1) } + +// Select sets v to a if cond == 1, and to b if cond == 0. +func (v *Element) Select(a, b *Element, cond int) *Element { + m := mask64Bits(cond) + v.l0 = (m & a.l0) | (^m & b.l0) + v.l1 = (m & a.l1) | (^m & b.l1) + v.l2 = (m & a.l2) | (^m & b.l2) + v.l3 = (m & a.l3) | (^m & b.l3) + v.l4 = (m & a.l4) | (^m & b.l4) + return v +} + +// Swap swaps v and u if cond == 1 or leaves them unchanged if cond == 0, and returns v. +func (v *Element) Swap(u *Element, cond int) { + m := mask64Bits(cond) + t := m & (v.l0 ^ u.l0) + v.l0 ^= t + u.l0 ^= t + t = m & (v.l1 ^ u.l1) + v.l1 ^= t + u.l1 ^= t + t = m & (v.l2 ^ u.l2) + v.l2 ^= t + u.l2 ^= t + t = m & (v.l3 ^ u.l3) + v.l3 ^= t + u.l3 ^= t + t = m & (v.l4 ^ u.l4) + v.l4 ^= t + u.l4 ^= t +} + +// IsNegative returns 1 if v is negative, and 0 otherwise. +func (v *Element) IsNegative() int { + return int(v.Bytes()[0] & 1) +} + +// Absolute sets v to |u|, and returns v. +func (v *Element) Absolute(u *Element) *Element { + return v.Select(new(Element).Negate(u), u, u.IsNegative()) +} + +// Multiply sets v = x * y, and returns v. +func (v *Element) Multiply(x, y *Element) *Element { + feMul(v, x, y) + return v +} + +// Square sets v = x * x, and returns v. +func (v *Element) Square(x *Element) *Element { + feSquare(v, x) + return v +} + +// Mult32 sets v = x * y, and returns v. +func (v *Element) Mult32(x *Element, y uint32) *Element { + x0lo, x0hi := mul51(x.l0, y) + x1lo, x1hi := mul51(x.l1, y) + x2lo, x2hi := mul51(x.l2, y) + x3lo, x3hi := mul51(x.l3, y) + x4lo, x4hi := mul51(x.l4, y) + v.l0 = x0lo + 19*x4hi // carried over per the reduction identity + v.l1 = x1lo + x0hi + v.l2 = x2lo + x1hi + v.l3 = x3lo + x2hi + v.l4 = x4lo + x3hi + // The hi portions are going to be only 32 bits, plus any previous excess, + // so we can skip the carry propagation. + return v +} + +// mul51 returns lo + hi * 2⁵¹ = a * b. +func mul51(a uint64, b uint32) (lo uint64, hi uint64) { + mh, ml := bits.Mul64(a, uint64(b)) + lo = ml & maskLow51Bits + hi = (mh << 13) | (ml >> 51) + return +} + +// Pow22523 set v = x^((p-5)/8), and returns v. (p-5)/8 is 2^252-3. +func (v *Element) Pow22523(x *Element) *Element { + var t0, t1, t2 Element + + t0.Square(x) // x^2 + t1.Square(&t0) // x^4 + t1.Square(&t1) // x^8 + t1.Multiply(x, &t1) // x^9 + t0.Multiply(&t0, &t1) // x^11 + t0.Square(&t0) // x^22 + t0.Multiply(&t1, &t0) // x^31 + t1.Square(&t0) // x^62 + for i := 1; i < 5; i++ { // x^992 + t1.Square(&t1) + } + t0.Multiply(&t1, &t0) // x^1023 -> 1023 = 2^10 - 1 + t1.Square(&t0) // 2^11 - 2 + for i := 1; i < 10; i++ { // 2^20 - 2^10 + t1.Square(&t1) + } + t1.Multiply(&t1, &t0) // 2^20 - 1 + t2.Square(&t1) // 2^21 - 2 + for i := 1; i < 20; i++ { // 2^40 - 2^20 + t2.Square(&t2) + } + t1.Multiply(&t2, &t1) // 2^40 - 1 + t1.Square(&t1) // 2^41 - 2 + for i := 1; i < 10; i++ { // 2^50 - 2^10 + t1.Square(&t1) + } + t0.Multiply(&t1, &t0) // 2^50 - 1 + t1.Square(&t0) // 2^51 - 2 + for i := 1; i < 50; i++ { // 2^100 - 2^50 + t1.Square(&t1) + } + t1.Multiply(&t1, &t0) // 2^100 - 1 + t2.Square(&t1) // 2^101 - 2 + for i := 1; i < 100; i++ { // 2^200 - 2^100 + t2.Square(&t2) + } + t1.Multiply(&t2, &t1) // 2^200 - 1 + t1.Square(&t1) // 2^201 - 2 + for i := 1; i < 50; i++ { // 2^250 - 2^50 + t1.Square(&t1) + } + t0.Multiply(&t1, &t0) // 2^250 - 1 + t0.Square(&t0) // 2^251 - 2 + t0.Square(&t0) // 2^252 - 4 + return v.Multiply(&t0, x) // 2^252 - 3 -> x^(2^252-3) +} + +// sqrtM1 is 2^((p-1)/4), which squared is equal to -1 by Euler's Criterion. +var sqrtM1 = &Element{1718705420411056, 234908883556509, + 2233514472574048, 2117202627021982, 765476049583133} + +// SqrtRatio sets r to the non-negative square root of the ratio of u and v. +// +// If u/v is square, SqrtRatio returns r and 1. If u/v is not square, SqrtRatio +// sets r according to Section 4.3 of draft-irtf-cfrg-ristretto255-decaf448-00, +// and returns r and 0. +func (r *Element) SqrtRatio(u, v *Element) (R *Element, wasSquare int) { + t0 := new(Element) + + // r = (u * v3) * (u * v7)^((p-5)/8) + v2 := new(Element).Square(v) + uv3 := new(Element).Multiply(u, t0.Multiply(v2, v)) + uv7 := new(Element).Multiply(uv3, t0.Square(v2)) + rr := new(Element).Multiply(uv3, t0.Pow22523(uv7)) + + check := new(Element).Multiply(v, t0.Square(rr)) // check = v * r^2 + + uNeg := new(Element).Negate(u) + correctSignSqrt := check.Equal(u) + flippedSignSqrt := check.Equal(uNeg) + flippedSignSqrtI := check.Equal(t0.Multiply(uNeg, sqrtM1)) + + rPrime := new(Element).Multiply(rr, sqrtM1) // r_prime = SQRT_M1 * r + // r = CT_SELECT(r_prime IF flipped_sign_sqrt | flipped_sign_sqrt_i ELSE r) + rr.Select(rPrime, rr, flippedSignSqrt|flippedSignSqrtI) + + r.Absolute(rr) // Choose the nonnegative square root. + return r, correctSignSqrt | flippedSignSqrt +} diff --git a/vendor/filippo.io/edwards25519/field/fe_amd64.go b/vendor/filippo.io/edwards25519/field/fe_amd64.go new file mode 100644 index 0000000000..00bf8f4479 --- /dev/null +++ b/vendor/filippo.io/edwards25519/field/fe_amd64.go @@ -0,0 +1,15 @@ +// Code generated by command: go run fe_amd64_asm.go -out ../fe_amd64.s -stubs ../fe_amd64.go -pkg field. DO NOT EDIT. + +//go:build !purego + +package field + +// feMul sets out = a * b. It works like feMulGeneric. +// +//go:noescape +func feMul(out *Element, a *Element, b *Element) + +// feSquare sets out = a * a. It works like feSquareGeneric. +// +//go:noescape +func feSquare(out *Element, a *Element) diff --git a/vendor/filippo.io/edwards25519/field/fe_amd64.s b/vendor/filippo.io/edwards25519/field/fe_amd64.s new file mode 100644 index 0000000000..5e06e242ed --- /dev/null +++ b/vendor/filippo.io/edwards25519/field/fe_amd64.s @@ -0,0 +1,398 @@ +// Code generated by command: go run fe_amd64_asm.go -out ../fe_amd64.s -stubs ../fe_amd64.go -pkg field. DO NOT EDIT. + +//go:build !purego + +#include "textflag.h" + +// func feMul(out *Element, a *Element, b *Element) +TEXT ·feMul(SB), NOSPLIT, $0-24 + MOVQ a+8(FP), CX + MOVQ b+16(FP), BX + + // r0 = a0×b0 + MOVQ (CX), AX + MULQ (BX) + MOVQ AX, DI + MOVQ DX, SI + + // r0 += 19×a1×b4 + MOVQ 8(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 32(BX) + ADDQ AX, DI + ADCQ DX, SI + + // r0 += 19×a2×b3 + MOVQ 16(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 24(BX) + ADDQ AX, DI + ADCQ DX, SI + + // r0 += 19×a3×b2 + MOVQ 24(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 16(BX) + ADDQ AX, DI + ADCQ DX, SI + + // r0 += 19×a4×b1 + MOVQ 32(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 8(BX) + ADDQ AX, DI + ADCQ DX, SI + + // r1 = a0×b1 + MOVQ (CX), AX + MULQ 8(BX) + MOVQ AX, R9 + MOVQ DX, R8 + + // r1 += a1×b0 + MOVQ 8(CX), AX + MULQ (BX) + ADDQ AX, R9 + ADCQ DX, R8 + + // r1 += 19×a2×b4 + MOVQ 16(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 32(BX) + ADDQ AX, R9 + ADCQ DX, R8 + + // r1 += 19×a3×b3 + MOVQ 24(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 24(BX) + ADDQ AX, R9 + ADCQ DX, R8 + + // r1 += 19×a4×b2 + MOVQ 32(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 16(BX) + ADDQ AX, R9 + ADCQ DX, R8 + + // r2 = a0×b2 + MOVQ (CX), AX + MULQ 16(BX) + MOVQ AX, R11 + MOVQ DX, R10 + + // r2 += a1×b1 + MOVQ 8(CX), AX + MULQ 8(BX) + ADDQ AX, R11 + ADCQ DX, R10 + + // r2 += a2×b0 + MOVQ 16(CX), AX + MULQ (BX) + ADDQ AX, R11 + ADCQ DX, R10 + + // r2 += 19×a3×b4 + MOVQ 24(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 32(BX) + ADDQ AX, R11 + ADCQ DX, R10 + + // r2 += 19×a4×b3 + MOVQ 32(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 24(BX) + ADDQ AX, R11 + ADCQ DX, R10 + + // r3 = a0×b3 + MOVQ (CX), AX + MULQ 24(BX) + MOVQ AX, R13 + MOVQ DX, R12 + + // r3 += a1×b2 + MOVQ 8(CX), AX + MULQ 16(BX) + ADDQ AX, R13 + ADCQ DX, R12 + + // r3 += a2×b1 + MOVQ 16(CX), AX + MULQ 8(BX) + ADDQ AX, R13 + ADCQ DX, R12 + + // r3 += a3×b0 + MOVQ 24(CX), AX + MULQ (BX) + ADDQ AX, R13 + ADCQ DX, R12 + + // r3 += 19×a4×b4 + MOVQ 32(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 32(BX) + ADDQ AX, R13 + ADCQ DX, R12 + + // r4 = a0×b4 + MOVQ (CX), AX + MULQ 32(BX) + MOVQ AX, R15 + MOVQ DX, R14 + + // r4 += a1×b3 + MOVQ 8(CX), AX + MULQ 24(BX) + ADDQ AX, R15 + ADCQ DX, R14 + + // r4 += a2×b2 + MOVQ 16(CX), AX + MULQ 16(BX) + ADDQ AX, R15 + ADCQ DX, R14 + + // r4 += a3×b1 + MOVQ 24(CX), AX + MULQ 8(BX) + ADDQ AX, R15 + ADCQ DX, R14 + + // r4 += a4×b0 + MOVQ 32(CX), AX + MULQ (BX) + ADDQ AX, R15 + ADCQ DX, R14 + + // First reduction chain + MOVQ $0x0007ffffffffffff, AX + SHLQ $0x0d, DI, SI + SHLQ $0x0d, R9, R8 + SHLQ $0x0d, R11, R10 + SHLQ $0x0d, R13, R12 + SHLQ $0x0d, R15, R14 + ANDQ AX, DI + IMUL3Q $0x13, R14, R14 + ADDQ R14, DI + ANDQ AX, R9 + ADDQ SI, R9 + ANDQ AX, R11 + ADDQ R8, R11 + ANDQ AX, R13 + ADDQ R10, R13 + ANDQ AX, R15 + ADDQ R12, R15 + + // Second reduction chain (carryPropagate) + MOVQ DI, SI + SHRQ $0x33, SI + MOVQ R9, R8 + SHRQ $0x33, R8 + MOVQ R11, R10 + SHRQ $0x33, R10 + MOVQ R13, R12 + SHRQ $0x33, R12 + MOVQ R15, R14 + SHRQ $0x33, R14 + ANDQ AX, DI + IMUL3Q $0x13, R14, R14 + ADDQ R14, DI + ANDQ AX, R9 + ADDQ SI, R9 + ANDQ AX, R11 + ADDQ R8, R11 + ANDQ AX, R13 + ADDQ R10, R13 + ANDQ AX, R15 + ADDQ R12, R15 + + // Store output + MOVQ out+0(FP), AX + MOVQ DI, (AX) + MOVQ R9, 8(AX) + MOVQ R11, 16(AX) + MOVQ R13, 24(AX) + MOVQ R15, 32(AX) + RET + +// func feSquare(out *Element, a *Element) +TEXT ·feSquare(SB), NOSPLIT, $0-16 + MOVQ a+8(FP), CX + + // r0 = l0×l0 + MOVQ (CX), AX + MULQ (CX) + MOVQ AX, SI + MOVQ DX, BX + + // r0 += 38×l1×l4 + MOVQ 8(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + SHLQ $0x01, AX + MULQ 32(CX) + ADDQ AX, SI + ADCQ DX, BX + + // r0 += 38×l2×l3 + MOVQ 16(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + SHLQ $0x01, AX + MULQ 24(CX) + ADDQ AX, SI + ADCQ DX, BX + + // r1 = 2×l0×l1 + MOVQ (CX), AX + SHLQ $0x01, AX + MULQ 8(CX) + MOVQ AX, R8 + MOVQ DX, DI + + // r1 += 38×l2×l4 + MOVQ 16(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + SHLQ $0x01, AX + MULQ 32(CX) + ADDQ AX, R8 + ADCQ DX, DI + + // r1 += 19×l3×l3 + MOVQ 24(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 24(CX) + ADDQ AX, R8 + ADCQ DX, DI + + // r2 = 2×l0×l2 + MOVQ (CX), AX + SHLQ $0x01, AX + MULQ 16(CX) + MOVQ AX, R10 + MOVQ DX, R9 + + // r2 += l1×l1 + MOVQ 8(CX), AX + MULQ 8(CX) + ADDQ AX, R10 + ADCQ DX, R9 + + // r2 += 38×l3×l4 + MOVQ 24(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + SHLQ $0x01, AX + MULQ 32(CX) + ADDQ AX, R10 + ADCQ DX, R9 + + // r3 = 2×l0×l3 + MOVQ (CX), AX + SHLQ $0x01, AX + MULQ 24(CX) + MOVQ AX, R12 + MOVQ DX, R11 + + // r3 += 2×l1×l2 + MOVQ 8(CX), AX + SHLQ $0x01, AX + MULQ 16(CX) + ADDQ AX, R12 + ADCQ DX, R11 + + // r3 += 19×l4×l4 + MOVQ 32(CX), DX + LEAQ (DX)(DX*8), AX + LEAQ (DX)(AX*2), AX + MULQ 32(CX) + ADDQ AX, R12 + ADCQ DX, R11 + + // r4 = 2×l0×l4 + MOVQ (CX), AX + SHLQ $0x01, AX + MULQ 32(CX) + MOVQ AX, R14 + MOVQ DX, R13 + + // r4 += 2×l1×l3 + MOVQ 8(CX), AX + SHLQ $0x01, AX + MULQ 24(CX) + ADDQ AX, R14 + ADCQ DX, R13 + + // r4 += l2×l2 + MOVQ 16(CX), AX + MULQ 16(CX) + ADDQ AX, R14 + ADCQ DX, R13 + + // First reduction chain + MOVQ $0x0007ffffffffffff, AX + SHLQ $0x0d, SI, BX + SHLQ $0x0d, R8, DI + SHLQ $0x0d, R10, R9 + SHLQ $0x0d, R12, R11 + SHLQ $0x0d, R14, R13 + ANDQ AX, SI + IMUL3Q $0x13, R13, R13 + ADDQ R13, SI + ANDQ AX, R8 + ADDQ BX, R8 + ANDQ AX, R10 + ADDQ DI, R10 + ANDQ AX, R12 + ADDQ R9, R12 + ANDQ AX, R14 + ADDQ R11, R14 + + // Second reduction chain (carryPropagate) + MOVQ SI, BX + SHRQ $0x33, BX + MOVQ R8, DI + SHRQ $0x33, DI + MOVQ R10, R9 + SHRQ $0x33, R9 + MOVQ R12, R11 + SHRQ $0x33, R11 + MOVQ R14, R13 + SHRQ $0x33, R13 + ANDQ AX, SI + IMUL3Q $0x13, R13, R13 + ADDQ R13, SI + ANDQ AX, R8 + ADDQ BX, R8 + ANDQ AX, R10 + ADDQ DI, R10 + ANDQ AX, R12 + ADDQ R9, R12 + ANDQ AX, R14 + ADDQ R11, R14 + + // Store output + MOVQ out+0(FP), AX + MOVQ SI, (AX) + MOVQ R8, 8(AX) + MOVQ R10, 16(AX) + MOVQ R12, 24(AX) + MOVQ R14, 32(AX) + RET diff --git a/vendor/filippo.io/edwards25519/field/fe_amd64_noasm.go b/vendor/filippo.io/edwards25519/field/fe_amd64_noasm.go new file mode 100644 index 0000000000..4b81f25d1d --- /dev/null +++ b/vendor/filippo.io/edwards25519/field/fe_amd64_noasm.go @@ -0,0 +1,11 @@ +// Copyright (c) 2019 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +//go:build !amd64 || purego + +package field + +func feMul(v, x, y *Element) { feMulGeneric(v, x, y) } + +func feSquare(v, x *Element) { feSquareGeneric(v, x) } diff --git a/vendor/filippo.io/edwards25519/field/fe_extra.go b/vendor/filippo.io/edwards25519/field/fe_extra.go new file mode 100644 index 0000000000..1ef503b9a2 --- /dev/null +++ b/vendor/filippo.io/edwards25519/field/fe_extra.go @@ -0,0 +1,50 @@ +// Copyright (c) 2021 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package field + +import "errors" + +// This file contains additional functionality that is not included in the +// upstream crypto/ed25519/edwards25519/field package. + +// SetWideBytes sets v to x, where x is a 64-byte little-endian encoding, which +// is reduced modulo the field order. If x is not of the right length, +// SetWideBytes returns nil and an error, and the receiver is unchanged. +// +// SetWideBytes is not necessary to select a uniformly distributed value, and is +// only provided for compatibility: SetBytes can be used instead as the chance +// of bias is less than 2⁻²⁵⁰. +func (v *Element) SetWideBytes(x []byte) (*Element, error) { + if len(x) != 64 { + return nil, errors.New("edwards25519: invalid SetWideBytes input size") + } + + // Split the 64 bytes into two elements, and extract the most significant + // bit of each, which is ignored by SetBytes. + lo, _ := new(Element).SetBytes(x[:32]) + loMSB := uint64(x[31] >> 7) + hi, _ := new(Element).SetBytes(x[32:]) + hiMSB := uint64(x[63] >> 7) + + // The output we want is + // + // v = lo + loMSB * 2²⁵⁵ + hi * 2²⁵⁶ + hiMSB * 2⁵¹¹ + // + // which applying the reduction identity comes out to + // + // v = lo + loMSB * 19 + hi * 2 * 19 + hiMSB * 2 * 19² + // + // l0 will be the sum of a 52 bits value (lo.l0), plus a 5 bits value + // (loMSB * 19), a 6 bits value (hi.l0 * 2 * 19), and a 10 bits value + // (hiMSB * 2 * 19²), so it fits in a uint64. + + v.l0 = lo.l0 + loMSB*19 + hi.l0*2*19 + hiMSB*2*19*19 + v.l1 = lo.l1 + hi.l1*2*19 + v.l2 = lo.l2 + hi.l2*2*19 + v.l3 = lo.l3 + hi.l3*2*19 + v.l4 = lo.l4 + hi.l4*2*19 + + return v.carryPropagate(), nil +} diff --git a/vendor/filippo.io/edwards25519/field/fe_generic.go b/vendor/filippo.io/edwards25519/field/fe_generic.go new file mode 100644 index 0000000000..ef1f15a5dc --- /dev/null +++ b/vendor/filippo.io/edwards25519/field/fe_generic.go @@ -0,0 +1,272 @@ +// Copyright (c) 2017 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package field + +import "math/bits" + +// uint128 holds a 128-bit number as two 64-bit limbs, for use with the +// bits.Mul64 and bits.Add64 intrinsics. +type uint128 struct { + lo, hi uint64 +} + +// mul returns a * b. +func mul(a, b uint64) uint128 { + hi, lo := bits.Mul64(a, b) + return uint128{lo, hi} +} + +// addMul returns v + a * b. +func addMul(v uint128, a, b uint64) uint128 { + hi, lo := bits.Mul64(a, b) + lo, c := bits.Add64(lo, v.lo, 0) + hi, _ = bits.Add64(hi, v.hi, c) + return uint128{lo, hi} +} + +// mul19 returns v * 19. +func mul19(v uint64) uint64 { + // Using this approach seems to yield better optimizations than *19. + return v + (v+v<<3)<<1 +} + +// addMul19 returns v + 19 * a * b, where a and b are at most 52 bits. +func addMul19(v uint128, a, b uint64) uint128 { + hi, lo := bits.Mul64(mul19(a), b) + lo, c := bits.Add64(lo, v.lo, 0) + hi, _ = bits.Add64(hi, v.hi, c) + return uint128{lo, hi} +} + +// addMul38 returns v + 38 * a * b, where a and b are at most 52 bits. +func addMul38(v uint128, a, b uint64) uint128 { + hi, lo := bits.Mul64(mul19(a), b*2) + lo, c := bits.Add64(lo, v.lo, 0) + hi, _ = bits.Add64(hi, v.hi, c) + return uint128{lo, hi} +} + +// shiftRightBy51 returns a >> 51. a is assumed to be at most 115 bits. +func shiftRightBy51(a uint128) uint64 { + return (a.hi << (64 - 51)) | (a.lo >> 51) +} + +func feMulGeneric(v, a, b *Element) { + a0 := a.l0 + a1 := a.l1 + a2 := a.l2 + a3 := a.l3 + a4 := a.l4 + + b0 := b.l0 + b1 := b.l1 + b2 := b.l2 + b3 := b.l3 + b4 := b.l4 + + // Limb multiplication works like pen-and-paper columnar multiplication, but + // with 51-bit limbs instead of digits. + // + // a4 a3 a2 a1 a0 x + // b4 b3 b2 b1 b0 = + // ------------------------ + // a4b0 a3b0 a2b0 a1b0 a0b0 + + // a4b1 a3b1 a2b1 a1b1 a0b1 + + // a4b2 a3b2 a2b2 a1b2 a0b2 + + // a4b3 a3b3 a2b3 a1b3 a0b3 + + // a4b4 a3b4 a2b4 a1b4 a0b4 = + // ---------------------------------------------- + // r8 r7 r6 r5 r4 r3 r2 r1 r0 + // + // We can then use the reduction identity (a * 2²⁵⁵ + b = a * 19 + b) to + // reduce the limbs that would overflow 255 bits. r5 * 2²⁵⁵ becomes 19 * r5, + // r6 * 2³⁰⁶ becomes 19 * r6 * 2⁵¹, etc. + // + // Reduction can be carried out simultaneously to multiplication. For + // example, we do not compute r5: whenever the result of a multiplication + // belongs to r5, like a1b4, we multiply it by 19 and add the result to r0. + // + // a4b0 a3b0 a2b0 a1b0 a0b0 + + // a3b1 a2b1 a1b1 a0b1 19×a4b1 + + // a2b2 a1b2 a0b2 19×a4b2 19×a3b2 + + // a1b3 a0b3 19×a4b3 19×a3b3 19×a2b3 + + // a0b4 19×a4b4 19×a3b4 19×a2b4 19×a1b4 = + // -------------------------------------- + // r4 r3 r2 r1 r0 + // + // Finally we add up the columns into wide, overlapping limbs. + + // r0 = a0×b0 + 19×(a1×b4 + a2×b3 + a3×b2 + a4×b1) + r0 := mul(a0, b0) + r0 = addMul19(r0, a1, b4) + r0 = addMul19(r0, a2, b3) + r0 = addMul19(r0, a3, b2) + r0 = addMul19(r0, a4, b1) + + // r1 = a0×b1 + a1×b0 + 19×(a2×b4 + a3×b3 + a4×b2) + r1 := mul(a0, b1) + r1 = addMul(r1, a1, b0) + r1 = addMul19(r1, a2, b4) + r1 = addMul19(r1, a3, b3) + r1 = addMul19(r1, a4, b2) + + // r2 = a0×b2 + a1×b1 + a2×b0 + 19×(a3×b4 + a4×b3) + r2 := mul(a0, b2) + r2 = addMul(r2, a1, b1) + r2 = addMul(r2, a2, b0) + r2 = addMul19(r2, a3, b4) + r2 = addMul19(r2, a4, b3) + + // r3 = a0×b3 + a1×b2 + a2×b1 + a3×b0 + 19×a4×b4 + r3 := mul(a0, b3) + r3 = addMul(r3, a1, b2) + r3 = addMul(r3, a2, b1) + r3 = addMul(r3, a3, b0) + r3 = addMul19(r3, a4, b4) + + // r4 = a0×b4 + a1×b3 + a2×b2 + a3×b1 + a4×b0 + r4 := mul(a0, b4) + r4 = addMul(r4, a1, b3) + r4 = addMul(r4, a2, b2) + r4 = addMul(r4, a3, b1) + r4 = addMul(r4, a4, b0) + + // After the multiplication, we need to reduce (carry) the five coefficients + // to obtain a result with limbs that are at most slightly larger than 2⁵¹, + // to respect the Element invariant. + // + // Overall, the reduction works the same as carryPropagate, except with + // wider inputs: we take the carry for each coefficient by shifting it right + // by 51, and add it to the limb above it. The top carry is multiplied by 19 + // according to the reduction identity and added to the lowest limb. + // + // The largest coefficient (r0) will be at most 111 bits, which guarantees + // that all carries are at most 111 - 51 = 60 bits, which fits in a uint64. + // + // r0 = a0×b0 + 19×(a1×b4 + a2×b3 + a3×b2 + a4×b1) + // r0 < 2⁵²×2⁵² + 19×(2⁵²×2⁵² + 2⁵²×2⁵² + 2⁵²×2⁵² + 2⁵²×2⁵²) + // r0 < (1 + 19 × 4) × 2⁵² × 2⁵² + // r0 < 2⁷ × 2⁵² × 2⁵² + // r0 < 2¹¹¹ + // + // Moreover, the top coefficient (r4) is at most 107 bits, so c4 is at most + // 56 bits, and c4 * 19 is at most 61 bits, which again fits in a uint64 and + // allows us to easily apply the reduction identity. + // + // r4 = a0×b4 + a1×b3 + a2×b2 + a3×b1 + a4×b0 + // r4 < 5 × 2⁵² × 2⁵² + // r4 < 2¹⁰⁷ + // + + c0 := shiftRightBy51(r0) + c1 := shiftRightBy51(r1) + c2 := shiftRightBy51(r2) + c3 := shiftRightBy51(r3) + c4 := shiftRightBy51(r4) + + rr0 := r0.lo&maskLow51Bits + mul19(c4) + rr1 := r1.lo&maskLow51Bits + c0 + rr2 := r2.lo&maskLow51Bits + c1 + rr3 := r3.lo&maskLow51Bits + c2 + rr4 := r4.lo&maskLow51Bits + c3 + + // Now all coefficients fit into 64-bit registers but are still too large to + // be passed around as an Element. We therefore do one last carry chain, + // where the carries will be small enough to fit in the wiggle room above 2⁵¹. + + v.l0 = rr0&maskLow51Bits + mul19(rr4>>51) + v.l1 = rr1&maskLow51Bits + rr0>>51 + v.l2 = rr2&maskLow51Bits + rr1>>51 + v.l3 = rr3&maskLow51Bits + rr2>>51 + v.l4 = rr4&maskLow51Bits + rr3>>51 +} + +func feSquareGeneric(v, a *Element) { + l0 := a.l0 + l1 := a.l1 + l2 := a.l2 + l3 := a.l3 + l4 := a.l4 + + // Squaring works precisely like multiplication above, but thanks to its + // symmetry we get to group a few terms together. + // + // l4 l3 l2 l1 l0 x + // l4 l3 l2 l1 l0 = + // ------------------------ + // l4l0 l3l0 l2l0 l1l0 l0l0 + + // l4l1 l3l1 l2l1 l1l1 l0l1 + + // l4l2 l3l2 l2l2 l1l2 l0l2 + + // l4l3 l3l3 l2l3 l1l3 l0l3 + + // l4l4 l3l4 l2l4 l1l4 l0l4 = + // ---------------------------------------------- + // r8 r7 r6 r5 r4 r3 r2 r1 r0 + // + // l4l0 l3l0 l2l0 l1l0 l0l0 + + // l3l1 l2l1 l1l1 l0l1 19×l4l1 + + // l2l2 l1l2 l0l2 19×l4l2 19×l3l2 + + // l1l3 l0l3 19×l4l3 19×l3l3 19×l2l3 + + // l0l4 19×l4l4 19×l3l4 19×l2l4 19×l1l4 = + // -------------------------------------- + // r4 r3 r2 r1 r0 + + // r0 = l0×l0 + 19×(l1×l4 + l2×l3 + l3×l2 + l4×l1) = l0×l0 + 19×2×(l1×l4 + l2×l3) + r0 := mul(l0, l0) + r0 = addMul38(r0, l1, l4) + r0 = addMul38(r0, l2, l3) + + // r1 = l0×l1 + l1×l0 + 19×(l2×l4 + l3×l3 + l4×l2) = 2×l0×l1 + 19×2×l2×l4 + 19×l3×l3 + r1 := mul(l0*2, l1) + r1 = addMul38(r1, l2, l4) + r1 = addMul19(r1, l3, l3) + + // r2 = l0×l2 + l1×l1 + l2×l0 + 19×(l3×l4 + l4×l3) = 2×l0×l2 + l1×l1 + 19×2×l3×l4 + r2 := mul(l0*2, l2) + r2 = addMul(r2, l1, l1) + r2 = addMul38(r2, l3, l4) + + // r3 = l0×l3 + l1×l2 + l2×l1 + l3×l0 + 19×l4×l4 = 2×l0×l3 + 2×l1×l2 + 19×l4×l4 + r3 := mul(l0*2, l3) + r3 = addMul(r3, l1*2, l2) + r3 = addMul19(r3, l4, l4) + + // r4 = l0×l4 + l1×l3 + l2×l2 + l3×l1 + l4×l0 = 2×l0×l4 + 2×l1×l3 + l2×l2 + r4 := mul(l0*2, l4) + r4 = addMul(r4, l1*2, l3) + r4 = addMul(r4, l2, l2) + + c0 := shiftRightBy51(r0) + c1 := shiftRightBy51(r1) + c2 := shiftRightBy51(r2) + c3 := shiftRightBy51(r3) + c4 := shiftRightBy51(r4) + + rr0 := r0.lo&maskLow51Bits + mul19(c4) + rr1 := r1.lo&maskLow51Bits + c0 + rr2 := r2.lo&maskLow51Bits + c1 + rr3 := r3.lo&maskLow51Bits + c2 + rr4 := r4.lo&maskLow51Bits + c3 + + v.l0 = rr0&maskLow51Bits + mul19(rr4>>51) + v.l1 = rr1&maskLow51Bits + rr0>>51 + v.l2 = rr2&maskLow51Bits + rr1>>51 + v.l3 = rr3&maskLow51Bits + rr2>>51 + v.l4 = rr4&maskLow51Bits + rr3>>51 +} + +// carryPropagate brings the limbs below 52 bits by applying the reduction +// identity (a * 2²⁵⁵ + b = a * 19 + b) to the l4 carry. +func (v *Element) carryPropagate() *Element { + // (l4>>51) is at most 64 - 51 = 13 bits, so (l4>>51)*19 is at most 18 bits, and + // the final l0 will be at most 52 bits. Similarly for the rest. + l0 := v.l0 + v.l0 = v.l0&maskLow51Bits + mul19(v.l4>>51) + v.l4 = v.l4&maskLow51Bits + v.l3>>51 + v.l3 = v.l3&maskLow51Bits + v.l2>>51 + v.l2 = v.l2&maskLow51Bits + v.l1>>51 + v.l1 = v.l1&maskLow51Bits + l0>>51 + + return v +} diff --git a/vendor/filippo.io/edwards25519/pull.sh b/vendor/filippo.io/edwards25519/pull.sh new file mode 100644 index 0000000000..f6217c96e2 --- /dev/null +++ b/vendor/filippo.io/edwards25519/pull.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [ "$#" -ne 1 ]; then + echo "Usage: $0 " + exit 1 +fi + +TAG="$1" +TMPDIR="$(mktemp -d)" + +cleanup() { + rm -rf "$TMPDIR" +} +trap cleanup EXIT + +command -v git >/dev/null +command -v git-filter-repo >/dev/null + +if [ -d "$HOME/go/.git" ]; then + REFERENCE=(--reference "$HOME/go" --dissociate) +else + REFERENCE=() +fi + +git -c advice.detachedHead=false clone --no-checkout "${REFERENCE[@]}" \ + -b "$TAG" https://go.googlesource.com/go.git "$TMPDIR" + +# Simplify the history graph by removing the dev.boringcrypto branches, whose +# merges end up empty after grafting anyway. This also fixes a weird quirk +# (maybe a git-filter-repo bug?) where only one file from an old path, +# src/crypto/ed25519/internal/edwards25519/const.go, would still exist in the +# filtered repo. +git -C "$TMPDIR" replace --graft f771edd7f9 99f1bf54eb +git -C "$TMPDIR" replace --graft 109c13b64f c2f96e686f +git -C "$TMPDIR" replace --graft aa4da4f189 912f075047 + +git -C "$TMPDIR" filter-repo --force \ + --paths-from-file /dev/stdin \ + --prune-empty always \ + --prune-degenerate always \ + --tag-callback 'tag.skip()' <<'EOF' +src/crypto/internal/fips140/edwards25519 +src/crypto/internal/edwards25519 +src/crypto/ed25519/internal/edwards25519 +EOF + +git fetch "$TMPDIR" +git update-ref "refs/heads/upstream/$TAG" FETCH_HEAD + +echo +echo "Fetched upstream history up to $TAG. Merge with:" +echo -e "\tgit merge --no-ff --no-commit --allow-unrelated-histories upstream/$TAG" diff --git a/vendor/filippo.io/edwards25519/scalar.go b/vendor/filippo.io/edwards25519/scalar.go new file mode 100644 index 0000000000..f08b26245c --- /dev/null +++ b/vendor/filippo.io/edwards25519/scalar.go @@ -0,0 +1,352 @@ +// Copyright (c) 2016 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package edwards25519 + +import ( + "encoding/binary" + "errors" + "math/bits" +) + +// A Scalar is an integer modulo +// +// l = 2^252 + 27742317777372353535851937790883648493 +// +// which is the prime order of the edwards25519 group. +// +// This type works similarly to math/big.Int, and all arguments and +// receivers are allowed to alias. +// +// The zero value is a valid zero element. +type Scalar struct { + // s is the scalar in the Montgomery domain, in the format of the + // fiat-crypto implementation. + s fiatScalarMontgomeryDomainFieldElement +} + +// The field implementation in scalar_fiat.go is generated by the fiat-crypto +// project (https://github.com/mit-plv/fiat-crypto) at version v0.0.9 (23d2dbc) +// from a formally verified model. +// +// fiat-crypto code comes under the following license. +// +// Copyright (c) 2015-2020 The fiat-crypto Authors. All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// 1. Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// +// THIS SOFTWARE IS PROVIDED BY the fiat-crypto authors "AS IS" +// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +// THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +// PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL Berkeley Software Design, +// Inc. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +// EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +// PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +// LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +// NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +// + +// NewScalar returns a new zero Scalar. +func NewScalar() *Scalar { + return &Scalar{} +} + +// MultiplyAdd sets s = x * y + z mod l, and returns s. It is equivalent to +// using Multiply and then Add. +func (s *Scalar) MultiplyAdd(x, y, z *Scalar) *Scalar { + // Make a copy of z in case it aliases s. + zCopy := new(Scalar).Set(z) + return s.Multiply(x, y).Add(s, zCopy) +} + +// Add sets s = x + y mod l, and returns s. +func (s *Scalar) Add(x, y *Scalar) *Scalar { + // s = 1 * x + y mod l + fiatScalarAdd(&s.s, &x.s, &y.s) + return s +} + +// Subtract sets s = x - y mod l, and returns s. +func (s *Scalar) Subtract(x, y *Scalar) *Scalar { + // s = -1 * y + x mod l + fiatScalarSub(&s.s, &x.s, &y.s) + return s +} + +// Negate sets s = -x mod l, and returns s. +func (s *Scalar) Negate(x *Scalar) *Scalar { + // s = -1 * x + 0 mod l + fiatScalarOpp(&s.s, &x.s) + return s +} + +// Multiply sets s = x * y mod l, and returns s. +func (s *Scalar) Multiply(x, y *Scalar) *Scalar { + // s = x * y + 0 mod l + fiatScalarMul(&s.s, &x.s, &y.s) + return s +} + +// Set sets s = x, and returns s. +func (s *Scalar) Set(x *Scalar) *Scalar { + *s = *x + return s +} + +// SetUniformBytes sets s = x mod l, where x is a 64-byte little-endian integer. +// If x is not of the right length, SetUniformBytes returns nil and an error, +// and the receiver is unchanged. +// +// SetUniformBytes can be used to set s to a uniformly distributed value given +// 64 uniformly distributed random bytes. +func (s *Scalar) SetUniformBytes(x []byte) (*Scalar, error) { + if len(x) != 64 { + return nil, errors.New("edwards25519: invalid SetUniformBytes input length") + } + + // We have a value x of 512 bits, but our fiatScalarFromBytes function + // expects an input lower than l, which is a little over 252 bits. + // + // Instead of writing a reduction function that operates on wider inputs, we + // can interpret x as the sum of three shorter values a, b, and c. + // + // x = a + b * 2^168 + c * 2^336 mod l + // + // We then precompute 2^168 and 2^336 modulo l, and perform the reduction + // with two multiplications and two additions. + + s.setShortBytes(x[:21]) + t := new(Scalar).setShortBytes(x[21:42]) + s.Add(s, t.Multiply(t, scalarTwo168)) + t.setShortBytes(x[42:]) + s.Add(s, t.Multiply(t, scalarTwo336)) + + return s, nil +} + +// scalarTwo168 and scalarTwo336 are 2^168 and 2^336 modulo l, encoded as a +// fiatScalarMontgomeryDomainFieldElement, which is a little-endian 4-limb value +// in the 2^256 Montgomery domain. +var scalarTwo168 = &Scalar{s: [4]uint64{0x5b8ab432eac74798, 0x38afddd6de59d5d7, + 0xa2c131b399411b7c, 0x6329a7ed9ce5a30}} +var scalarTwo336 = &Scalar{s: [4]uint64{0xbd3d108e2b35ecc5, 0x5c3a3718bdf9c90b, + 0x63aa97a331b4f2ee, 0x3d217f5be65cb5c}} + +// setShortBytes sets s = x mod l, where x is a little-endian integer shorter +// than 32 bytes. +func (s *Scalar) setShortBytes(x []byte) *Scalar { + if len(x) >= 32 { + panic("edwards25519: internal error: setShortBytes called with a long string") + } + var buf [32]byte + copy(buf[:], x) + fiatScalarFromBytes((*[4]uint64)(&s.s), &buf) + fiatScalarToMontgomery(&s.s, (*fiatScalarNonMontgomeryDomainFieldElement)(&s.s)) + return s +} + +// SetCanonicalBytes sets s = x, where x is a 32-byte little-endian encoding of +// s, and returns s. If x is not a canonical encoding of s, SetCanonicalBytes +// returns nil and an error, and the receiver is unchanged. +func (s *Scalar) SetCanonicalBytes(x []byte) (*Scalar, error) { + if len(x) != 32 { + return nil, errors.New("invalid scalar length") + } + if !isReduced(x) { + return nil, errors.New("invalid scalar encoding") + } + + fiatScalarFromBytes((*[4]uint64)(&s.s), (*[32]byte)(x)) + fiatScalarToMontgomery(&s.s, (*fiatScalarNonMontgomeryDomainFieldElement)(&s.s)) + + return s, nil +} + +// scalarMinusOneBytes is l - 1 in little endian. +var scalarMinusOneBytes = [32]byte{236, 211, 245, 92, 26, 99, 18, 88, 214, 156, 247, 162, 222, 249, 222, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16} + +// isReduced returns whether the given scalar in 32-byte little endian encoded +// form is reduced modulo l. +func isReduced(s []byte) bool { + if len(s) != 32 { + return false + } + + s0 := binary.LittleEndian.Uint64(s[:8]) + s1 := binary.LittleEndian.Uint64(s[8:16]) + s2 := binary.LittleEndian.Uint64(s[16:24]) + s3 := binary.LittleEndian.Uint64(s[24:]) + + l0 := binary.LittleEndian.Uint64(scalarMinusOneBytes[:8]) + l1 := binary.LittleEndian.Uint64(scalarMinusOneBytes[8:16]) + l2 := binary.LittleEndian.Uint64(scalarMinusOneBytes[16:24]) + l3 := binary.LittleEndian.Uint64(scalarMinusOneBytes[24:]) + + // Do a constant time subtraction chain scalarMinusOneBytes - s. If there is + // a borrow at the end, then s > scalarMinusOneBytes. + _, b := bits.Sub64(l0, s0, 0) + _, b = bits.Sub64(l1, s1, b) + _, b = bits.Sub64(l2, s2, b) + _, b = bits.Sub64(l3, s3, b) + return b == 0 +} + +// SetBytesWithClamping applies the buffer pruning described in RFC 8032, +// Section 5.1.5 (also known as clamping) and sets s to the result. The input +// must be 32 bytes, and it is not modified. If x is not of the right length, +// SetBytesWithClamping returns nil and an error, and the receiver is unchanged. +// +// Note that since Scalar values are always reduced modulo the prime order of +// the curve, the resulting value will not preserve any of the cofactor-clearing +// properties that clamping is meant to provide. It will however work as +// expected as long as it is applied to points on the prime order subgroup, like +// in Ed25519. In fact, it is lost to history why RFC 8032 adopted the +// irrelevant RFC 7748 clamping, but it is now required for compatibility. +func (s *Scalar) SetBytesWithClamping(x []byte) (*Scalar, error) { + // The description above omits the purpose of the high bits of the clamping + // for brevity, but those are also lost to reductions, and are also + // irrelevant to edwards25519 as they protect against a specific + // implementation bug that was once observed in a generic Montgomery ladder. + if len(x) != 32 { + return nil, errors.New("edwards25519: invalid SetBytesWithClamping input length") + } + + // We need to use the wide reduction from SetUniformBytes, since clamping + // sets the 2^254 bit, making the value higher than the order. + var wideBytes [64]byte + copy(wideBytes[:], x[:]) + wideBytes[0] &= 248 + wideBytes[31] &= 63 + wideBytes[31] |= 64 + return s.SetUniformBytes(wideBytes[:]) +} + +// Bytes returns the canonical 32-byte little-endian encoding of s. +func (s *Scalar) Bytes() []byte { + // This function is outlined to make the allocations inline in the caller + // rather than happen on the heap. + var encoded [32]byte + return s.bytes(&encoded) +} + +func (s *Scalar) bytes(out *[32]byte) []byte { + var ss fiatScalarNonMontgomeryDomainFieldElement + fiatScalarFromMontgomery(&ss, &s.s) + fiatScalarToBytes(out, (*[4]uint64)(&ss)) + return out[:] +} + +// Equal returns 1 if s and t are equal, and 0 otherwise. +func (s *Scalar) Equal(t *Scalar) int { + var diff fiatScalarMontgomeryDomainFieldElement + fiatScalarSub(&diff, &s.s, &t.s) + var nonzero uint64 + fiatScalarNonzero(&nonzero, (*[4]uint64)(&diff)) + nonzero |= nonzero >> 32 + nonzero |= nonzero >> 16 + nonzero |= nonzero >> 8 + nonzero |= nonzero >> 4 + nonzero |= nonzero >> 2 + nonzero |= nonzero >> 1 + return int(^nonzero) & 1 +} + +// nonAdjacentForm computes a width-w non-adjacent form for this scalar. +// +// w must be between 2 and 8, or nonAdjacentForm will panic. +func (s *Scalar) nonAdjacentForm(w uint) [256]int8 { + // This implementation is adapted from the one + // in curve25519-dalek and is documented there: + // https://github.com/dalek-cryptography/curve25519-dalek/blob/f630041af28e9a405255f98a8a93adca18e4315b/src/scalar.rs#L800-L871 + b := s.Bytes() + if b[31] > 127 { + panic("scalar has high bit set illegally") + } + if w < 2 { + panic("w must be at least 2 by the definition of NAF") + } else if w > 8 { + panic("NAF digits must fit in int8") + } + + var naf [256]int8 + var digits [5]uint64 + + for i := 0; i < 4; i++ { + digits[i] = binary.LittleEndian.Uint64(b[i*8:]) + } + + width := uint64(1 << w) + windowMask := uint64(width - 1) + + pos := uint(0) + carry := uint64(0) + for pos < 256 { + indexU64 := pos / 64 + indexBit := pos % 64 + var bitBuf uint64 + if indexBit < 64-w { + // This window's bits are contained in a single u64 + bitBuf = digits[indexU64] >> indexBit + } else { + // Combine the current 64 bits with bits from the next 64 + bitBuf = (digits[indexU64] >> indexBit) | (digits[1+indexU64] << (64 - indexBit)) + } + + // Add carry into the current window + window := carry + (bitBuf & windowMask) + + if window&1 == 0 { + // If the window value is even, preserve the carry and continue. + // Why is the carry preserved? + // If carry == 0 and window & 1 == 0, + // then the next carry should be 0 + // If carry == 1 and window & 1 == 0, + // then bit_buf & 1 == 1 so the next carry should be 1 + pos += 1 + continue + } + + if window < width/2 { + carry = 0 + naf[pos] = int8(window) + } else { + carry = 1 + naf[pos] = int8(window) - int8(width) + } + + pos += w + } + return naf +} + +func (s *Scalar) signedRadix16() [64]int8 { + b := s.Bytes() + if b[31] > 127 { + panic("scalar has high bit set illegally") + } + + var digits [64]int8 + + // Compute unsigned radix-16 digits: + for i := 0; i < 32; i++ { + digits[2*i] = int8(b[i] & 15) + digits[2*i+1] = int8((b[i] >> 4) & 15) + } + + // Recenter coefficients: + for i := 0; i < 63; i++ { + carry := (digits[i] + 8) >> 4 + digits[i] -= carry << 4 + digits[i+1] += carry + } + + return digits +} diff --git a/vendor/filippo.io/edwards25519/scalar_fiat.go b/vendor/filippo.io/edwards25519/scalar_fiat.go new file mode 100644 index 0000000000..2e5782b605 --- /dev/null +++ b/vendor/filippo.io/edwards25519/scalar_fiat.go @@ -0,0 +1,1147 @@ +// Code generated by Fiat Cryptography. DO NOT EDIT. +// +// Autogenerated: word_by_word_montgomery --lang Go --cmovznz-by-mul --relax-primitive-carry-to-bitwidth 32,64 --public-function-case camelCase --public-type-case camelCase --private-function-case camelCase --private-type-case camelCase --doc-text-before-function-name '' --doc-newline-before-package-declaration --doc-prepend-header 'Code generated by Fiat Cryptography. DO NOT EDIT.' --package-name edwards25519 Scalar 64 '2^252 + 27742317777372353535851937790883648493' mul add sub opp nonzero from_montgomery to_montgomery to_bytes from_bytes +// +// curve description: Scalar +// +// machine_wordsize = 64 (from "64") +// +// requested operations: mul, add, sub, opp, nonzero, from_montgomery, to_montgomery, to_bytes, from_bytes +// +// m = 0x1000000000000000000000000000000014def9dea2f79cd65812631a5cf5d3ed (from "2^252 + 27742317777372353535851937790883648493") +// +// +// +// NOTE: In addition to the bounds specified above each function, all +// +// functions synthesized for this Montgomery arithmetic require the +// +// input to be strictly less than the prime modulus (m), and also +// +// require the input to be in the unique saturated representation. +// +// All functions also ensure that these two properties are true of +// +// return values. +// +// +// +// Computed values: +// +// eval z = z[0] + (z[1] << 64) + (z[2] << 128) + (z[3] << 192) +// +// bytes_eval z = z[0] + (z[1] << 8) + (z[2] << 16) + (z[3] << 24) + (z[4] << 32) + (z[5] << 40) + (z[6] << 48) + (z[7] << 56) + (z[8] << 64) + (z[9] << 72) + (z[10] << 80) + (z[11] << 88) + (z[12] << 96) + (z[13] << 104) + (z[14] << 112) + (z[15] << 120) + (z[16] << 128) + (z[17] << 136) + (z[18] << 144) + (z[19] << 152) + (z[20] << 160) + (z[21] << 168) + (z[22] << 176) + (z[23] << 184) + (z[24] << 192) + (z[25] << 200) + (z[26] << 208) + (z[27] << 216) + (z[28] << 224) + (z[29] << 232) + (z[30] << 240) + (z[31] << 248) +// +// twos_complement_eval z = let x1 := z[0] + (z[1] << 64) + (z[2] << 128) + (z[3] << 192) in +// +// if x1 & (2^256-1) < 2^255 then x1 & (2^256-1) else (x1 & (2^256-1)) - 2^256 + +package edwards25519 + +import "math/bits" + +type fiatScalarUint1 uint64 // We use uint64 instead of a more narrow type for performance reasons; see https://github.com/mit-plv/fiat-crypto/pull/1006#issuecomment-892625927 +type fiatScalarInt1 int64 // We use uint64 instead of a more narrow type for performance reasons; see https://github.com/mit-plv/fiat-crypto/pull/1006#issuecomment-892625927 + +// The type fiatScalarMontgomeryDomainFieldElement is a field element in the Montgomery domain. +// +// Bounds: [[0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff]] +type fiatScalarMontgomeryDomainFieldElement [4]uint64 + +// The type fiatScalarNonMontgomeryDomainFieldElement is a field element NOT in the Montgomery domain. +// +// Bounds: [[0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff]] +type fiatScalarNonMontgomeryDomainFieldElement [4]uint64 + +// fiatScalarCmovznzU64 is a single-word conditional move. +// +// Postconditions: +// +// out1 = (if arg1 = 0 then arg2 else arg3) +// +// Input Bounds: +// +// arg1: [0x0 ~> 0x1] +// arg2: [0x0 ~> 0xffffffffffffffff] +// arg3: [0x0 ~> 0xffffffffffffffff] +// +// Output Bounds: +// +// out1: [0x0 ~> 0xffffffffffffffff] +func fiatScalarCmovznzU64(out1 *uint64, arg1 fiatScalarUint1, arg2 uint64, arg3 uint64) { + x1 := (uint64(arg1) * 0xffffffffffffffff) + x2 := ((x1 & arg3) | ((^x1) & arg2)) + *out1 = x2 +} + +// fiatScalarMul multiplies two field elements in the Montgomery domain. +// +// Preconditions: +// +// 0 ≤ eval arg1 < m +// 0 ≤ eval arg2 < m +// +// Postconditions: +// +// eval (from_montgomery out1) mod m = (eval (from_montgomery arg1) * eval (from_montgomery arg2)) mod m +// 0 ≤ eval out1 < m +func fiatScalarMul(out1 *fiatScalarMontgomeryDomainFieldElement, arg1 *fiatScalarMontgomeryDomainFieldElement, arg2 *fiatScalarMontgomeryDomainFieldElement) { + x1 := arg1[1] + x2 := arg1[2] + x3 := arg1[3] + x4 := arg1[0] + var x5 uint64 + var x6 uint64 + x6, x5 = bits.Mul64(x4, arg2[3]) + var x7 uint64 + var x8 uint64 + x8, x7 = bits.Mul64(x4, arg2[2]) + var x9 uint64 + var x10 uint64 + x10, x9 = bits.Mul64(x4, arg2[1]) + var x11 uint64 + var x12 uint64 + x12, x11 = bits.Mul64(x4, arg2[0]) + var x13 uint64 + var x14 uint64 + x13, x14 = bits.Add64(x12, x9, uint64(0x0)) + var x15 uint64 + var x16 uint64 + x15, x16 = bits.Add64(x10, x7, uint64(fiatScalarUint1(x14))) + var x17 uint64 + var x18 uint64 + x17, x18 = bits.Add64(x8, x5, uint64(fiatScalarUint1(x16))) + x19 := (uint64(fiatScalarUint1(x18)) + x6) + var x20 uint64 + _, x20 = bits.Mul64(x11, 0xd2b51da312547e1b) + var x22 uint64 + var x23 uint64 + x23, x22 = bits.Mul64(x20, 0x1000000000000000) + var x24 uint64 + var x25 uint64 + x25, x24 = bits.Mul64(x20, 0x14def9dea2f79cd6) + var x26 uint64 + var x27 uint64 + x27, x26 = bits.Mul64(x20, 0x5812631a5cf5d3ed) + var x28 uint64 + var x29 uint64 + x28, x29 = bits.Add64(x27, x24, uint64(0x0)) + x30 := (uint64(fiatScalarUint1(x29)) + x25) + var x32 uint64 + _, x32 = bits.Add64(x11, x26, uint64(0x0)) + var x33 uint64 + var x34 uint64 + x33, x34 = bits.Add64(x13, x28, uint64(fiatScalarUint1(x32))) + var x35 uint64 + var x36 uint64 + x35, x36 = bits.Add64(x15, x30, uint64(fiatScalarUint1(x34))) + var x37 uint64 + var x38 uint64 + x37, x38 = bits.Add64(x17, x22, uint64(fiatScalarUint1(x36))) + var x39 uint64 + var x40 uint64 + x39, x40 = bits.Add64(x19, x23, uint64(fiatScalarUint1(x38))) + var x41 uint64 + var x42 uint64 + x42, x41 = bits.Mul64(x1, arg2[3]) + var x43 uint64 + var x44 uint64 + x44, x43 = bits.Mul64(x1, arg2[2]) + var x45 uint64 + var x46 uint64 + x46, x45 = bits.Mul64(x1, arg2[1]) + var x47 uint64 + var x48 uint64 + x48, x47 = bits.Mul64(x1, arg2[0]) + var x49 uint64 + var x50 uint64 + x49, x50 = bits.Add64(x48, x45, uint64(0x0)) + var x51 uint64 + var x52 uint64 + x51, x52 = bits.Add64(x46, x43, uint64(fiatScalarUint1(x50))) + var x53 uint64 + var x54 uint64 + x53, x54 = bits.Add64(x44, x41, uint64(fiatScalarUint1(x52))) + x55 := (uint64(fiatScalarUint1(x54)) + x42) + var x56 uint64 + var x57 uint64 + x56, x57 = bits.Add64(x33, x47, uint64(0x0)) + var x58 uint64 + var x59 uint64 + x58, x59 = bits.Add64(x35, x49, uint64(fiatScalarUint1(x57))) + var x60 uint64 + var x61 uint64 + x60, x61 = bits.Add64(x37, x51, uint64(fiatScalarUint1(x59))) + var x62 uint64 + var x63 uint64 + x62, x63 = bits.Add64(x39, x53, uint64(fiatScalarUint1(x61))) + var x64 uint64 + var x65 uint64 + x64, x65 = bits.Add64(uint64(fiatScalarUint1(x40)), x55, uint64(fiatScalarUint1(x63))) + var x66 uint64 + _, x66 = bits.Mul64(x56, 0xd2b51da312547e1b) + var x68 uint64 + var x69 uint64 + x69, x68 = bits.Mul64(x66, 0x1000000000000000) + var x70 uint64 + var x71 uint64 + x71, x70 = bits.Mul64(x66, 0x14def9dea2f79cd6) + var x72 uint64 + var x73 uint64 + x73, x72 = bits.Mul64(x66, 0x5812631a5cf5d3ed) + var x74 uint64 + var x75 uint64 + x74, x75 = bits.Add64(x73, x70, uint64(0x0)) + x76 := (uint64(fiatScalarUint1(x75)) + x71) + var x78 uint64 + _, x78 = bits.Add64(x56, x72, uint64(0x0)) + var x79 uint64 + var x80 uint64 + x79, x80 = bits.Add64(x58, x74, uint64(fiatScalarUint1(x78))) + var x81 uint64 + var x82 uint64 + x81, x82 = bits.Add64(x60, x76, uint64(fiatScalarUint1(x80))) + var x83 uint64 + var x84 uint64 + x83, x84 = bits.Add64(x62, x68, uint64(fiatScalarUint1(x82))) + var x85 uint64 + var x86 uint64 + x85, x86 = bits.Add64(x64, x69, uint64(fiatScalarUint1(x84))) + x87 := (uint64(fiatScalarUint1(x86)) + uint64(fiatScalarUint1(x65))) + var x88 uint64 + var x89 uint64 + x89, x88 = bits.Mul64(x2, arg2[3]) + var x90 uint64 + var x91 uint64 + x91, x90 = bits.Mul64(x2, arg2[2]) + var x92 uint64 + var x93 uint64 + x93, x92 = bits.Mul64(x2, arg2[1]) + var x94 uint64 + var x95 uint64 + x95, x94 = bits.Mul64(x2, arg2[0]) + var x96 uint64 + var x97 uint64 + x96, x97 = bits.Add64(x95, x92, uint64(0x0)) + var x98 uint64 + var x99 uint64 + x98, x99 = bits.Add64(x93, x90, uint64(fiatScalarUint1(x97))) + var x100 uint64 + var x101 uint64 + x100, x101 = bits.Add64(x91, x88, uint64(fiatScalarUint1(x99))) + x102 := (uint64(fiatScalarUint1(x101)) + x89) + var x103 uint64 + var x104 uint64 + x103, x104 = bits.Add64(x79, x94, uint64(0x0)) + var x105 uint64 + var x106 uint64 + x105, x106 = bits.Add64(x81, x96, uint64(fiatScalarUint1(x104))) + var x107 uint64 + var x108 uint64 + x107, x108 = bits.Add64(x83, x98, uint64(fiatScalarUint1(x106))) + var x109 uint64 + var x110 uint64 + x109, x110 = bits.Add64(x85, x100, uint64(fiatScalarUint1(x108))) + var x111 uint64 + var x112 uint64 + x111, x112 = bits.Add64(x87, x102, uint64(fiatScalarUint1(x110))) + var x113 uint64 + _, x113 = bits.Mul64(x103, 0xd2b51da312547e1b) + var x115 uint64 + var x116 uint64 + x116, x115 = bits.Mul64(x113, 0x1000000000000000) + var x117 uint64 + var x118 uint64 + x118, x117 = bits.Mul64(x113, 0x14def9dea2f79cd6) + var x119 uint64 + var x120 uint64 + x120, x119 = bits.Mul64(x113, 0x5812631a5cf5d3ed) + var x121 uint64 + var x122 uint64 + x121, x122 = bits.Add64(x120, x117, uint64(0x0)) + x123 := (uint64(fiatScalarUint1(x122)) + x118) + var x125 uint64 + _, x125 = bits.Add64(x103, x119, uint64(0x0)) + var x126 uint64 + var x127 uint64 + x126, x127 = bits.Add64(x105, x121, uint64(fiatScalarUint1(x125))) + var x128 uint64 + var x129 uint64 + x128, x129 = bits.Add64(x107, x123, uint64(fiatScalarUint1(x127))) + var x130 uint64 + var x131 uint64 + x130, x131 = bits.Add64(x109, x115, uint64(fiatScalarUint1(x129))) + var x132 uint64 + var x133 uint64 + x132, x133 = bits.Add64(x111, x116, uint64(fiatScalarUint1(x131))) + x134 := (uint64(fiatScalarUint1(x133)) + uint64(fiatScalarUint1(x112))) + var x135 uint64 + var x136 uint64 + x136, x135 = bits.Mul64(x3, arg2[3]) + var x137 uint64 + var x138 uint64 + x138, x137 = bits.Mul64(x3, arg2[2]) + var x139 uint64 + var x140 uint64 + x140, x139 = bits.Mul64(x3, arg2[1]) + var x141 uint64 + var x142 uint64 + x142, x141 = bits.Mul64(x3, arg2[0]) + var x143 uint64 + var x144 uint64 + x143, x144 = bits.Add64(x142, x139, uint64(0x0)) + var x145 uint64 + var x146 uint64 + x145, x146 = bits.Add64(x140, x137, uint64(fiatScalarUint1(x144))) + var x147 uint64 + var x148 uint64 + x147, x148 = bits.Add64(x138, x135, uint64(fiatScalarUint1(x146))) + x149 := (uint64(fiatScalarUint1(x148)) + x136) + var x150 uint64 + var x151 uint64 + x150, x151 = bits.Add64(x126, x141, uint64(0x0)) + var x152 uint64 + var x153 uint64 + x152, x153 = bits.Add64(x128, x143, uint64(fiatScalarUint1(x151))) + var x154 uint64 + var x155 uint64 + x154, x155 = bits.Add64(x130, x145, uint64(fiatScalarUint1(x153))) + var x156 uint64 + var x157 uint64 + x156, x157 = bits.Add64(x132, x147, uint64(fiatScalarUint1(x155))) + var x158 uint64 + var x159 uint64 + x158, x159 = bits.Add64(x134, x149, uint64(fiatScalarUint1(x157))) + var x160 uint64 + _, x160 = bits.Mul64(x150, 0xd2b51da312547e1b) + var x162 uint64 + var x163 uint64 + x163, x162 = bits.Mul64(x160, 0x1000000000000000) + var x164 uint64 + var x165 uint64 + x165, x164 = bits.Mul64(x160, 0x14def9dea2f79cd6) + var x166 uint64 + var x167 uint64 + x167, x166 = bits.Mul64(x160, 0x5812631a5cf5d3ed) + var x168 uint64 + var x169 uint64 + x168, x169 = bits.Add64(x167, x164, uint64(0x0)) + x170 := (uint64(fiatScalarUint1(x169)) + x165) + var x172 uint64 + _, x172 = bits.Add64(x150, x166, uint64(0x0)) + var x173 uint64 + var x174 uint64 + x173, x174 = bits.Add64(x152, x168, uint64(fiatScalarUint1(x172))) + var x175 uint64 + var x176 uint64 + x175, x176 = bits.Add64(x154, x170, uint64(fiatScalarUint1(x174))) + var x177 uint64 + var x178 uint64 + x177, x178 = bits.Add64(x156, x162, uint64(fiatScalarUint1(x176))) + var x179 uint64 + var x180 uint64 + x179, x180 = bits.Add64(x158, x163, uint64(fiatScalarUint1(x178))) + x181 := (uint64(fiatScalarUint1(x180)) + uint64(fiatScalarUint1(x159))) + var x182 uint64 + var x183 uint64 + x182, x183 = bits.Sub64(x173, 0x5812631a5cf5d3ed, uint64(0x0)) + var x184 uint64 + var x185 uint64 + x184, x185 = bits.Sub64(x175, 0x14def9dea2f79cd6, uint64(fiatScalarUint1(x183))) + var x186 uint64 + var x187 uint64 + x186, x187 = bits.Sub64(x177, uint64(0x0), uint64(fiatScalarUint1(x185))) + var x188 uint64 + var x189 uint64 + x188, x189 = bits.Sub64(x179, 0x1000000000000000, uint64(fiatScalarUint1(x187))) + var x191 uint64 + _, x191 = bits.Sub64(x181, uint64(0x0), uint64(fiatScalarUint1(x189))) + var x192 uint64 + fiatScalarCmovznzU64(&x192, fiatScalarUint1(x191), x182, x173) + var x193 uint64 + fiatScalarCmovznzU64(&x193, fiatScalarUint1(x191), x184, x175) + var x194 uint64 + fiatScalarCmovznzU64(&x194, fiatScalarUint1(x191), x186, x177) + var x195 uint64 + fiatScalarCmovznzU64(&x195, fiatScalarUint1(x191), x188, x179) + out1[0] = x192 + out1[1] = x193 + out1[2] = x194 + out1[3] = x195 +} + +// fiatScalarAdd adds two field elements in the Montgomery domain. +// +// Preconditions: +// +// 0 ≤ eval arg1 < m +// 0 ≤ eval arg2 < m +// +// Postconditions: +// +// eval (from_montgomery out1) mod m = (eval (from_montgomery arg1) + eval (from_montgomery arg2)) mod m +// 0 ≤ eval out1 < m +func fiatScalarAdd(out1 *fiatScalarMontgomeryDomainFieldElement, arg1 *fiatScalarMontgomeryDomainFieldElement, arg2 *fiatScalarMontgomeryDomainFieldElement) { + var x1 uint64 + var x2 uint64 + x1, x2 = bits.Add64(arg1[0], arg2[0], uint64(0x0)) + var x3 uint64 + var x4 uint64 + x3, x4 = bits.Add64(arg1[1], arg2[1], uint64(fiatScalarUint1(x2))) + var x5 uint64 + var x6 uint64 + x5, x6 = bits.Add64(arg1[2], arg2[2], uint64(fiatScalarUint1(x4))) + var x7 uint64 + var x8 uint64 + x7, x8 = bits.Add64(arg1[3], arg2[3], uint64(fiatScalarUint1(x6))) + var x9 uint64 + var x10 uint64 + x9, x10 = bits.Sub64(x1, 0x5812631a5cf5d3ed, uint64(0x0)) + var x11 uint64 + var x12 uint64 + x11, x12 = bits.Sub64(x3, 0x14def9dea2f79cd6, uint64(fiatScalarUint1(x10))) + var x13 uint64 + var x14 uint64 + x13, x14 = bits.Sub64(x5, uint64(0x0), uint64(fiatScalarUint1(x12))) + var x15 uint64 + var x16 uint64 + x15, x16 = bits.Sub64(x7, 0x1000000000000000, uint64(fiatScalarUint1(x14))) + var x18 uint64 + _, x18 = bits.Sub64(uint64(fiatScalarUint1(x8)), uint64(0x0), uint64(fiatScalarUint1(x16))) + var x19 uint64 + fiatScalarCmovznzU64(&x19, fiatScalarUint1(x18), x9, x1) + var x20 uint64 + fiatScalarCmovznzU64(&x20, fiatScalarUint1(x18), x11, x3) + var x21 uint64 + fiatScalarCmovznzU64(&x21, fiatScalarUint1(x18), x13, x5) + var x22 uint64 + fiatScalarCmovznzU64(&x22, fiatScalarUint1(x18), x15, x7) + out1[0] = x19 + out1[1] = x20 + out1[2] = x21 + out1[3] = x22 +} + +// fiatScalarSub subtracts two field elements in the Montgomery domain. +// +// Preconditions: +// +// 0 ≤ eval arg1 < m +// 0 ≤ eval arg2 < m +// +// Postconditions: +// +// eval (from_montgomery out1) mod m = (eval (from_montgomery arg1) - eval (from_montgomery arg2)) mod m +// 0 ≤ eval out1 < m +func fiatScalarSub(out1 *fiatScalarMontgomeryDomainFieldElement, arg1 *fiatScalarMontgomeryDomainFieldElement, arg2 *fiatScalarMontgomeryDomainFieldElement) { + var x1 uint64 + var x2 uint64 + x1, x2 = bits.Sub64(arg1[0], arg2[0], uint64(0x0)) + var x3 uint64 + var x4 uint64 + x3, x4 = bits.Sub64(arg1[1], arg2[1], uint64(fiatScalarUint1(x2))) + var x5 uint64 + var x6 uint64 + x5, x6 = bits.Sub64(arg1[2], arg2[2], uint64(fiatScalarUint1(x4))) + var x7 uint64 + var x8 uint64 + x7, x8 = bits.Sub64(arg1[3], arg2[3], uint64(fiatScalarUint1(x6))) + var x9 uint64 + fiatScalarCmovznzU64(&x9, fiatScalarUint1(x8), uint64(0x0), 0xffffffffffffffff) + var x10 uint64 + var x11 uint64 + x10, x11 = bits.Add64(x1, (x9 & 0x5812631a5cf5d3ed), uint64(0x0)) + var x12 uint64 + var x13 uint64 + x12, x13 = bits.Add64(x3, (x9 & 0x14def9dea2f79cd6), uint64(fiatScalarUint1(x11))) + var x14 uint64 + var x15 uint64 + x14, x15 = bits.Add64(x5, uint64(0x0), uint64(fiatScalarUint1(x13))) + var x16 uint64 + x16, _ = bits.Add64(x7, (x9 & 0x1000000000000000), uint64(fiatScalarUint1(x15))) + out1[0] = x10 + out1[1] = x12 + out1[2] = x14 + out1[3] = x16 +} + +// fiatScalarOpp negates a field element in the Montgomery domain. +// +// Preconditions: +// +// 0 ≤ eval arg1 < m +// +// Postconditions: +// +// eval (from_montgomery out1) mod m = -eval (from_montgomery arg1) mod m +// 0 ≤ eval out1 < m +func fiatScalarOpp(out1 *fiatScalarMontgomeryDomainFieldElement, arg1 *fiatScalarMontgomeryDomainFieldElement) { + var x1 uint64 + var x2 uint64 + x1, x2 = bits.Sub64(uint64(0x0), arg1[0], uint64(0x0)) + var x3 uint64 + var x4 uint64 + x3, x4 = bits.Sub64(uint64(0x0), arg1[1], uint64(fiatScalarUint1(x2))) + var x5 uint64 + var x6 uint64 + x5, x6 = bits.Sub64(uint64(0x0), arg1[2], uint64(fiatScalarUint1(x4))) + var x7 uint64 + var x8 uint64 + x7, x8 = bits.Sub64(uint64(0x0), arg1[3], uint64(fiatScalarUint1(x6))) + var x9 uint64 + fiatScalarCmovznzU64(&x9, fiatScalarUint1(x8), uint64(0x0), 0xffffffffffffffff) + var x10 uint64 + var x11 uint64 + x10, x11 = bits.Add64(x1, (x9 & 0x5812631a5cf5d3ed), uint64(0x0)) + var x12 uint64 + var x13 uint64 + x12, x13 = bits.Add64(x3, (x9 & 0x14def9dea2f79cd6), uint64(fiatScalarUint1(x11))) + var x14 uint64 + var x15 uint64 + x14, x15 = bits.Add64(x5, uint64(0x0), uint64(fiatScalarUint1(x13))) + var x16 uint64 + x16, _ = bits.Add64(x7, (x9 & 0x1000000000000000), uint64(fiatScalarUint1(x15))) + out1[0] = x10 + out1[1] = x12 + out1[2] = x14 + out1[3] = x16 +} + +// fiatScalarNonzero outputs a single non-zero word if the input is non-zero and zero otherwise. +// +// Preconditions: +// +// 0 ≤ eval arg1 < m +// +// Postconditions: +// +// out1 = 0 ↔ eval (from_montgomery arg1) mod m = 0 +// +// Input Bounds: +// +// arg1: [[0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff]] +// +// Output Bounds: +// +// out1: [0x0 ~> 0xffffffffffffffff] +func fiatScalarNonzero(out1 *uint64, arg1 *[4]uint64) { + x1 := (arg1[0] | (arg1[1] | (arg1[2] | arg1[3]))) + *out1 = x1 +} + +// fiatScalarFromMontgomery translates a field element out of the Montgomery domain. +// +// Preconditions: +// +// 0 ≤ eval arg1 < m +// +// Postconditions: +// +// eval out1 mod m = (eval arg1 * ((2^64)⁻¹ mod m)^4) mod m +// 0 ≤ eval out1 < m +func fiatScalarFromMontgomery(out1 *fiatScalarNonMontgomeryDomainFieldElement, arg1 *fiatScalarMontgomeryDomainFieldElement) { + x1 := arg1[0] + var x2 uint64 + _, x2 = bits.Mul64(x1, 0xd2b51da312547e1b) + var x4 uint64 + var x5 uint64 + x5, x4 = bits.Mul64(x2, 0x1000000000000000) + var x6 uint64 + var x7 uint64 + x7, x6 = bits.Mul64(x2, 0x14def9dea2f79cd6) + var x8 uint64 + var x9 uint64 + x9, x8 = bits.Mul64(x2, 0x5812631a5cf5d3ed) + var x10 uint64 + var x11 uint64 + x10, x11 = bits.Add64(x9, x6, uint64(0x0)) + var x13 uint64 + _, x13 = bits.Add64(x1, x8, uint64(0x0)) + var x14 uint64 + var x15 uint64 + x14, x15 = bits.Add64(uint64(0x0), x10, uint64(fiatScalarUint1(x13))) + var x16 uint64 + var x17 uint64 + x16, x17 = bits.Add64(x14, arg1[1], uint64(0x0)) + var x18 uint64 + _, x18 = bits.Mul64(x16, 0xd2b51da312547e1b) + var x20 uint64 + var x21 uint64 + x21, x20 = bits.Mul64(x18, 0x1000000000000000) + var x22 uint64 + var x23 uint64 + x23, x22 = bits.Mul64(x18, 0x14def9dea2f79cd6) + var x24 uint64 + var x25 uint64 + x25, x24 = bits.Mul64(x18, 0x5812631a5cf5d3ed) + var x26 uint64 + var x27 uint64 + x26, x27 = bits.Add64(x25, x22, uint64(0x0)) + var x29 uint64 + _, x29 = bits.Add64(x16, x24, uint64(0x0)) + var x30 uint64 + var x31 uint64 + x30, x31 = bits.Add64((uint64(fiatScalarUint1(x17)) + (uint64(fiatScalarUint1(x15)) + (uint64(fiatScalarUint1(x11)) + x7))), x26, uint64(fiatScalarUint1(x29))) + var x32 uint64 + var x33 uint64 + x32, x33 = bits.Add64(x4, (uint64(fiatScalarUint1(x27)) + x23), uint64(fiatScalarUint1(x31))) + var x34 uint64 + var x35 uint64 + x34, x35 = bits.Add64(x5, x20, uint64(fiatScalarUint1(x33))) + var x36 uint64 + var x37 uint64 + x36, x37 = bits.Add64(x30, arg1[2], uint64(0x0)) + var x38 uint64 + var x39 uint64 + x38, x39 = bits.Add64(x32, uint64(0x0), uint64(fiatScalarUint1(x37))) + var x40 uint64 + var x41 uint64 + x40, x41 = bits.Add64(x34, uint64(0x0), uint64(fiatScalarUint1(x39))) + var x42 uint64 + _, x42 = bits.Mul64(x36, 0xd2b51da312547e1b) + var x44 uint64 + var x45 uint64 + x45, x44 = bits.Mul64(x42, 0x1000000000000000) + var x46 uint64 + var x47 uint64 + x47, x46 = bits.Mul64(x42, 0x14def9dea2f79cd6) + var x48 uint64 + var x49 uint64 + x49, x48 = bits.Mul64(x42, 0x5812631a5cf5d3ed) + var x50 uint64 + var x51 uint64 + x50, x51 = bits.Add64(x49, x46, uint64(0x0)) + var x53 uint64 + _, x53 = bits.Add64(x36, x48, uint64(0x0)) + var x54 uint64 + var x55 uint64 + x54, x55 = bits.Add64(x38, x50, uint64(fiatScalarUint1(x53))) + var x56 uint64 + var x57 uint64 + x56, x57 = bits.Add64(x40, (uint64(fiatScalarUint1(x51)) + x47), uint64(fiatScalarUint1(x55))) + var x58 uint64 + var x59 uint64 + x58, x59 = bits.Add64((uint64(fiatScalarUint1(x41)) + (uint64(fiatScalarUint1(x35)) + x21)), x44, uint64(fiatScalarUint1(x57))) + var x60 uint64 + var x61 uint64 + x60, x61 = bits.Add64(x54, arg1[3], uint64(0x0)) + var x62 uint64 + var x63 uint64 + x62, x63 = bits.Add64(x56, uint64(0x0), uint64(fiatScalarUint1(x61))) + var x64 uint64 + var x65 uint64 + x64, x65 = bits.Add64(x58, uint64(0x0), uint64(fiatScalarUint1(x63))) + var x66 uint64 + _, x66 = bits.Mul64(x60, 0xd2b51da312547e1b) + var x68 uint64 + var x69 uint64 + x69, x68 = bits.Mul64(x66, 0x1000000000000000) + var x70 uint64 + var x71 uint64 + x71, x70 = bits.Mul64(x66, 0x14def9dea2f79cd6) + var x72 uint64 + var x73 uint64 + x73, x72 = bits.Mul64(x66, 0x5812631a5cf5d3ed) + var x74 uint64 + var x75 uint64 + x74, x75 = bits.Add64(x73, x70, uint64(0x0)) + var x77 uint64 + _, x77 = bits.Add64(x60, x72, uint64(0x0)) + var x78 uint64 + var x79 uint64 + x78, x79 = bits.Add64(x62, x74, uint64(fiatScalarUint1(x77))) + var x80 uint64 + var x81 uint64 + x80, x81 = bits.Add64(x64, (uint64(fiatScalarUint1(x75)) + x71), uint64(fiatScalarUint1(x79))) + var x82 uint64 + var x83 uint64 + x82, x83 = bits.Add64((uint64(fiatScalarUint1(x65)) + (uint64(fiatScalarUint1(x59)) + x45)), x68, uint64(fiatScalarUint1(x81))) + x84 := (uint64(fiatScalarUint1(x83)) + x69) + var x85 uint64 + var x86 uint64 + x85, x86 = bits.Sub64(x78, 0x5812631a5cf5d3ed, uint64(0x0)) + var x87 uint64 + var x88 uint64 + x87, x88 = bits.Sub64(x80, 0x14def9dea2f79cd6, uint64(fiatScalarUint1(x86))) + var x89 uint64 + var x90 uint64 + x89, x90 = bits.Sub64(x82, uint64(0x0), uint64(fiatScalarUint1(x88))) + var x91 uint64 + var x92 uint64 + x91, x92 = bits.Sub64(x84, 0x1000000000000000, uint64(fiatScalarUint1(x90))) + var x94 uint64 + _, x94 = bits.Sub64(uint64(0x0), uint64(0x0), uint64(fiatScalarUint1(x92))) + var x95 uint64 + fiatScalarCmovznzU64(&x95, fiatScalarUint1(x94), x85, x78) + var x96 uint64 + fiatScalarCmovznzU64(&x96, fiatScalarUint1(x94), x87, x80) + var x97 uint64 + fiatScalarCmovznzU64(&x97, fiatScalarUint1(x94), x89, x82) + var x98 uint64 + fiatScalarCmovznzU64(&x98, fiatScalarUint1(x94), x91, x84) + out1[0] = x95 + out1[1] = x96 + out1[2] = x97 + out1[3] = x98 +} + +// fiatScalarToMontgomery translates a field element into the Montgomery domain. +// +// Preconditions: +// +// 0 ≤ eval arg1 < m +// +// Postconditions: +// +// eval (from_montgomery out1) mod m = eval arg1 mod m +// 0 ≤ eval out1 < m +func fiatScalarToMontgomery(out1 *fiatScalarMontgomeryDomainFieldElement, arg1 *fiatScalarNonMontgomeryDomainFieldElement) { + x1 := arg1[1] + x2 := arg1[2] + x3 := arg1[3] + x4 := arg1[0] + var x5 uint64 + var x6 uint64 + x6, x5 = bits.Mul64(x4, 0x399411b7c309a3d) + var x7 uint64 + var x8 uint64 + x8, x7 = bits.Mul64(x4, 0xceec73d217f5be65) + var x9 uint64 + var x10 uint64 + x10, x9 = bits.Mul64(x4, 0xd00e1ba768859347) + var x11 uint64 + var x12 uint64 + x12, x11 = bits.Mul64(x4, 0xa40611e3449c0f01) + var x13 uint64 + var x14 uint64 + x13, x14 = bits.Add64(x12, x9, uint64(0x0)) + var x15 uint64 + var x16 uint64 + x15, x16 = bits.Add64(x10, x7, uint64(fiatScalarUint1(x14))) + var x17 uint64 + var x18 uint64 + x17, x18 = bits.Add64(x8, x5, uint64(fiatScalarUint1(x16))) + var x19 uint64 + _, x19 = bits.Mul64(x11, 0xd2b51da312547e1b) + var x21 uint64 + var x22 uint64 + x22, x21 = bits.Mul64(x19, 0x1000000000000000) + var x23 uint64 + var x24 uint64 + x24, x23 = bits.Mul64(x19, 0x14def9dea2f79cd6) + var x25 uint64 + var x26 uint64 + x26, x25 = bits.Mul64(x19, 0x5812631a5cf5d3ed) + var x27 uint64 + var x28 uint64 + x27, x28 = bits.Add64(x26, x23, uint64(0x0)) + var x30 uint64 + _, x30 = bits.Add64(x11, x25, uint64(0x0)) + var x31 uint64 + var x32 uint64 + x31, x32 = bits.Add64(x13, x27, uint64(fiatScalarUint1(x30))) + var x33 uint64 + var x34 uint64 + x33, x34 = bits.Add64(x15, (uint64(fiatScalarUint1(x28)) + x24), uint64(fiatScalarUint1(x32))) + var x35 uint64 + var x36 uint64 + x35, x36 = bits.Add64(x17, x21, uint64(fiatScalarUint1(x34))) + var x37 uint64 + var x38 uint64 + x38, x37 = bits.Mul64(x1, 0x399411b7c309a3d) + var x39 uint64 + var x40 uint64 + x40, x39 = bits.Mul64(x1, 0xceec73d217f5be65) + var x41 uint64 + var x42 uint64 + x42, x41 = bits.Mul64(x1, 0xd00e1ba768859347) + var x43 uint64 + var x44 uint64 + x44, x43 = bits.Mul64(x1, 0xa40611e3449c0f01) + var x45 uint64 + var x46 uint64 + x45, x46 = bits.Add64(x44, x41, uint64(0x0)) + var x47 uint64 + var x48 uint64 + x47, x48 = bits.Add64(x42, x39, uint64(fiatScalarUint1(x46))) + var x49 uint64 + var x50 uint64 + x49, x50 = bits.Add64(x40, x37, uint64(fiatScalarUint1(x48))) + var x51 uint64 + var x52 uint64 + x51, x52 = bits.Add64(x31, x43, uint64(0x0)) + var x53 uint64 + var x54 uint64 + x53, x54 = bits.Add64(x33, x45, uint64(fiatScalarUint1(x52))) + var x55 uint64 + var x56 uint64 + x55, x56 = bits.Add64(x35, x47, uint64(fiatScalarUint1(x54))) + var x57 uint64 + var x58 uint64 + x57, x58 = bits.Add64(((uint64(fiatScalarUint1(x36)) + (uint64(fiatScalarUint1(x18)) + x6)) + x22), x49, uint64(fiatScalarUint1(x56))) + var x59 uint64 + _, x59 = bits.Mul64(x51, 0xd2b51da312547e1b) + var x61 uint64 + var x62 uint64 + x62, x61 = bits.Mul64(x59, 0x1000000000000000) + var x63 uint64 + var x64 uint64 + x64, x63 = bits.Mul64(x59, 0x14def9dea2f79cd6) + var x65 uint64 + var x66 uint64 + x66, x65 = bits.Mul64(x59, 0x5812631a5cf5d3ed) + var x67 uint64 + var x68 uint64 + x67, x68 = bits.Add64(x66, x63, uint64(0x0)) + var x70 uint64 + _, x70 = bits.Add64(x51, x65, uint64(0x0)) + var x71 uint64 + var x72 uint64 + x71, x72 = bits.Add64(x53, x67, uint64(fiatScalarUint1(x70))) + var x73 uint64 + var x74 uint64 + x73, x74 = bits.Add64(x55, (uint64(fiatScalarUint1(x68)) + x64), uint64(fiatScalarUint1(x72))) + var x75 uint64 + var x76 uint64 + x75, x76 = bits.Add64(x57, x61, uint64(fiatScalarUint1(x74))) + var x77 uint64 + var x78 uint64 + x78, x77 = bits.Mul64(x2, 0x399411b7c309a3d) + var x79 uint64 + var x80 uint64 + x80, x79 = bits.Mul64(x2, 0xceec73d217f5be65) + var x81 uint64 + var x82 uint64 + x82, x81 = bits.Mul64(x2, 0xd00e1ba768859347) + var x83 uint64 + var x84 uint64 + x84, x83 = bits.Mul64(x2, 0xa40611e3449c0f01) + var x85 uint64 + var x86 uint64 + x85, x86 = bits.Add64(x84, x81, uint64(0x0)) + var x87 uint64 + var x88 uint64 + x87, x88 = bits.Add64(x82, x79, uint64(fiatScalarUint1(x86))) + var x89 uint64 + var x90 uint64 + x89, x90 = bits.Add64(x80, x77, uint64(fiatScalarUint1(x88))) + var x91 uint64 + var x92 uint64 + x91, x92 = bits.Add64(x71, x83, uint64(0x0)) + var x93 uint64 + var x94 uint64 + x93, x94 = bits.Add64(x73, x85, uint64(fiatScalarUint1(x92))) + var x95 uint64 + var x96 uint64 + x95, x96 = bits.Add64(x75, x87, uint64(fiatScalarUint1(x94))) + var x97 uint64 + var x98 uint64 + x97, x98 = bits.Add64(((uint64(fiatScalarUint1(x76)) + (uint64(fiatScalarUint1(x58)) + (uint64(fiatScalarUint1(x50)) + x38))) + x62), x89, uint64(fiatScalarUint1(x96))) + var x99 uint64 + _, x99 = bits.Mul64(x91, 0xd2b51da312547e1b) + var x101 uint64 + var x102 uint64 + x102, x101 = bits.Mul64(x99, 0x1000000000000000) + var x103 uint64 + var x104 uint64 + x104, x103 = bits.Mul64(x99, 0x14def9dea2f79cd6) + var x105 uint64 + var x106 uint64 + x106, x105 = bits.Mul64(x99, 0x5812631a5cf5d3ed) + var x107 uint64 + var x108 uint64 + x107, x108 = bits.Add64(x106, x103, uint64(0x0)) + var x110 uint64 + _, x110 = bits.Add64(x91, x105, uint64(0x0)) + var x111 uint64 + var x112 uint64 + x111, x112 = bits.Add64(x93, x107, uint64(fiatScalarUint1(x110))) + var x113 uint64 + var x114 uint64 + x113, x114 = bits.Add64(x95, (uint64(fiatScalarUint1(x108)) + x104), uint64(fiatScalarUint1(x112))) + var x115 uint64 + var x116 uint64 + x115, x116 = bits.Add64(x97, x101, uint64(fiatScalarUint1(x114))) + var x117 uint64 + var x118 uint64 + x118, x117 = bits.Mul64(x3, 0x399411b7c309a3d) + var x119 uint64 + var x120 uint64 + x120, x119 = bits.Mul64(x3, 0xceec73d217f5be65) + var x121 uint64 + var x122 uint64 + x122, x121 = bits.Mul64(x3, 0xd00e1ba768859347) + var x123 uint64 + var x124 uint64 + x124, x123 = bits.Mul64(x3, 0xa40611e3449c0f01) + var x125 uint64 + var x126 uint64 + x125, x126 = bits.Add64(x124, x121, uint64(0x0)) + var x127 uint64 + var x128 uint64 + x127, x128 = bits.Add64(x122, x119, uint64(fiatScalarUint1(x126))) + var x129 uint64 + var x130 uint64 + x129, x130 = bits.Add64(x120, x117, uint64(fiatScalarUint1(x128))) + var x131 uint64 + var x132 uint64 + x131, x132 = bits.Add64(x111, x123, uint64(0x0)) + var x133 uint64 + var x134 uint64 + x133, x134 = bits.Add64(x113, x125, uint64(fiatScalarUint1(x132))) + var x135 uint64 + var x136 uint64 + x135, x136 = bits.Add64(x115, x127, uint64(fiatScalarUint1(x134))) + var x137 uint64 + var x138 uint64 + x137, x138 = bits.Add64(((uint64(fiatScalarUint1(x116)) + (uint64(fiatScalarUint1(x98)) + (uint64(fiatScalarUint1(x90)) + x78))) + x102), x129, uint64(fiatScalarUint1(x136))) + var x139 uint64 + _, x139 = bits.Mul64(x131, 0xd2b51da312547e1b) + var x141 uint64 + var x142 uint64 + x142, x141 = bits.Mul64(x139, 0x1000000000000000) + var x143 uint64 + var x144 uint64 + x144, x143 = bits.Mul64(x139, 0x14def9dea2f79cd6) + var x145 uint64 + var x146 uint64 + x146, x145 = bits.Mul64(x139, 0x5812631a5cf5d3ed) + var x147 uint64 + var x148 uint64 + x147, x148 = bits.Add64(x146, x143, uint64(0x0)) + var x150 uint64 + _, x150 = bits.Add64(x131, x145, uint64(0x0)) + var x151 uint64 + var x152 uint64 + x151, x152 = bits.Add64(x133, x147, uint64(fiatScalarUint1(x150))) + var x153 uint64 + var x154 uint64 + x153, x154 = bits.Add64(x135, (uint64(fiatScalarUint1(x148)) + x144), uint64(fiatScalarUint1(x152))) + var x155 uint64 + var x156 uint64 + x155, x156 = bits.Add64(x137, x141, uint64(fiatScalarUint1(x154))) + x157 := ((uint64(fiatScalarUint1(x156)) + (uint64(fiatScalarUint1(x138)) + (uint64(fiatScalarUint1(x130)) + x118))) + x142) + var x158 uint64 + var x159 uint64 + x158, x159 = bits.Sub64(x151, 0x5812631a5cf5d3ed, uint64(0x0)) + var x160 uint64 + var x161 uint64 + x160, x161 = bits.Sub64(x153, 0x14def9dea2f79cd6, uint64(fiatScalarUint1(x159))) + var x162 uint64 + var x163 uint64 + x162, x163 = bits.Sub64(x155, uint64(0x0), uint64(fiatScalarUint1(x161))) + var x164 uint64 + var x165 uint64 + x164, x165 = bits.Sub64(x157, 0x1000000000000000, uint64(fiatScalarUint1(x163))) + var x167 uint64 + _, x167 = bits.Sub64(uint64(0x0), uint64(0x0), uint64(fiatScalarUint1(x165))) + var x168 uint64 + fiatScalarCmovznzU64(&x168, fiatScalarUint1(x167), x158, x151) + var x169 uint64 + fiatScalarCmovznzU64(&x169, fiatScalarUint1(x167), x160, x153) + var x170 uint64 + fiatScalarCmovznzU64(&x170, fiatScalarUint1(x167), x162, x155) + var x171 uint64 + fiatScalarCmovznzU64(&x171, fiatScalarUint1(x167), x164, x157) + out1[0] = x168 + out1[1] = x169 + out1[2] = x170 + out1[3] = x171 +} + +// fiatScalarToBytes serializes a field element NOT in the Montgomery domain to bytes in little-endian order. +// +// Preconditions: +// +// 0 ≤ eval arg1 < m +// +// Postconditions: +// +// out1 = map (λ x, ⌊((eval arg1 mod m) mod 2^(8 * (x + 1))) / 2^(8 * x)⌋) [0..31] +// +// Input Bounds: +// +// arg1: [[0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0x1fffffffffffffff]] +// +// Output Bounds: +// +// out1: [[0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0x1f]] +func fiatScalarToBytes(out1 *[32]uint8, arg1 *[4]uint64) { + x1 := arg1[3] + x2 := arg1[2] + x3 := arg1[1] + x4 := arg1[0] + x5 := (uint8(x4) & 0xff) + x6 := (x4 >> 8) + x7 := (uint8(x6) & 0xff) + x8 := (x6 >> 8) + x9 := (uint8(x8) & 0xff) + x10 := (x8 >> 8) + x11 := (uint8(x10) & 0xff) + x12 := (x10 >> 8) + x13 := (uint8(x12) & 0xff) + x14 := (x12 >> 8) + x15 := (uint8(x14) & 0xff) + x16 := (x14 >> 8) + x17 := (uint8(x16) & 0xff) + x18 := uint8((x16 >> 8)) + x19 := (uint8(x3) & 0xff) + x20 := (x3 >> 8) + x21 := (uint8(x20) & 0xff) + x22 := (x20 >> 8) + x23 := (uint8(x22) & 0xff) + x24 := (x22 >> 8) + x25 := (uint8(x24) & 0xff) + x26 := (x24 >> 8) + x27 := (uint8(x26) & 0xff) + x28 := (x26 >> 8) + x29 := (uint8(x28) & 0xff) + x30 := (x28 >> 8) + x31 := (uint8(x30) & 0xff) + x32 := uint8((x30 >> 8)) + x33 := (uint8(x2) & 0xff) + x34 := (x2 >> 8) + x35 := (uint8(x34) & 0xff) + x36 := (x34 >> 8) + x37 := (uint8(x36) & 0xff) + x38 := (x36 >> 8) + x39 := (uint8(x38) & 0xff) + x40 := (x38 >> 8) + x41 := (uint8(x40) & 0xff) + x42 := (x40 >> 8) + x43 := (uint8(x42) & 0xff) + x44 := (x42 >> 8) + x45 := (uint8(x44) & 0xff) + x46 := uint8((x44 >> 8)) + x47 := (uint8(x1) & 0xff) + x48 := (x1 >> 8) + x49 := (uint8(x48) & 0xff) + x50 := (x48 >> 8) + x51 := (uint8(x50) & 0xff) + x52 := (x50 >> 8) + x53 := (uint8(x52) & 0xff) + x54 := (x52 >> 8) + x55 := (uint8(x54) & 0xff) + x56 := (x54 >> 8) + x57 := (uint8(x56) & 0xff) + x58 := (x56 >> 8) + x59 := (uint8(x58) & 0xff) + x60 := uint8((x58 >> 8)) + out1[0] = x5 + out1[1] = x7 + out1[2] = x9 + out1[3] = x11 + out1[4] = x13 + out1[5] = x15 + out1[6] = x17 + out1[7] = x18 + out1[8] = x19 + out1[9] = x21 + out1[10] = x23 + out1[11] = x25 + out1[12] = x27 + out1[13] = x29 + out1[14] = x31 + out1[15] = x32 + out1[16] = x33 + out1[17] = x35 + out1[18] = x37 + out1[19] = x39 + out1[20] = x41 + out1[21] = x43 + out1[22] = x45 + out1[23] = x46 + out1[24] = x47 + out1[25] = x49 + out1[26] = x51 + out1[27] = x53 + out1[28] = x55 + out1[29] = x57 + out1[30] = x59 + out1[31] = x60 +} + +// fiatScalarFromBytes deserializes a field element NOT in the Montgomery domain from bytes in little-endian order. +// +// Preconditions: +// +// 0 ≤ bytes_eval arg1 < m +// +// Postconditions: +// +// eval out1 mod m = bytes_eval arg1 mod m +// 0 ≤ eval out1 < m +// +// Input Bounds: +// +// arg1: [[0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0xff], [0x0 ~> 0x1f]] +// +// Output Bounds: +// +// out1: [[0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0xffffffffffffffff], [0x0 ~> 0x1fffffffffffffff]] +func fiatScalarFromBytes(out1 *[4]uint64, arg1 *[32]uint8) { + x1 := (uint64(arg1[31]) << 56) + x2 := (uint64(arg1[30]) << 48) + x3 := (uint64(arg1[29]) << 40) + x4 := (uint64(arg1[28]) << 32) + x5 := (uint64(arg1[27]) << 24) + x6 := (uint64(arg1[26]) << 16) + x7 := (uint64(arg1[25]) << 8) + x8 := arg1[24] + x9 := (uint64(arg1[23]) << 56) + x10 := (uint64(arg1[22]) << 48) + x11 := (uint64(arg1[21]) << 40) + x12 := (uint64(arg1[20]) << 32) + x13 := (uint64(arg1[19]) << 24) + x14 := (uint64(arg1[18]) << 16) + x15 := (uint64(arg1[17]) << 8) + x16 := arg1[16] + x17 := (uint64(arg1[15]) << 56) + x18 := (uint64(arg1[14]) << 48) + x19 := (uint64(arg1[13]) << 40) + x20 := (uint64(arg1[12]) << 32) + x21 := (uint64(arg1[11]) << 24) + x22 := (uint64(arg1[10]) << 16) + x23 := (uint64(arg1[9]) << 8) + x24 := arg1[8] + x25 := (uint64(arg1[7]) << 56) + x26 := (uint64(arg1[6]) << 48) + x27 := (uint64(arg1[5]) << 40) + x28 := (uint64(arg1[4]) << 32) + x29 := (uint64(arg1[3]) << 24) + x30 := (uint64(arg1[2]) << 16) + x31 := (uint64(arg1[1]) << 8) + x32 := arg1[0] + x33 := (x31 + uint64(x32)) + x34 := (x30 + x33) + x35 := (x29 + x34) + x36 := (x28 + x35) + x37 := (x27 + x36) + x38 := (x26 + x37) + x39 := (x25 + x38) + x40 := (x23 + uint64(x24)) + x41 := (x22 + x40) + x42 := (x21 + x41) + x43 := (x20 + x42) + x44 := (x19 + x43) + x45 := (x18 + x44) + x46 := (x17 + x45) + x47 := (x15 + uint64(x16)) + x48 := (x14 + x47) + x49 := (x13 + x48) + x50 := (x12 + x49) + x51 := (x11 + x50) + x52 := (x10 + x51) + x53 := (x9 + x52) + x54 := (x7 + uint64(x8)) + x55 := (x6 + x54) + x56 := (x5 + x55) + x57 := (x4 + x56) + x58 := (x3 + x57) + x59 := (x2 + x58) + x60 := (x1 + x59) + out1[0] = x39 + out1[1] = x46 + out1[2] = x53 + out1[3] = x60 +} diff --git a/vendor/filippo.io/edwards25519/scalarmult.go b/vendor/filippo.io/edwards25519/scalarmult.go new file mode 100644 index 0000000000..f7ca3cef99 --- /dev/null +++ b/vendor/filippo.io/edwards25519/scalarmult.go @@ -0,0 +1,214 @@ +// Copyright (c) 2019 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package edwards25519 + +import "sync" + +// basepointTable is a set of 32 affineLookupTables, where table i is generated +// from 256i * basepoint. It is precomputed the first time it's used. +func basepointTable() *[32]affineLookupTable { + basepointTablePrecomp.initOnce.Do(func() { + p := NewGeneratorPoint() + for i := 0; i < 32; i++ { + basepointTablePrecomp.table[i].FromP3(p) + for j := 0; j < 8; j++ { + p.Add(p, p) + } + } + }) + return &basepointTablePrecomp.table +} + +var basepointTablePrecomp struct { + table [32]affineLookupTable + initOnce sync.Once +} + +// ScalarBaseMult sets v = x * B, where B is the canonical generator, and +// returns v. +// +// The scalar multiplication is done in constant time. +func (v *Point) ScalarBaseMult(x *Scalar) *Point { + basepointTable := basepointTable() + + // Write x = sum(x_i * 16^i) so x*B = sum( B*x_i*16^i ) + // as described in the Ed25519 paper + // + // Group even and odd coefficients + // x*B = x_0*16^0*B + x_2*16^2*B + ... + x_62*16^62*B + // + x_1*16^1*B + x_3*16^3*B + ... + x_63*16^63*B + // x*B = x_0*16^0*B + x_2*16^2*B + ... + x_62*16^62*B + // + 16*( x_1*16^0*B + x_3*16^2*B + ... + x_63*16^62*B) + // + // We use a lookup table for each i to get x_i*16^(2*i)*B + // and do four doublings to multiply by 16. + digits := x.signedRadix16() + + multiple := &affineCached{} + tmp1 := &projP1xP1{} + tmp2 := &projP2{} + + // Accumulate the odd components first + v.Set(NewIdentityPoint()) + for i := 1; i < 64; i += 2 { + basepointTable[i/2].SelectInto(multiple, digits[i]) + tmp1.AddAffine(v, multiple) + v.fromP1xP1(tmp1) + } + + // Multiply by 16 + tmp2.FromP3(v) // tmp2 = v in P2 coords + tmp1.Double(tmp2) // tmp1 = 2*v in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 2*v in P2 coords + tmp1.Double(tmp2) // tmp1 = 4*v in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 4*v in P2 coords + tmp1.Double(tmp2) // tmp1 = 8*v in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 8*v in P2 coords + tmp1.Double(tmp2) // tmp1 = 16*v in P1xP1 coords + v.fromP1xP1(tmp1) // now v = 16*(odd components) + + // Accumulate the even components + for i := 0; i < 64; i += 2 { + basepointTable[i/2].SelectInto(multiple, digits[i]) + tmp1.AddAffine(v, multiple) + v.fromP1xP1(tmp1) + } + + return v +} + +// ScalarMult sets v = x * q, and returns v. +// +// The scalar multiplication is done in constant time. +func (v *Point) ScalarMult(x *Scalar, q *Point) *Point { + checkInitialized(q) + + var table projLookupTable + table.FromP3(q) + + // Write x = sum(x_i * 16^i) + // so x*Q = sum( Q*x_i*16^i ) + // = Q*x_0 + 16*(Q*x_1 + 16*( ... + Q*x_63) ... ) + // <------compute inside out--------- + // + // We use the lookup table to get the x_i*Q values + // and do four doublings to compute 16*Q + digits := x.signedRadix16() + + // Unwrap first loop iteration to save computing 16*identity + multiple := &projCached{} + tmp1 := &projP1xP1{} + tmp2 := &projP2{} + table.SelectInto(multiple, digits[63]) + + v.Set(NewIdentityPoint()) + tmp1.Add(v, multiple) // tmp1 = x_63*Q in P1xP1 coords + for i := 62; i >= 0; i-- { + tmp2.FromP1xP1(tmp1) // tmp2 = (prev) in P2 coords + tmp1.Double(tmp2) // tmp1 = 2*(prev) in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 2*(prev) in P2 coords + tmp1.Double(tmp2) // tmp1 = 4*(prev) in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 4*(prev) in P2 coords + tmp1.Double(tmp2) // tmp1 = 8*(prev) in P1xP1 coords + tmp2.FromP1xP1(tmp1) // tmp2 = 8*(prev) in P2 coords + tmp1.Double(tmp2) // tmp1 = 16*(prev) in P1xP1 coords + v.fromP1xP1(tmp1) // v = 16*(prev) in P3 coords + table.SelectInto(multiple, digits[i]) + tmp1.Add(v, multiple) // tmp1 = x_i*Q + 16*(prev) in P1xP1 coords + } + v.fromP1xP1(tmp1) + return v +} + +// basepointNafTable is the nafLookupTable8 for the basepoint. +// It is precomputed the first time it's used. +func basepointNafTable() *nafLookupTable8 { + basepointNafTablePrecomp.initOnce.Do(func() { + basepointNafTablePrecomp.table.FromP3(NewGeneratorPoint()) + }) + return &basepointNafTablePrecomp.table +} + +var basepointNafTablePrecomp struct { + table nafLookupTable8 + initOnce sync.Once +} + +// VarTimeDoubleScalarBaseMult sets v = a * A + b * B, where B is the canonical +// generator, and returns v. +// +// Execution time depends on the inputs. +func (v *Point) VarTimeDoubleScalarBaseMult(a *Scalar, A *Point, b *Scalar) *Point { + checkInitialized(A) + + // Similarly to the single variable-base approach, we compute + // digits and use them with a lookup table. However, because + // we are allowed to do variable-time operations, we don't + // need constant-time lookups or constant-time digit + // computations. + // + // So we use a non-adjacent form of some width w instead of + // radix 16. This is like a binary representation (one digit + // for each binary place) but we allow the digits to grow in + // magnitude up to 2^{w-1} so that the nonzero digits are as + // sparse as possible. Intuitively, this "condenses" the + // "mass" of the scalar onto sparse coefficients (meaning + // fewer additions). + + basepointNafTable := basepointNafTable() + var aTable nafLookupTable5 + aTable.FromP3(A) + // Because the basepoint is fixed, we can use a wider NAF + // corresponding to a bigger table. + aNaf := a.nonAdjacentForm(5) + bNaf := b.nonAdjacentForm(8) + + // Find the first nonzero coefficient. + i := 255 + for j := i; j >= 0; j-- { + if aNaf[j] != 0 || bNaf[j] != 0 { + break + } + } + + multA := &projCached{} + multB := &affineCached{} + tmp1 := &projP1xP1{} + tmp2 := &projP2{} + tmp2.Zero() + + // Move from high to low bits, doubling the accumulator + // at each iteration and checking whether there is a nonzero + // coefficient to look up a multiple of. + for ; i >= 0; i-- { + tmp1.Double(tmp2) + + // Only update v if we have a nonzero coeff to add in. + if aNaf[i] > 0 { + v.fromP1xP1(tmp1) + aTable.SelectInto(multA, aNaf[i]) + tmp1.Add(v, multA) + } else if aNaf[i] < 0 { + v.fromP1xP1(tmp1) + aTable.SelectInto(multA, -aNaf[i]) + tmp1.Sub(v, multA) + } + + if bNaf[i] > 0 { + v.fromP1xP1(tmp1) + basepointNafTable.SelectInto(multB, bNaf[i]) + tmp1.AddAffine(v, multB) + } else if bNaf[i] < 0 { + v.fromP1xP1(tmp1) + basepointNafTable.SelectInto(multB, -bNaf[i]) + tmp1.SubAffine(v, multB) + } + + tmp2.FromP1xP1(tmp1) + } + + v.fromP2(tmp2) + return v +} diff --git a/vendor/filippo.io/edwards25519/tables.go b/vendor/filippo.io/edwards25519/tables.go new file mode 100644 index 0000000000..4a2b54ebad --- /dev/null +++ b/vendor/filippo.io/edwards25519/tables.go @@ -0,0 +1,127 @@ +// Copyright (c) 2019 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package edwards25519 + +import "crypto/subtle" + +// A dynamic lookup table for variable-base, constant-time scalar muls. +type projLookupTable struct { + points [8]projCached +} + +// A precomputed lookup table for fixed-base, constant-time scalar muls. +type affineLookupTable struct { + points [8]affineCached +} + +// A dynamic lookup table for variable-base, variable-time scalar muls. +type nafLookupTable5 struct { + points [8]projCached +} + +// A precomputed lookup table for fixed-base, variable-time scalar muls. +type nafLookupTable8 struct { + points [64]affineCached +} + +// Constructors. + +// Builds a lookup table at runtime. Fast. +func (v *projLookupTable) FromP3(q *Point) { + // Goal: v.points[i] = (i+1)*Q, i.e., Q, 2Q, ..., 8Q + // This allows lookup of -8Q, ..., -Q, 0, Q, ..., 8Q + v.points[0].FromP3(q) + tmpP3 := Point{} + tmpP1xP1 := projP1xP1{} + for i := 0; i < 7; i++ { + // Compute (i+1)*Q as Q + i*Q and convert to a projCached + // This is needlessly complicated because the API has explicit + // receivers instead of creating stack objects and relying on RVO + v.points[i+1].FromP3(tmpP3.fromP1xP1(tmpP1xP1.Add(q, &v.points[i]))) + } +} + +// This is not optimised for speed; fixed-base tables should be precomputed. +func (v *affineLookupTable) FromP3(q *Point) { + // Goal: v.points[i] = (i+1)*Q, i.e., Q, 2Q, ..., 8Q + // This allows lookup of -8Q, ..., -Q, 0, Q, ..., 8Q + v.points[0].FromP3(q) + tmpP3 := Point{} + tmpP1xP1 := projP1xP1{} + for i := 0; i < 7; i++ { + // Compute (i+1)*Q as Q + i*Q and convert to affineCached + v.points[i+1].FromP3(tmpP3.fromP1xP1(tmpP1xP1.AddAffine(q, &v.points[i]))) + } +} + +// Builds a lookup table at runtime. Fast. +func (v *nafLookupTable5) FromP3(q *Point) { + // Goal: v.points[i] = (2*i+1)*Q, i.e., Q, 3Q, 5Q, ..., 15Q + // This allows lookup of -15Q, ..., -3Q, -Q, 0, Q, 3Q, ..., 15Q + v.points[0].FromP3(q) + q2 := Point{} + q2.Add(q, q) + tmpP3 := Point{} + tmpP1xP1 := projP1xP1{} + for i := 0; i < 7; i++ { + v.points[i+1].FromP3(tmpP3.fromP1xP1(tmpP1xP1.Add(&q2, &v.points[i]))) + } +} + +// This is not optimised for speed; fixed-base tables should be precomputed. +func (v *nafLookupTable8) FromP3(q *Point) { + v.points[0].FromP3(q) + q2 := Point{} + q2.Add(q, q) + tmpP3 := Point{} + tmpP1xP1 := projP1xP1{} + for i := 0; i < 63; i++ { + v.points[i+1].FromP3(tmpP3.fromP1xP1(tmpP1xP1.AddAffine(&q2, &v.points[i]))) + } +} + +// Selectors. + +// Set dest to x*Q, where -8 <= x <= 8, in constant time. +func (v *projLookupTable) SelectInto(dest *projCached, x int8) { + // Compute xabs = |x| + xmask := x >> 7 + xabs := uint8((x + xmask) ^ xmask) + + dest.Zero() + for j := 1; j <= 8; j++ { + // Set dest = j*Q if |x| = j + cond := subtle.ConstantTimeByteEq(xabs, uint8(j)) + dest.Select(&v.points[j-1], dest, cond) + } + // Now dest = |x|*Q, conditionally negate to get x*Q + dest.CondNeg(int(xmask & 1)) +} + +// Set dest to x*Q, where -8 <= x <= 8, in constant time. +func (v *affineLookupTable) SelectInto(dest *affineCached, x int8) { + // Compute xabs = |x| + xmask := x >> 7 + xabs := uint8((x + xmask) ^ xmask) + + dest.Zero() + for j := 1; j <= 8; j++ { + // Set dest = j*Q if |x| = j + cond := subtle.ConstantTimeByteEq(xabs, uint8(j)) + dest.Select(&v.points[j-1], dest, cond) + } + // Now dest = |x|*Q, conditionally negate to get x*Q + dest.CondNeg(int(xmask & 1)) +} + +// Given odd x with 0 < x < 2^4, return x*Q (in variable time). +func (v *nafLookupTable5) SelectInto(dest *projCached, x int8) { + *dest = v.points[x/2] +} + +// Given odd x with 0 < x < 2^7, return x*Q (in variable time). +func (v *nafLookupTable8) SelectInto(dest *affineCached, x int8) { + *dest = v.points[x/2] +} diff --git a/vendor/github.com/go-sql-driver/mysql/.gitignore b/vendor/github.com/go-sql-driver/mysql/.gitignore new file mode 100644 index 0000000000..2de28da166 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/.gitignore @@ -0,0 +1,9 @@ +.DS_Store +.DS_Store? +._* +.Spotlight-V100 +.Trashes +Icon? +ehthumbs.db +Thumbs.db +.idea diff --git a/vendor/github.com/go-sql-driver/mysql/AUTHORS b/vendor/github.com/go-sql-driver/mysql/AUTHORS new file mode 100644 index 0000000000..37c96208a5 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/AUTHORS @@ -0,0 +1,160 @@ +# This is the official list of Go-MySQL-Driver authors for copyright purposes. + +# If you are submitting a patch, please add your name or the name of the +# organization which holds the copyright to this list in alphabetical order. + +# Names should be added to this file as +# Name +# The email address is not required for organizations. +# Please keep the list sorted. + + +# Individual Persons + +Aaron Hopkins +Achille Roussel +Aidan +Alex Snast +Alexey Palazhchenko +Andrew Reid +Animesh Ray +Ariel Mashraki +Arne Hormann +Artur Melanchyk +Asta Xie +B Lamarche +Bes Dollma +Bogdan Constantinescu +Brad Higgins +Brian Hendriks +Bulat Gaifullin +Caine Jette +Carlos Nieto +Chris Kirkland +Chris Moos +Craig Wilson +Daemonxiao <735462752 at qq.com> +Daniel Montoya +Daniel Nichter +Daniël van Eeden +Dave Protasowski +Demouth +Diego Dupin +Dirkjan Bussink +DisposaBoy +Egor Smolyakov +Erwan Martin +Evan Elias +Evan Shaw +Frederick Mayle +Gustavo Kristic +Gusted +Hajime Nakagami +Hanno Braun +Henri Yandell +Hirotaka Yamamoto +Huyiguang +ICHINOSE Shogo +Ilia Cimpoes +INADA Naoki +Jacek Szwec +Jakub Adamus +James Harr +Janek Vedock +Jason Ng +Jean-Yves Pellé +Jeff Hodges +Jeffrey Charles +Jennifer Purevsuren +Jerome Meyer +Jiabin Zhang +Jiajia Zhong +Jian Zhen +Joe Mann +Joshua Prunier +Julien Lefevre +Julien Schmidt +Justin Li +Justin Nuß +Kamil Dziedzic +Kei Kamikawa +Kevin Malachowski +Kieron Woodhouse +Lance Tian +Lennart Rudolph +Leonardo YongUk Kim +Linh Tran Tuan +Lion Yang +Luca Looz +Lucas Liu +Luke Scott +Lunny Xiao +Maciej Zimnoch +Michael Woolnough +Minh Quang +Morgan Tocker +Nao Yokotsuka +Nathanial Murphy +Nicola Peduzzi +Oliver Bone +Olivier Mengué +oscarzhao +Paul Bonser +Paulius Lozys +Peter Schultz +Phil Porada +Rebecca Chin +Reed Allman +Richard Wilkes +Robert Russell +Runrioter Wung +Samantha Frank +Santhosh Kumar Tekuri +Sho Iizuka +Sho Ikeda +Shuode Li +Simon J Mudd +Soroush Pour +Stan Putrya +Stanley Gunawan +Steven Hartland +Tan Jinhua <312841925 at qq.com> +Tetsuro Aoki +Thomas Wodarek +Tim Ruffles +Tom Jenkinson +Vladimir Kovpak +Vladyslav Zhelezniak +Xiangyu Hu +Xiaobing Jiang +Xiuming Chen +Xuehong Chan +Yoshito Ohata +Zhang Xiang +Zhenye Xie +Zhixin Wen +Ziheng Lyu + +# Organizations + +Barracuda Networks, Inc. +Block, Inc. +Counting Ltd. +Defined Networking Inc. +DigitalOcean Inc. +Dolthub Inc. +dyves labs AG +Facebook Inc. +GitHub Inc. +Google Inc. +InfoSum Ltd. +Keybase Inc. +Microsoft Corp. +Multiplay Ltd. +Percona LLC +PingCAP Inc. +Pivotal Inc. +Shattered Silicon Ltd. +Stripe Inc. +ThousandEyes +Zendesk Inc. diff --git a/vendor/github.com/go-sql-driver/mysql/CHANGELOG.md b/vendor/github.com/go-sql-driver/mysql/CHANGELOG.md new file mode 100644 index 0000000000..72a35516dc --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/CHANGELOG.md @@ -0,0 +1,383 @@ +# Changelog + +## v1.10.1 (2026-09-02) + +* Fix `Config.FormatDSN()` dropping `Addr` when `Net` is empty. + It now uses the default `tcp` network so configs with only `Addr` round-trip correctly. (#1770) + +* Fix typed-nil `json.RawMessage` with `interpolateParams=true` being interpolated as an empty string. + It is now interpolated as SQL `NULL`, matching server-side prepared statements. (#1782) + +* Add MariaDB 11.8 and 12.3 to the test matrix. (#1774) + +## v1.10.0 (2026-04-28) + +* Fix `getSystemVar("max_allowed_packet")` potentially returned wrong value. (#1754) + This affects only when `maxAllowedPacket=0` is set. + +* Bump filippo.io/edwards25519 from 1.1.1 to 1.2.0. (#1756) + While older versions have reported CVEs, they do not affect go-mysql. + +* Update Go versions to 1.24-1.26. (#1763) + +* Enhance interpolateParams to correctly handle placeholders. (#1732) + The question mark (?) within strings and comments will no longer be treated as a placeholder. + + +## v1.9.3 (2025-06-13) + +* `tx.Commit()` and `tx.Rollback()` returned `ErrInvalidConn` always. + Now they return cached real error if present. (#1690) + +* Optimize reading small result sets to fix a performance regression + introduced by compression protocol support. (`#1707`) +* Fix `db.Ping()` on compressed connection. (#1723) + + +## v1.9.2 (2025-04-07) + +v1.9.2 is a re-release of v1.9.1 due to a release process issue; no changes were made to the content. + + +## v1.9.1 (2025-03-21) + +### Major Changes + +* Add Charset() option. (#1679) + +### Bugfixes + +* go.mod: fix go version format (#1682) +* Fix FormatDSN missing ConnectionAttributes (#1619) + +## v1.9.0 (2025-02-18) + +### Major Changes + +- Implement zlib compression. (#1487) +- Supported Go version is updated to Go 1.21+. (#1639) +- Add support for VECTOR type introduced in MySQL 9.0. (#1609) +- Config object can have custom dial function. (#1527) + +### Bugfixes + +- Fix auth errors when username/password are too long. (#1625) +- Check if MySQL supports CLIENT_CONNECT_ATTRS before sending client attributes. (#1640) +- Fix auth switch request handling. (#1666) + +### Other changes + +- Add "filename:line" prefix to log in go-mysql. Custom loggers now show it. (#1589) +- Improve error handling. It reduces the "busy buffer" errors. (#1595, #1601, #1641) +- Use `strconv.Atoi` to parse max_allowed_packet. (#1661) +- `rejectReadOnly` option now handles ER_READ_ONLY_MODE (1290) error too. (#1660) + + +## Version 1.8.1 (2024-03-26) + +Bugfixes: + +- fix race condition when context is canceled in [#1562](https://github.com/go-sql-driver/mysql/pull/1562) and [#1570](https://github.com/go-sql-driver/mysql/pull/1570) + +## Version 1.8.0 (2024-03-09) + +Major Changes: + +- Use `SET NAMES charset COLLATE collation`. by @methane in [#1437](https://github.com/go-sql-driver/mysql/pull/1437) + - Older go-mysql-driver used `collation_id` in the handshake packet. But it caused collation mismatch in some situation. + - If you don't specify charset nor collation, go-mysql-driver sends `SET NAMES utf8mb4` for new connection. This uses server's default collation for utf8mb4. + - If you specify charset, go-mysql-driver sends `SET NAMES `. This uses the server's default collation for ``. + - If you specify collation and/or charset, go-mysql-driver sends `SET NAMES charset COLLATE collation`. +- PathEscape dbname in DSN. by @methane in [#1432](https://github.com/go-sql-driver/mysql/pull/1432) + - This is backward incompatible in rare case. Check your DSN. +- Drop Go 1.13-17 support by @methane in [#1420](https://github.com/go-sql-driver/mysql/pull/1420) + - Use Go 1.18+ +- Parse numbers on text protocol too by @methane in [#1452](https://github.com/go-sql-driver/mysql/pull/1452) + - When text protocol is used, go-mysql-driver passed bare `[]byte` to database/sql for avoid unnecessary allocation and conversion. + - If user specified `*any` to `Scan()`, database/sql passed the `[]byte` into the target variable. + - This confused users because most user doesn't know when text/binary protocol used. + - go-mysql-driver 1.8 converts integer/float values into int64/double even in text protocol. This doesn't increase allocation compared to `[]byte` and conversion cost is negatable. +- New options start using the Functional Option Pattern to avoid increasing technical debt in the Config object. Future version may introduce Functional Option for existing options, but not for now. + - Make TimeTruncate functional option by @methane in [1552](https://github.com/go-sql-driver/mysql/pull/1552) + - Add BeforeConnect callback to configuration object by @ItalyPaleAle in [#1469](https://github.com/go-sql-driver/mysql/pull/1469) + + +Other changes: + +- Adding DeregisterDialContext to prevent memory leaks with dialers we don't need anymore by @jypelle in https://github.com/go-sql-driver/mysql/pull/1422 +- Make logger configurable per connection by @frozenbonito in https://github.com/go-sql-driver/mysql/pull/1408 +- Fix ColumnType.DatabaseTypeName for mediumint unsigned by @evanelias in https://github.com/go-sql-driver/mysql/pull/1428 +- Add connection attributes by @Daemonxiao in https://github.com/go-sql-driver/mysql/pull/1389 +- Stop `ColumnTypeScanType()` from returning `sql.RawBytes` by @methane in https://github.com/go-sql-driver/mysql/pull/1424 +- Exec() now provides access to status of multiple statements. by @mherr-google in https://github.com/go-sql-driver/mysql/pull/1309 +- Allow to change (or disable) the default driver name for registration by @dolmen in https://github.com/go-sql-driver/mysql/pull/1499 +- Add default connection attribute '_server_host' by @oblitorum in https://github.com/go-sql-driver/mysql/pull/1506 +- QueryUnescape DSN ConnectionAttribute value by @zhangyangyu in https://github.com/go-sql-driver/mysql/pull/1470 +- Add client_ed25519 authentication by @Gusted in https://github.com/go-sql-driver/mysql/pull/1518 + +## Version 1.7.1 (2023-04-25) + +Changes: + + - bump actions/checkout@v3 and actions/setup-go@v3 (#1375) + - Add go1.20 and mariadb10.11 to the testing matrix (#1403) + - Increase default maxAllowedPacket size. (#1411) + +Bugfixes: + + - Use SET syntax as specified in the MySQL documentation (#1402) + + +## Version 1.7 (2022-11-29) + +Changes: + + - Drop support of Go 1.12 (#1211) + - Refactoring `(*textRows).readRow` in a more clear way (#1230) + - util: Reduce boundary check in escape functions. (#1316) + - enhancement for mysqlConn handleAuthResult (#1250) + +New Features: + + - support Is comparison on MySQLError (#1210) + - return unsigned in database type name when necessary (#1238) + - Add API to express like a --ssl-mode=PREFERRED MySQL client (#1370) + - Add SQLState to MySQLError (#1321) + +Bugfixes: + + - Fix parsing 0 year. (#1257) + + +## Version 1.6 (2021-04-01) + +Changes: + + - Migrate the CI service from travis-ci to GitHub Actions (#1176, #1183, #1190) + - `NullTime` is deprecated (#960, #1144) + - Reduce allocations when building SET command (#1111) + - Performance improvement for time formatting (#1118) + - Performance improvement for time parsing (#1098, #1113) + +New Features: + + - Implement `driver.Validator` interface (#1106, #1174) + - Support returning `uint64` from `Valuer` in `ConvertValue` (#1143) + - Add `json.RawMessage` for converter and prepared statement (#1059) + - Interpolate `json.RawMessage` as `string` (#1058) + - Implements `CheckNamedValue` (#1090) + +Bugfixes: + + - Stop rounding times (#1121, #1172) + - Put zero filler into the SSL handshake packet (#1066) + - Fix checking cancelled connections back into the connection pool (#1095) + - Fix remove last 0 byte for mysql_old_password when password is empty (#1133) + + +## Version 1.5 (2020-01-07) + +Changes: + + - Dropped support Go 1.9 and lower (#823, #829, #886, #1016, #1017) + - Improve buffer handling (#890) + - Document potentially insecure TLS configs (#901) + - Use a double-buffering scheme to prevent data races (#943) + - Pass uint64 values without converting them to string (#838, #955) + - Update collations and make utf8mb4 default (#877, #1054) + - Make NullTime compatible with sql.NullTime in Go 1.13+ (#995) + - Removed CloudSQL support (#993, #1007) + - Add Go Module support (#1003) + +New Features: + + - Implement support of optional TLS (#900) + - Check connection liveness (#934, #964, #997, #1048, #1051, #1052) + - Implement Connector Interface (#941, #958, #1020, #1035) + +Bugfixes: + + - Mark connections as bad on error during ping (#875) + - Mark connections as bad on error during dial (#867) + - Fix connection leak caused by rapid context cancellation (#1024) + - Mark connections as bad on error during Conn.Prepare (#1030) + + +## Version 1.4.1 (2018-11-14) + +Bugfixes: + + - Fix TIME format for binary columns (#818) + - Fix handling of empty auth plugin names (#835) + - Fix caching_sha2_password with empty password (#826) + - Fix canceled context broke mysqlConn (#862) + - Fix OldAuthSwitchRequest support (#870) + - Fix Auth Response packet for cleartext password (#887) + +## Version 1.4 (2018-06-03) + +Changes: + + - Documentation fixes (#530, #535, #567) + - Refactoring (#575, #579, #580, #581, #603, #615, #704) + - Cache column names (#444) + - Sort the DSN parameters in DSNs generated from a config (#637) + - Allow native password authentication by default (#644) + - Use the default port if it is missing in the DSN (#668) + - Removed the `strict` mode (#676) + - Do not query `max_allowed_packet` by default (#680) + - Dropped support Go 1.6 and lower (#696) + - Updated `ConvertValue()` to match the database/sql/driver implementation (#760) + - Document the usage of `0000-00-00T00:00:00` as the time.Time zero value (#783) + - Improved the compatibility of the authentication system (#807) + +New Features: + + - Multi-Results support (#537) + - `rejectReadOnly` DSN option (#604) + - `context.Context` support (#608, #612, #627, #761) + - Transaction isolation level support (#619, #744) + - Read-Only transactions support (#618, #634) + - `NewConfig` function which initializes a config with default values (#679) + - Implemented the `ColumnType` interfaces (#667, #724) + - Support for custom string types in `ConvertValue` (#623) + - Implemented `NamedValueChecker`, improving support for uint64 with high bit set (#690, #709, #710) + - `caching_sha2_password` authentication plugin support (#794, #800, #801, #802) + - Implemented `driver.SessionResetter` (#779) + - `sha256_password` authentication plugin support (#808) + +Bugfixes: + + - Use the DSN hostname as TLS default ServerName if `tls=true` (#564, #718) + - Fixed LOAD LOCAL DATA INFILE for empty files (#590) + - Removed columns definition cache since it sometimes cached invalid data (#592) + - Don't mutate registered TLS configs (#600) + - Make RegisterTLSConfig concurrency-safe (#613) + - Handle missing auth data in the handshake packet correctly (#646) + - Do not retry queries when data was written to avoid data corruption (#302, #736) + - Cache the connection pointer for error handling before invalidating it (#678) + - Fixed imports for appengine/cloudsql (#700) + - Fix sending STMT_LONG_DATA for 0 byte data (#734) + - Set correct capacity for []bytes read from length-encoded strings (#766) + - Make RegisterDial concurrency-safe (#773) + + +## Version 1.3 (2016-12-01) + +Changes: + + - Go 1.1 is no longer supported + - Use decimals fields in MySQL to format time types (#249) + - Buffer optimizations (#269) + - TLS ServerName defaults to the host (#283) + - Refactoring (#400, #410, #437) + - Adjusted documentation for second generation CloudSQL (#485) + - Documented DSN system var quoting rules (#502) + - Made statement.Close() calls idempotent to avoid errors in Go 1.6+ (#512) + +New Features: + + - Enable microsecond resolution on TIME, DATETIME and TIMESTAMP (#249) + - Support for returning table alias on Columns() (#289, #359, #382) + - Placeholder interpolation, can be activated with the DSN parameter `interpolateParams=true` (#309, #318, #490) + - Support for uint64 parameters with high bit set (#332, #345) + - Cleartext authentication plugin support (#327) + - Exported ParseDSN function and the Config struct (#403, #419, #429) + - Read / Write timeouts (#401) + - Support for JSON field type (#414) + - Support for multi-statements and multi-results (#411, #431) + - DSN parameter to set the driver-side max_allowed_packet value manually (#489) + - Native password authentication plugin support (#494, #524) + +Bugfixes: + + - Fixed handling of queries without columns and rows (#255) + - Fixed a panic when SetKeepAlive() failed (#298) + - Handle ERR packets while reading rows (#321) + - Fixed reading NULL length-encoded integers in MySQL 5.6+ (#349) + - Fixed absolute paths support in LOAD LOCAL DATA INFILE (#356) + - Actually zero out bytes in handshake response (#378) + - Fixed race condition in registering LOAD DATA INFILE handler (#383) + - Fixed tests with MySQL 5.7.9+ (#380) + - QueryUnescape TLS config names (#397) + - Fixed "broken pipe" error by writing to closed socket (#390) + - Fixed LOAD LOCAL DATA INFILE buffering (#424) + - Fixed parsing of floats into float64 when placeholders are used (#434) + - Fixed DSN tests with Go 1.7+ (#459) + - Handle ERR packets while waiting for EOF (#473) + - Invalidate connection on error while discarding additional results (#513) + - Allow terminating packets of length 0 (#516) + + +## Version 1.2 (2014-06-03) + +Changes: + + - We switched back to a "rolling release". `go get` installs the current master branch again + - Version v1 of the driver will not be maintained anymore. Go 1.0 is no longer supported by this driver + - Exported errors to allow easy checking from application code + - Enabled TCP Keepalives on TCP connections + - Optimized INFILE handling (better buffer size calculation, lazy init, ...) + - The DSN parser also checks for a missing separating slash + - Faster binary date / datetime to string formatting + - Also exported the MySQLWarning type + - mysqlConn.Close returns the first error encountered instead of ignoring all errors + - writePacket() automatically writes the packet size to the header + - readPacket() uses an iterative approach instead of the recursive approach to merge split packets + +New Features: + + - `RegisterDial` allows the usage of a custom dial function to establish the network connection + - Setting the connection collation is possible with the `collation` DSN parameter. This parameter should be preferred over the `charset` parameter + - Logging of critical errors is configurable with `SetLogger` + - Google CloudSQL support + +Bugfixes: + + - Allow more than 32 parameters in prepared statements + - Various old_password fixes + - Fixed TestConcurrent test to pass Go's race detection + - Fixed appendLengthEncodedInteger for large numbers + - Renamed readLengthEnodedString to readLengthEncodedString and skipLengthEnodedString to skipLengthEncodedString (fixed typo) + + +## Version 1.1 (2013-11-02) + +Changes: + + - Go-MySQL-Driver now requires Go 1.1 + - Connections now use the collation `utf8_general_ci` by default. Adding `&charset=UTF8` to the DSN should not be necessary anymore + - Made closing rows and connections error tolerant. This allows for example deferring rows.Close() without checking for errors + - `[]byte(nil)` is now treated as a NULL value. Before, it was treated like an empty string / `[]byte("")` + - DSN parameter values must now be url.QueryEscape'ed. This allows text values to contain special characters, such as '&'. + - Use the IO buffer also for writing. This results in zero allocations (by the driver) for most queries + - Optimized the buffer for reading + - stmt.Query now caches column metadata + - New Logo + - Changed the copyright header to include all contributors + - Improved the LOAD INFILE documentation + - The driver struct is now exported to make the driver directly accessible + - Refactored the driver tests + - Added more benchmarks and moved all to a separate file + - Other small refactoring + +New Features: + + - Added *old_passwords* support: Required in some cases, but must be enabled by adding `allowOldPasswords=true` to the DSN since it is insecure + - Added a `clientFoundRows` parameter: Return the number of matching rows instead of the number of rows changed on UPDATEs + - Added TLS/SSL support: Use a TLS/SSL encrypted connection to the server. Custom TLS configs can be registered and used + +Bugfixes: + + - Fixed MySQL 4.1 support: MySQL 4.1 sends packets with lengths which differ from the specification + - Convert to DB timezone when inserting `time.Time` + - Split packets (more than 16MB) are now merged correctly + - Fixed false positive `io.EOF` errors when the data was fully read + - Avoid panics on reuse of closed connections + - Fixed empty string producing false nil values + - Fixed sign byte for positive TIME fields + + +## Version 1.0 (2013-05-14) + +Initial Release diff --git a/vendor/github.com/go-sql-driver/mysql/LICENSE b/vendor/github.com/go-sql-driver/mysql/LICENSE new file mode 100644 index 0000000000..14e2f777f6 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/LICENSE @@ -0,0 +1,373 @@ +Mozilla Public License Version 2.0 +================================== + +1. Definitions +-------------- + +1.1. "Contributor" + means each individual or legal entity that creates, contributes to + the creation of, or owns Covered Software. + +1.2. "Contributor Version" + means the combination of the Contributions of others (if any) used + by a Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + means Source Code Form to which the initial Contributor has attached + the notice in Exhibit A, the Executable Form of such Source Code + Form, and Modifications of such Source Code Form, in each case + including portions thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + (a) that the initial Contributor has attached the notice described + in Exhibit B to the Covered Software; or + + (b) that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the + terms of a Secondary License. + +1.6. "Executable Form" + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + means a work that combines Covered Software with other material, in + a separate file or files, that is not Covered Software. + +1.8. "License" + means this document. + +1.9. "Licensable" + means having the right to grant, to the maximum extent possible, + whether at the time of the initial grant or subsequently, any and + all of the rights conveyed by this License. + +1.10. "Modifications" + means any of the following: + + (a) any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered + Software; or + + (b) any new file in Source Code Form that contains any Covered + Software. + +1.11. "Patent Claims" of a Contributor + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the + License, by the making, using, selling, offering for sale, having + made, import, or transfer of either its Contributions or its + Contributor Version. + +1.12. "Secondary License" + means either the GNU General Public License, Version 2.0, the GNU + Lesser General Public License, Version 2.1, the GNU Affero General + Public License, Version 3.0, or any later versions of those + licenses. + +1.13. "Source Code Form" + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that + controls, is controlled by, or is under common control with You. For + purposes of this definition, "control" means (a) the power, direct + or indirect, to cause the direction or management of such entity, + whether by contract or otherwise, or (b) ownership of more than + fifty percent (50%) of the outstanding shares or beneficial + ownership of such entity. + +2. License Grants and Conditions +-------------------------------- + +2.1. Grants + +Each Contributor hereby grants You a world-wide, royalty-free, +non-exclusive license: + +(a) under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + +(b) under Patent Claims of such Contributor to make, use, sell, offer + for sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + +The licenses granted in Section 2.1 with respect to any Contribution +become effective for each Contribution on the date the Contributor first +distributes such Contribution. + +2.3. Limitations on Grant Scope + +The licenses granted in this Section 2 are the only rights granted under +this License. No additional rights or licenses will be implied from the +distribution or licensing of Covered Software under this License. +Notwithstanding Section 2.1(b) above, no patent license is granted by a +Contributor: + +(a) for any code that a Contributor has removed from Covered Software; + or + +(b) for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + +(c) under Patent Claims infringed by Covered Software in the absence of + its Contributions. + +This License does not grant any rights in the trademarks, service marks, +or logos of any Contributor (except as may be necessary to comply with +the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + +No Contributor makes additional grants as a result of Your choice to +distribute the Covered Software under a subsequent version of this +License (see Section 10.2) or under the terms of a Secondary License (if +permitted under the terms of Section 3.3). + +2.5. Representation + +Each Contributor represents that the Contributor believes its +Contributions are its original creation(s) or it has sufficient rights +to grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + +This License is not intended to limit any rights You have under +applicable copyright doctrines of fair use, fair dealing, or other +equivalents. + +2.7. Conditions + +Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted +in Section 2.1. + +3. Responsibilities +------------------- + +3.1. Distribution of Source Form + +All distribution of Covered Software in Source Code Form, including any +Modifications that You create or to which You contribute, must be under +the terms of this License. You must inform recipients that the Source +Code Form of the Covered Software is governed by the terms of this +License, and how they can obtain a copy of this License. You may not +attempt to alter or restrict the recipients' rights in the Source Code +Form. + +3.2. Distribution of Executable Form + +If You distribute Covered Software in Executable Form then: + +(a) such Covered Software must also be made available in Source Code + Form, as described in Section 3.1, and You must inform recipients of + the Executable Form how they can obtain a copy of such Source Code + Form by reasonable means in a timely manner, at a charge no more + than the cost of distribution to the recipient; and + +(b) You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter + the recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + +You may create and distribute a Larger Work under terms of Your choice, +provided that You also comply with the requirements of this License for +the Covered Software. If the Larger Work is a combination of Covered +Software with a work governed by one or more Secondary Licenses, and the +Covered Software is not Incompatible With Secondary Licenses, this +License permits You to additionally distribute such Covered Software +under the terms of such Secondary License(s), so that the recipient of +the Larger Work may, at their option, further distribute the Covered +Software under the terms of either this License or such Secondary +License(s). + +3.4. Notices + +You may not remove or alter the substance of any license notices +(including copyright notices, patent notices, disclaimers of warranty, +or limitations of liability) contained within the Source Code Form of +the Covered Software, except that You may alter any license notices to +the extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + +You may choose to offer, and to charge a fee for, warranty, support, +indemnity or liability obligations to one or more recipients of Covered +Software. However, You may do so only on Your own behalf, and not on +behalf of any Contributor. You must make it absolutely clear that any +such warranty, support, indemnity, or liability obligation is offered by +You alone, and You hereby agree to indemnify every Contributor for any +liability incurred by such Contributor as a result of warranty, support, +indemnity or liability terms You offer. You may include additional +disclaimers of warranty and limitations of liability specific to any +jurisdiction. + +4. Inability to Comply Due to Statute or Regulation +--------------------------------------------------- + +If it is impossible for You to comply with any of the terms of this +License with respect to some or all of the Covered Software due to +statute, judicial order, or regulation then You must: (a) comply with +the terms of this License to the maximum extent possible; and (b) +describe the limitations and the code they affect. Such description must +be placed in a text file included with all distributions of the Covered +Software under this License. Except to the extent prohibited by statute +or regulation, such description must be sufficiently detailed for a +recipient of ordinary skill to be able to understand it. + +5. Termination +-------------- + +5.1. The rights granted under this License will terminate automatically +if You fail to comply with any of its terms. However, if You become +compliant, then the rights granted under this License from a particular +Contributor are reinstated (a) provisionally, unless and until such +Contributor explicitly and finally terminates Your grants, and (b) on an +ongoing basis, if such Contributor fails to notify You of the +non-compliance by some reasonable means prior to 60 days after You have +come back into compliance. Moreover, Your grants from a particular +Contributor are reinstated on an ongoing basis if such Contributor +notifies You of the non-compliance by some reasonable means, this is the +first time You have received notice of non-compliance with this License +from such Contributor, and You become compliant prior to 30 days after +Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent +infringement claim (excluding declaratory judgment actions, +counter-claims, and cross-claims) alleging that a Contributor Version +directly or indirectly infringes any patent, then the rights granted to +You by any and all Contributors for the Covered Software under Section +2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all +end user license agreements (excluding distributors and resellers) which +have been validly granted by You or Your distributors under this License +prior to termination shall survive termination. + +************************************************************************ +* * +* 6. Disclaimer of Warranty * +* ------------------------- * +* * +* Covered Software is provided under this License on an "as is" * +* basis, without warranty of any kind, either expressed, implied, or * +* statutory, including, without limitation, warranties that the * +* Covered Software is free of defects, merchantable, fit for a * +* particular purpose or non-infringing. The entire risk as to the * +* quality and performance of the Covered Software is with You. * +* Should any Covered Software prove defective in any respect, You * +* (not any Contributor) assume the cost of any necessary servicing, * +* repair, or correction. This disclaimer of warranty constitutes an * +* essential part of this License. No use of any Covered Software is * +* authorized under this License except under this disclaimer. * +* * +************************************************************************ + +************************************************************************ +* * +* 7. Limitation of Liability * +* -------------------------- * +* * +* Under no circumstances and under no legal theory, whether tort * +* (including negligence), contract, or otherwise, shall any * +* Contributor, or anyone who distributes Covered Software as * +* permitted above, be liable to You for any direct, indirect, * +* special, incidental, or consequential damages of any character * +* including, without limitation, damages for lost profits, loss of * +* goodwill, work stoppage, computer failure or malfunction, or any * +* and all other commercial damages or losses, even if such party * +* shall have been informed of the possibility of such damages. This * +* limitation of liability shall not apply to liability for death or * +* personal injury resulting from such party's negligence to the * +* extent applicable law prohibits such limitation. Some * +* jurisdictions do not allow the exclusion or limitation of * +* incidental or consequential damages, so this exclusion and * +* limitation may not apply to You. * +* * +************************************************************************ + +8. Litigation +------------- + +Any litigation relating to this License may be brought only in the +courts of a jurisdiction where the defendant maintains its principal +place of business and such litigation shall be governed by laws of that +jurisdiction, without reference to its conflict-of-law provisions. +Nothing in this Section shall prevent a party's ability to bring +cross-claims or counter-claims. + +9. Miscellaneous +---------------- + +This License represents the complete agreement concerning the subject +matter hereof. If any provision of this License is held to be +unenforceable, such provision shall be reformed only to the extent +necessary to make it enforceable. Any law or regulation which provides +that the language of a contract shall be construed against the drafter +shall not be used to construe this License against a Contributor. + +10. Versions of the License +--------------------------- + +10.1. New Versions + +Mozilla Foundation is the license steward. Except as provided in Section +10.3, no one other than the license steward has the right to modify or +publish new versions of this License. Each version will be given a +distinguishing version number. + +10.2. Effect of New Versions + +You may distribute the Covered Software under the terms of the version +of the License under which You originally received the Covered Software, +or under the terms of any subsequent version published by the license +steward. + +10.3. Modified Versions + +If you create software not governed by this License, and you want to +create a new license for such software, you may create and use a +modified version of this License if you rename the license and remove +any references to the name of the license steward (except to note that +such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary +Licenses + +If You choose to distribute Source Code Form that is Incompatible With +Secondary Licenses under the terms of this version of the License, the +notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice +------------------------------------------- + + This Source Code Form is subject to the terms of the Mozilla Public + License, v. 2.0. If a copy of the MPL was not distributed with this + file, You can obtain one at http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular +file, then You may include the notice in a location (such as a LICENSE +file in a relevant directory) where a recipient would be likely to look +for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice +--------------------------------------------------------- + + This Source Code Form is "Incompatible With Secondary Licenses", as + defined by the Mozilla Public License, v. 2.0. diff --git a/vendor/github.com/go-sql-driver/mysql/README.md b/vendor/github.com/go-sql-driver/mysql/README.md new file mode 100644 index 0000000000..ccbe6d0787 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/README.md @@ -0,0 +1,601 @@ +# Go-MySQL-Driver + +[![DeepWiki](https://img.shields.io/badge/DeepWiki-go--sql--driver%2Fmysql-blue.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McDcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==)](https://deepwiki.com/go-sql-driver/mysql) + + +A MySQL-Driver for Go's [database/sql](https://golang.org/pkg/database/sql/) package + +![Go-MySQL-Driver logo](https://raw.github.com/wiki/go-sql-driver/mysql/gomysql_m.png "Golang Gopher holding the MySQL Dolphin") + +--------------------------------------- + * [Features](#features) + * [Requirements](#requirements) + * [Installation](#installation) + * [Usage](#usage) + * [DSN (Data Source Name)](#dsn-data-source-name) + * [Password](#password) + * [Protocol](#protocol) + * [Address](#address) + * [Parameters](#parameters) + * [Examples](#examples) + * [Connection pool and timeouts](#connection-pool-and-timeouts) + * [context.Context Support](#contextcontext-support) + * [ColumnType Support](#columntype-support) + * [LOAD DATA LOCAL INFILE support](#load-data-local-infile-support) + * [time.Time support](#timetime-support) + * [Unicode support](#unicode-support) + * [Testing / Development](#testing--development) + * [License](#license) + +--------------------------------------- + +## Features + * Lightweight and [fast](https://github.com/go-sql-driver/sql-benchmark "golang MySQL-Driver performance") + * Native Go implementation. No C-bindings, just pure Go + * Connections over TCP/IPv4, TCP/IPv6, Unix domain sockets or [custom protocols](https://godoc.org/github.com/go-sql-driver/mysql#DialFunc) + * Automatic handling of broken connections + * Automatic Connection Pooling *(by database/sql package)* + * Supports queries larger than 16MB + * Full [`sql.RawBytes`](https://golang.org/pkg/database/sql/#RawBytes) support. + * Intelligent `LONG DATA` handling in prepared statements + * Secure `LOAD DATA LOCAL INFILE` support with file allowlisting and `io.Reader` support + * Optional `time.Time` parsing + * Optional placeholder interpolation + * Supports zlib compression. + +## Requirements + +* Go 1.24 or higher. We aim to support the 3 latest versions of Go. +* MySQL (5.7+) and MariaDB (10.5+) are supported by maintainers. +* [TiDB](https://github.com/pingcap/tidb) is supported by PingCAP. + * Do not ask questions about TiDB in our issue tracker or forum. + * [Document](https://docs.pingcap.com/tidb/v6.1/dev-guide-sample-application-golang) + * [Forum](https://ask.pingcap.com/) +* go-mysql would work with Percona Server, Google CloudSQL or Sphinx (2.2.3+). + * Maintainers won't support them. Do not expect issues are investigated and resolved by maintainers. + * Investigate issues yourself and please send a pull request to fix it. + +--------------------------------------- + +## Installation +Simple install the package to your [$GOPATH](https://github.com/golang/go/wiki/GOPATH "GOPATH") with the [go tool](https://golang.org/cmd/go/ "go command") from shell: +```bash +go get -u github.com/go-sql-driver/mysql +``` +Make sure [Git is installed](https://git-scm.com/downloads) on your machine and in your system's `PATH`. + +## Usage +_Go MySQL Driver_ is an implementation of Go's `database/sql/driver` interface. You only need to import the driver and can use the full [`database/sql`](https://golang.org/pkg/database/sql/) API then. + +Use `mysql` as `driverName` and a valid [DSN](#dsn-data-source-name) as `dataSourceName`: + +```go +import ( + "database/sql" + "time" + + _ "github.com/go-sql-driver/mysql" +) + +// ... + +db, err := sql.Open("mysql", "user:password@/dbname") +if err != nil { + panic(err) +} +// See "Important settings" section. +db.SetConnMaxLifetime(time.Minute * 3) +db.SetMaxOpenConns(10) +db.SetMaxIdleConns(10) +``` + +[Examples are available in our Wiki](https://github.com/go-sql-driver/mysql/wiki/Examples "Go-MySQL-Driver Examples"). + +### Important settings + +`db.SetConnMaxLifetime()` is required to ensure connections are closed by the driver safely before connection is closed by MySQL server, OS, or other middlewares. Since some middlewares close idle connections by 5 minutes, we recommend timeout shorter than 5 minutes. This setting helps load balancing and changing system variables too. + +`db.SetMaxOpenConns()` is highly recommended to limit the number of connection used by the application. There is no recommended limit number because it depends on application and MySQL server. + +`db.SetMaxIdleConns()` is recommended to be set same to `db.SetMaxOpenConns()`. When it is smaller than `SetMaxOpenConns()`, connections can be opened and closed much more frequently than you expect. Idle connections can be closed by the `db.SetConnMaxLifetime()`. If you want to close idle connections more rapidly, you can use `db.SetConnMaxIdleTime()` since Go 1.15. + + +### DSN (Data Source Name) + +The Data Source Name has a common format, like e.g. [PEAR DB](http://pear.php.net/manual/en/package.database.db.intro-dsn.php) uses it, but without type-prefix (optional parts marked by squared brackets): +``` +[username[:password]@][protocol[(address)]]/dbname[?param1=value1&...¶mN=valueN] +``` + +A DSN in its fullest form: +``` +username:password@protocol(address)/dbname?param=value +``` + +Except for the databasename, all values are optional. So the minimal DSN is: +``` +/dbname +``` + +If you do not want to preselect a database, leave `dbname` empty: +``` +/ +``` +This has the same effect as an empty DSN string: +``` + +``` + +`dbname` is escaped by [PathEscape()](https://pkg.go.dev/net/url#PathEscape) since v1.8.0. If your database name is `dbname/withslash`, it becomes: + +``` +/dbname%2Fwithslash +``` + +Alternatively, [Config.FormatDSN](https://godoc.org/github.com/go-sql-driver/mysql#Config.FormatDSN) can be used to create a DSN string by filling a struct. + +#### Password +Passwords can consist of any character. Escaping is **not** necessary. + +#### Protocol +See [net.Dial](https://golang.org/pkg/net/#Dial) for more information which networks are available. +In general you should use a Unix domain socket if available and TCP otherwise for best performance. + +#### Address +For TCP and UDP networks, addresses have the form `host[:port]`. +If `port` is omitted, the default port will be used. +If `host` is a literal IPv6 address, it must be enclosed in square brackets. +The functions [net.JoinHostPort](https://golang.org/pkg/net/#JoinHostPort) and [net.SplitHostPort](https://golang.org/pkg/net/#SplitHostPort) manipulate addresses in this form. + +For Unix domain sockets the address is the absolute path to the MySQL-Server-socket, e.g. `/var/run/mysqld/mysqld.sock` or `/tmp/mysql.sock`. + +#### Parameters +*Parameters are case-sensitive!* + +Notice that any of `true`, `TRUE`, `True` or `1` is accepted to stand for a true boolean value. Not surprisingly, false can be specified as any of: `false`, `FALSE`, `False` or `0`. + +##### `allowAllFiles` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`allowAllFiles=true` disables the file allowlist for `LOAD DATA LOCAL INFILE` and allows *all* files. +[*Might be insecure!*](https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-local) + +##### `allowCleartextPasswords` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`allowCleartextPasswords=true` allows using the [cleartext client side plugin](https://dev.mysql.com/doc/en/cleartext-pluggable-authentication.html) if required by an account, such as one defined with the [PAM authentication plugin](http://dev.mysql.com/doc/en/pam-authentication-plugin.html). Sending passwords in clear text may be a security problem in some configurations. To avoid problems if there is any possibility that the password would be intercepted, clients should connect to MySQL Server using a method that protects the password. Possibilities include [TLS / SSL](#tls), IPsec, or a private network. + + +##### `allowFallbackToPlaintext` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`allowFallbackToPlaintext=true` acts like a `--ssl-mode=PREFERRED` MySQL client as described in [Command Options for Connecting to the Server](https://dev.mysql.com/doc/refman/5.7/en/connection-options.html#option_general_ssl-mode) + +##### `allowNativePasswords` + +``` +Type: bool +Valid Values: true, false +Default: true +``` +`allowNativePasswords=false` disallows the usage of MySQL native password method. + +##### `allowOldPasswords` + +``` +Type: bool +Valid Values: true, false +Default: false +``` +`allowOldPasswords=true` allows the usage of the insecure old password method. This should be avoided, but is necessary in some cases. See also [the old_passwords wiki page](https://github.com/go-sql-driver/mysql/wiki/old_passwords). + +##### `charset` + +``` +Type: string +Valid Values: +Default: none +``` + +Sets the charset used for client-server interaction (`"SET NAMES "`). If multiple charsets are set (separated by a comma), the following charset is used if setting the charset fails. This enables for example support for `utf8mb4` ([introduced in MySQL 5.5.3](http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html)) with fallback to `utf8` for older servers (`charset=utf8mb4,utf8`). + +See also [Unicode Support](#unicode-support). + +##### `checkConnLiveness` + +``` +Type: bool +Valid Values: true, false +Default: true +``` + +On supported platforms connections retrieved from the connection pool are checked for liveness before using them. If the check fails, the respective connection is marked as bad and the query retried with another connection. +`checkConnLiveness=false` disables this liveness check of connections. + +##### `collation` + +``` +Type: string +Valid Values: +Default: utf8mb4_general_ci +``` + +Sets the collation used for client-server interaction on connection. In contrast to `charset`, `collation` does not issue additional queries. If the specified collation is unavailable on the target server, the connection will fail. + +A list of valid charsets for a server is retrievable with `SHOW COLLATION`. + +The default collation (`utf8mb4_general_ci`) is supported from MySQL 5.5. You should use an older collation (e.g. `utf8_general_ci`) for older MySQL. + +Collations for charset "ucs2", "utf16", "utf16le", and "utf32" can not be used ([ref](https://dev.mysql.com/doc/refman/5.7/en/charset-connection.html#charset-connection-impermissible-client-charset)). + +See also [Unicode Support](#unicode-support). + +##### `clientFoundRows` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`clientFoundRows=true` causes an UPDATE to return the number of matching rows instead of the number of rows changed. + +##### `columnsWithAlias` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +When `columnsWithAlias` is true, calls to `sql.Rows.Columns()` will return the table alias and the column name separated by a dot. For example: + +``` +SELECT u.id FROM users as u +``` + +will return `u.id` instead of just `id` if `columnsWithAlias=true`. + +##### `compress` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +Toggles zlib compression. false by default. + +##### `interpolateParams` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +If `interpolateParams` is true, placeholders (`?`) in calls to `db.Query()` and `db.Exec()` are interpolated into a single query string with given parameters. This reduces the number of roundtrips, since the driver has to prepare a statement, execute it with given parameters and close the statement again with `interpolateParams=false`. + +*This can not be used together with the multibyte encodings BIG5, CP932, GB2312, GBK or SJIS. These are rejected as they may [introduce a SQL injection vulnerability](http://stackoverflow.com/a/12118602/3430118)!* + +##### `loc` + +``` +Type: string +Valid Values: +Default: UTC +``` + +Sets the location for time.Time values (when using `parseTime=true`). *"Local"* sets the system's location. See [time.LoadLocation](https://golang.org/pkg/time/#LoadLocation) for details. + +Note that this sets the location for time.Time values but does not change MySQL's [time_zone setting](https://dev.mysql.com/doc/refman/5.5/en/time-zone-support.html). For that see the [time_zone system variable](#system-variables), which can also be set as a DSN parameter. + +Please keep in mind, that param values must be [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape)'ed. Alternatively you can manually replace the `/` with `%2F`. For example `US/Pacific` would be `loc=US%2FPacific`. + +##### `timeTruncate` + +``` +Type: duration +Default: 0 +``` + +[Truncate time values](https://pkg.go.dev/time#Duration.Truncate) in query arguments to the specified duration. The value must be a decimal number with a unit suffix (*"ns"*, *"us"*, *"ms"*, etc...), such as "1us", "1ms", or "10ns". + +> [!NOTE] +> `time.Time` arguments are sent with up to nanosecond precision, so a value from `time.Now()` usually has more fractional-second digits than a `DATETIME(N)` or `TIMESTAMP(N)` column stores. On MariaDB, comparing such a value against an indexed column can prevent an index range scan, turning it into a full index scan. Truncating to the column's precision (`1us` for `DATETIME(6)`) avoids this. Only arguments sent to the server are truncated; values read from the server are not affected. + +##### `maxAllowedPacket` +``` +Type: decimal number +Default: 64*1024*1024 +``` + +Max packet size allowed in bytes. The default value is 64 MiB and should be adjusted to match the server settings. `maxAllowedPacket=0` can be used to automatically fetch the `max_allowed_packet` variable from server *on every connection*. + +##### `multiStatements` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +Allow multiple statements in one query. This can be used to bach multiple queries. Use [Rows.NextResultSet()](https://pkg.go.dev/database/sql#Rows.NextResultSet) to get result of the second and subsequent queries. + +When `multiStatements` is used, `?` parameters must only be used in the first statement. [interpolateParams](#interpolateparams) can be used to avoid this limitation unless prepared statement is used explicitly. + +It's possible to access the last inserted ID and number of affected rows for multiple statements by using `sql.Conn.Raw()` and the `mysql.Result`. For example: + +```go +conn, _ := db.Conn(ctx) +conn.Raw(func(conn any) error { + ex := conn.(driver.Execer) + res, err := ex.Exec(` + UPDATE point SET x = 1 WHERE y = 2; + UPDATE point SET x = 2 WHERE y = 3; + `, nil) + // Both slices have 2 elements. + log.Print(res.(mysql.Result).AllRowsAffected()) + log.Print(res.(mysql.Result).AllLastInsertIds()) +}) +``` + +##### `parseTime` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + +`parseTime=true` changes the output type of `DATE` and `DATETIME` values to `time.Time` instead of `[]byte` / `string` +The date or datetime like `0000-00-00 00:00:00` is converted into zero value of `time.Time`. + + +##### `readTimeout` + +``` +Type: duration +Default: 0 +``` + +I/O read timeout. The value must be a decimal number with a unit suffix (*"ms"*, *"s"*, *"m"*, *"h"*), such as *"30s"*, *"0.5m"* or *"1m30s"*. + +##### `rejectReadOnly` + +``` +Type: bool +Valid Values: true, false +Default: false +``` + + +`rejectReadOnly=true` causes the driver to reject read-only connections. This +is for a possible race condition during an automatic failover, where the mysql +client gets connected to a read-only replica after the failover. + +Note that this should be a fairly rare case, as an automatic failover normally +happens when the primary is down, and the race condition shouldn't happen +unless it comes back up online as soon as the failover is kicked off. On the +other hand, when this happens, a MySQL application can get stuck on a +read-only connection until restarted. It is however fairly easy to reproduce, +for example, using a manual failover on AWS Aurora's MySQL-compatible cluster. + +If you are not relying on read-only transactions to reject writes that aren't +supposed to happen, setting this on some MySQL providers (such as AWS Aurora) +is safer for failovers. + +Note that ERROR 1290 can be returned for a `read-only` server and this option will +cause a retry for that error. However the same error number is used for some +other cases. You should ensure your application will never cause an ERROR 1290 +except for `read-only` mode when enabling this option. + + +##### `serverPubKey` + +``` +Type: string +Valid Values: +Default: none +``` + +Server public keys can be registered with [`mysql.RegisterServerPubKey`](https://godoc.org/github.com/go-sql-driver/mysql#RegisterServerPubKey), which can then be used by the assigned name in the DSN. +Public keys are used to transmit encrypted data, e.g. for authentication. +If the server's public key is known, it should be set manually to avoid expensive and potentially insecure transmissions of the public key from the server to the client each time it is required. + + +##### `timeout` + +``` +Type: duration +Default: OS default +``` + +Timeout for establishing connections, aka dial timeout. The value must be a decimal number with a unit suffix (*"ms"*, *"s"*, *"m"*, *"h"*), such as *"30s"*, *"0.5m"* or *"1m30s"*. + + +##### `tls` + +``` +Type: bool / string +Valid Values: true, false, skip-verify, preferred, +Default: false +``` + +`tls=true` enables TLS / SSL encrypted connection to the server. Use `skip-verify` if you want to use a self-signed or invalid certificate (server side) or use `preferred` to use TLS only when advertised by the server. This is similar to `skip-verify`, but additionally allows a fallback to a connection which is not encrypted. Neither `skip-verify` nor `preferred` add any reliable security. You can use a custom TLS config after registering it with [`mysql.RegisterTLSConfig`](https://godoc.org/github.com/go-sql-driver/mysql#RegisterTLSConfig). + + +##### `writeTimeout` + +``` +Type: duration +Default: 0 +``` + +I/O write timeout. The value must be a decimal number with a unit suffix (*"ms"*, *"s"*, *"m"*, *"h"*), such as *"30s"*, *"0.5m"* or *"1m30s"*. + +##### `connectionAttributes` + +``` +Type: comma-delimited string of user-defined "key:value" pairs +Valid Values: (:,:,...) +Default: none +``` + +[Connection attributes](https://dev.mysql.com/doc/refman/8.0/en/performance-schema-connection-attribute-tables.html) are key-value pairs that application programs can pass to the server at connect time. + +##### System Variables + +Any other parameters are interpreted as system variables: + * `=`: `SET =` + * `=`: `SET =` + * `=%27%27`: `SET =''` + +Rules: +* The values for string variables must be quoted with `'`. +* The values must also be [url.QueryEscape](http://golang.org/pkg/net/url/#QueryEscape)'ed! + (which implies values of string variables must be wrapped with `%27`). + +Examples: + * `autocommit=1`: `SET autocommit=1` + * [`time_zone=%27Europe%2FParis%27`](https://dev.mysql.com/doc/refman/5.5/en/time-zone-support.html): `SET time_zone='Europe/Paris'` + * [`transaction_isolation=%27REPEATABLE-READ%27`](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_transaction_isolation): `SET transaction_isolation='REPEATABLE-READ'` + + +#### Examples +``` +user@unix(/path/to/socket)/dbname +``` + +``` +root:pw@unix(/tmp/mysql.sock)/myDatabase?loc=Local +``` + +``` +user:password@tcp(localhost:5555)/dbname?tls=skip-verify&autocommit=true +``` + +Treat warnings as errors by setting the system variable [`sql_mode`](https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html): +``` +user:password@/dbname?sql_mode=TRADITIONAL +``` + +TCP via IPv6: +``` +user:password@tcp([de:ad:be:ef::ca:fe]:80)/dbname?timeout=90s&collation=utf8mb4_unicode_ci +``` + +TCP on a remote host, e.g. Amazon RDS: +``` +id:password@tcp(your-amazonaws-uri.com:3306)/dbname +``` + +Google Cloud SQL on App Engine: +``` +user:password@unix(/cloudsql/project-id:region-name:instance-name)/dbname +``` + +TCP using default port (3306) on localhost: +``` +user:password@tcp/dbname?charset=utf8mb4,utf8&sys_var=esc%40ped +``` + +Use the default protocol (tcp) and host (localhost:3306): +``` +user:password@/dbname +``` + +No Database preselected: +``` +user:password@/ +``` + + +### Connection pool and timeouts +The connection pool is managed by Go's database/sql package. For details on how to configure the size of the pool and how long connections stay in the pool see `*DB.SetMaxOpenConns`, `*DB.SetMaxIdleConns`, and `*DB.SetConnMaxLifetime` in the [database/sql documentation](https://golang.org/pkg/database/sql/). The read, write, and dial timeouts for each individual connection are configured with the DSN parameters [`readTimeout`](#readtimeout), [`writeTimeout`](#writetimeout), and [`timeout`](#timeout), respectively. + +## `ColumnType` Support +This driver supports the [`ColumnType` interface](https://golang.org/pkg/database/sql/#ColumnType) introduced in Go 1.8, with the exception of [`ColumnType.Length()`](https://golang.org/pkg/database/sql/#ColumnType.Length), which is currently not supported. All Unsigned database type names will be returned `UNSIGNED ` with `INT`, `TINYINT`, `SMALLINT`, `MEDIUMINT`, `BIGINT`. + +## `context.Context` Support +Go 1.8 added `database/sql` support for `context.Context`. This driver supports query timeouts and cancellation via contexts. +See [context support in the database/sql package](https://golang.org/doc/go1.8#database_sql) for more details. + +> [!IMPORTANT] +> The `QueryContext`, `ExecContext`, etc. variants provided by `database/sql` will cause the connection to be closed if the provided context is cancelled or timed out before the result is received by the driver. + + +### `LOAD DATA LOCAL INFILE` support +For this feature you need direct access to the package. Therefore you must change the import path (no `_`): +```go +import "github.com/go-sql-driver/mysql" +``` + +Files must be explicitly allowed by registering them with `mysql.RegisterLocalFile(filepath)` (recommended) or the allowlist check must be deactivated by using the DSN parameter `allowAllFiles=true` ([*Might be insecure!*](https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-local)). + +To use a `io.Reader` a handler function must be registered with `mysql.RegisterReaderHandler(name, handler)` which returns a `io.Reader` or `io.ReadCloser`. The Reader is available with the filepath `Reader::` then. Choose different names for different handlers and `DeregisterReaderHandler` when you don't need it anymore. + +See the [godoc of Go-MySQL-Driver](https://godoc.org/github.com/go-sql-driver/mysql "golang mysql driver documentation") for details. + + +### `time.Time` support +The default internal output type of MySQL `DATE` and `DATETIME` values is `[]byte` which allows you to scan the value into a `[]byte`, `string` or `sql.RawBytes` variable in your program. + +However, many want to scan MySQL `DATE` and `DATETIME` values into `time.Time` variables, which is the logical equivalent in Go to `DATE` and `DATETIME` in MySQL. You can do that by changing the internal output type from `[]byte` to `time.Time` with the DSN parameter `parseTime=true`. You can set the default [`time.Time` location](https://golang.org/pkg/time/#Location) with the `loc` DSN parameter. + +**Caution:** As of Go 1.1, this makes `time.Time` the only variable type you can scan `DATE` and `DATETIME` values into. This breaks for example [`sql.RawBytes` support](https://github.com/go-sql-driver/mysql/wiki/Examples#rawbytes). + + +### Unicode support +Since version 1.5 Go-MySQL-Driver automatically uses the collation ` utf8mb4_general_ci` by default. + +Other charsets / collations can be set using the [`charset`](#charset) or [`collation`](#collation) DSN parameter. + +- When only the `charset` is specified, the `SET NAMES ` query is sent and the server's default collation is used. +- When both the `charset` and `collation` are specified, the `SET NAMES COLLATE ` query is sent. +- When only the `collation` is specified, the collation is specified in the protocol handshake and the `SET NAMES` query is not sent. This can save one roundtrip, but note that the server may ignore the specified collation silently and use the server's default charset/collation instead. + +See http://dev.mysql.com/doc/refman/8.0/en/charset-unicode.html for more details on MySQL's Unicode support. + +## Testing / Development +To run the driver tests you may need to adjust the configuration. See the [Testing Wiki-Page](https://github.com/go-sql-driver/mysql/wiki/Testing "Testing") for details. + +Go-MySQL-Driver is not feature-complete yet. Your help is very appreciated. +If you want to contribute, you can work on an [open issue](https://github.com/go-sql-driver/mysql/issues?state=open) or review a [pull request](https://github.com/go-sql-driver/mysql/pulls). + +See the [Contribution Guidelines](https://github.com/go-sql-driver/mysql/blob/master/.github/CONTRIBUTING.md) for details. + +--------------------------------------- + +## License +Go-MySQL-Driver is licensed under the [Mozilla Public License Version 2.0](https://raw.github.com/go-sql-driver/mysql/master/LICENSE) + +Mozilla summarizes the license scope as follows: +> MPL: The copyleft applies to any files containing MPLed code. + + +That means: + * You can **use** the **unchanged** source code both in private and commercially. + * When distributing, you **must publish** the source code of any **changed files** licensed under the MPL 2.0 under a) the MPL 2.0 itself or b) a compatible license (e.g. GPL 3.0 or Apache License 2.0). + * You **needn't publish** the source code of your library as long as the files licensed under the MPL 2.0 are **unchanged**. + +Please read the [MPL 2.0 FAQ](https://www.mozilla.org/en-US/MPL/2.0/FAQ/) if you have further questions regarding the license. + +You can read the full terms here: [LICENSE](https://raw.github.com/go-sql-driver/mysql/master/LICENSE). + +![Go Gopher and MySQL Dolphin](https://raw.github.com/wiki/go-sql-driver/mysql/go-mysql-driver_m.jpg "Golang Gopher transporting the MySQL Dolphin in a wheelbarrow") diff --git a/vendor/github.com/go-sql-driver/mysql/auth.go b/vendor/github.com/go-sql-driver/mysql/auth.go new file mode 100644 index 0000000000..610044fc16 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/auth.go @@ -0,0 +1,484 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2018 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "crypto/rand" + "crypto/rsa" + "crypto/sha1" + "crypto/sha256" + "crypto/sha512" + "crypto/x509" + "encoding/pem" + "fmt" + "sync" + + "filippo.io/edwards25519" +) + +// server pub keys registry +var ( + serverPubKeyLock sync.RWMutex + serverPubKeyRegistry map[string]*rsa.PublicKey +) + +// RegisterServerPubKey registers a server RSA public key which can be used to +// send data in a secure manner to the server without receiving the public key +// in a potentially insecure way from the server first. +// Registered keys can afterwards be used adding serverPubKey= to the DSN. +// +// Note: The provided rsa.PublicKey instance is exclusively owned by the driver +// after registering it and may not be modified. +// +// data, err := os.ReadFile("mykey.pem") +// if err != nil { +// log.Fatal(err) +// } +// +// block, _ := pem.Decode(data) +// if block == nil || block.Type != "PUBLIC KEY" { +// log.Fatal("failed to decode PEM block containing public key") +// } +// +// pub, err := x509.ParsePKIXPublicKey(block.Bytes) +// if err != nil { +// log.Fatal(err) +// } +// +// if rsaPubKey, ok := pub.(*rsa.PublicKey); ok { +// mysql.RegisterServerPubKey("mykey", rsaPubKey) +// } else { +// log.Fatal("not a RSA public key") +// } +func RegisterServerPubKey(name string, pubKey *rsa.PublicKey) { + serverPubKeyLock.Lock() + if serverPubKeyRegistry == nil { + serverPubKeyRegistry = make(map[string]*rsa.PublicKey) + } + + serverPubKeyRegistry[name] = pubKey + serverPubKeyLock.Unlock() +} + +// DeregisterServerPubKey removes the public key registered with the given name. +func DeregisterServerPubKey(name string) { + serverPubKeyLock.Lock() + if serverPubKeyRegistry != nil { + delete(serverPubKeyRegistry, name) + } + serverPubKeyLock.Unlock() +} + +func getServerPubKey(name string) (pubKey *rsa.PublicKey) { + serverPubKeyLock.RLock() + if v, ok := serverPubKeyRegistry[name]; ok { + pubKey = v + } + serverPubKeyLock.RUnlock() + return +} + +// Hash password using pre 4.1 (old password) method +// https://github.com/atcurtis/mariadb/blob/master/mysys/my_rnd.c +type myRnd struct { + seed1, seed2 uint32 +} + +const myRndMaxVal = 0x3FFFFFFF + +// Pseudo random number generator +func newMyRnd(seed1, seed2 uint32) *myRnd { + return &myRnd{ + seed1: seed1 % myRndMaxVal, + seed2: seed2 % myRndMaxVal, + } +} + +// Tested to be equivalent to MariaDB's floating point variant +// http://play.golang.org/p/QHvhd4qved +// http://play.golang.org/p/RG0q4ElWDx +func (r *myRnd) NextByte() byte { + r.seed1 = (r.seed1*3 + r.seed2) % myRndMaxVal + r.seed2 = (r.seed1 + r.seed2 + 33) % myRndMaxVal + + return byte(uint64(r.seed1) * 31 / myRndMaxVal) +} + +// Generate binary hash from byte string using insecure pre 4.1 method +func pwHash(password []byte) (result [2]uint32) { + var add uint32 = 7 + var tmp uint32 + + result[0] = 1345345333 + result[1] = 0x12345671 + + for _, c := range password { + // skip spaces and tabs in password + if c == ' ' || c == '\t' { + continue + } + + tmp = uint32(c) + result[0] ^= (((result[0] & 63) + add) * tmp) + (result[0] << 8) + result[1] += (result[1] << 8) ^ result[0] + add += tmp + } + + // Remove sign bit (1<<31)-1) + result[0] &= 0x7FFFFFFF + result[1] &= 0x7FFFFFFF + + return +} + +// Hash password using insecure pre 4.1 method +func scrambleOldPassword(scramble []byte, password string) []byte { + scramble = scramble[:8] + + hashPw := pwHash([]byte(password)) + hashSc := pwHash(scramble) + + r := newMyRnd(hashPw[0]^hashSc[0], hashPw[1]^hashSc[1]) + + var out [8]byte + for i := range out { + out[i] = r.NextByte() + 64 + } + + mask := r.NextByte() + for i := range out { + out[i] ^= mask + } + + return out[:] +} + +// Hash password using 4.1+ method (SHA1) +func scramblePassword(scramble []byte, password string) []byte { + if len(password) == 0 { + return nil + } + + // stage1Hash = SHA1(password) + crypt := sha1.New() + crypt.Write([]byte(password)) + stage1 := crypt.Sum(nil) + + // scrambleHash = SHA1(scramble + SHA1(stage1Hash)) + // inner Hash + crypt.Reset() + crypt.Write(stage1) + hash := crypt.Sum(nil) + + // outer Hash + crypt.Reset() + crypt.Write(scramble) + crypt.Write(hash) + scramble = crypt.Sum(nil) + + // token = scrambleHash XOR stage1Hash + for i := range scramble { + scramble[i] ^= stage1[i] + } + return scramble +} + +// Hash password using MySQL 8+ method (SHA256) +func scrambleSHA256Password(scramble []byte, password string) []byte { + if len(password) == 0 { + return nil + } + + // XOR(SHA256(password), SHA256(SHA256(SHA256(password)), scramble)) + + crypt := sha256.New() + crypt.Write([]byte(password)) + message1 := crypt.Sum(nil) + + crypt.Reset() + crypt.Write(message1) + message1Hash := crypt.Sum(nil) + + crypt.Reset() + crypt.Write(message1Hash) + crypt.Write(scramble) + message2 := crypt.Sum(nil) + + for i := range message1 { + message1[i] ^= message2[i] + } + + return message1 +} + +func encryptPassword(password string, seed []byte, pub *rsa.PublicKey) ([]byte, error) { + plain := make([]byte, len(password)+1) + copy(plain, password) + for i := range plain { + j := i % len(seed) + plain[i] ^= seed[j] + } + sha1 := sha1.New() + return rsa.EncryptOAEP(sha1, rand.Reader, pub, plain, nil) +} + +// authEd25519 does ed25519 authentication used by MariaDB. +func authEd25519(scramble []byte, password string) ([]byte, error) { + // Derived from https://github.com/MariaDB/server/blob/d8e6bb00888b1f82c031938f4c8ac5d97f6874c3/plugin/auth_ed25519/ref10/sign.c + // Code style is from https://cs.opensource.google/go/go/+/refs/tags/go1.21.5:src/crypto/ed25519/ed25519.go;l=207 + h := sha512.Sum512([]byte(password)) + + s, err := edwards25519.NewScalar().SetBytesWithClamping(h[:32]) + if err != nil { + return nil, err + } + A := (&edwards25519.Point{}).ScalarBaseMult(s) + + mh := sha512.New() + mh.Write(h[32:]) + mh.Write(scramble) + messageDigest := mh.Sum(nil) + r, err := edwards25519.NewScalar().SetUniformBytes(messageDigest) + if err != nil { + return nil, err + } + + R := (&edwards25519.Point{}).ScalarBaseMult(r) + + kh := sha512.New() + kh.Write(R.Bytes()) + kh.Write(A.Bytes()) + kh.Write(scramble) + hramDigest := kh.Sum(nil) + k, err := edwards25519.NewScalar().SetUniformBytes(hramDigest) + if err != nil { + return nil, err + } + + S := k.MultiplyAdd(k, s, r) + + return append(R.Bytes(), S.Bytes()...), nil +} + +func (mc *mysqlConn) sendEncryptedPassword(seed []byte, pub *rsa.PublicKey) error { + enc, err := encryptPassword(mc.cfg.Passwd, seed, pub) + if err != nil { + return err + } + return mc.writeAuthSwitchPacket(enc) +} + +func (mc *mysqlConn) auth(authData []byte, plugin string) ([]byte, error) { + switch plugin { + case "caching_sha2_password": + authResp := scrambleSHA256Password(authData, mc.cfg.Passwd) + return authResp, nil + + case "mysql_old_password": + if !mc.cfg.AllowOldPasswords { + return nil, ErrOldPassword + } + if len(mc.cfg.Passwd) == 0 { + return nil, nil + } + // Note: there are edge cases where this should work but doesn't; + // this is currently "wontfix": + // https://github.com/go-sql-driver/mysql/issues/184 + authResp := append(scrambleOldPassword(authData[:8], mc.cfg.Passwd), 0) + return authResp, nil + + case "mysql_clear_password": + if !mc.cfg.AllowCleartextPasswords { + return nil, ErrCleartextPassword + } + // http://dev.mysql.com/doc/refman/5.7/en/cleartext-authentication-plugin.html + // http://dev.mysql.com/doc/refman/5.7/en/pam-authentication-plugin.html + return append([]byte(mc.cfg.Passwd), 0), nil + + case "mysql_native_password": + if !mc.cfg.AllowNativePasswords { + return nil, ErrNativePassword + } + // https://dev.mysql.com/doc/dev/mysql-server/8.4.5/page_protocol_connection_phase_authentication_methods_native_password_authentication.html + // Native password authentication only need and will need 20-byte challenge. + authResp := scramblePassword(authData[:20], mc.cfg.Passwd) + return authResp, nil + + case "sha256_password": + if len(mc.cfg.Passwd) == 0 { + return []byte{0}, nil + } + // unlike caching_sha2_password, sha256_password does not accept + // cleartext password on unix transport. + if mc.cfg.TLS != nil { + // write cleartext auth packet + return append([]byte(mc.cfg.Passwd), 0), nil + } + + pubKey := mc.cfg.pubKey + if pubKey == nil { + // request public key from server + return []byte{1}, nil + } + + // encrypted password + enc, err := encryptPassword(mc.cfg.Passwd, authData, pubKey) + return enc, err + + case "client_ed25519": + if len(authData) != 32 { + return nil, ErrMalformPkt + } + return authEd25519(authData, mc.cfg.Passwd) + + default: + mc.log("unknown auth plugin:", plugin) + return nil, ErrUnknownPlugin + } +} + +func (mc *mysqlConn) handleAuthResult(oldAuthData []byte, plugin string) error { + // Read Result Packet + authData, newPlugin, err := mc.readAuthResult() + if err != nil { + return err + } + + // handle auth plugin switch, if requested + if newPlugin != "" { + // If CLIENT_PLUGIN_AUTH capability is not supported, no new cipher is + // sent and we have to keep using the cipher sent in the init packet. + if authData == nil { + authData = oldAuthData + } else { + // copy data from read buffer to owned slice + copy(oldAuthData, authData) + } + + plugin = newPlugin + + authResp, err := mc.auth(authData, plugin) + if err != nil { + return err + } + if err = mc.writeAuthSwitchPacket(authResp); err != nil { + return err + } + + // Read Result Packet + authData, newPlugin, err = mc.readAuthResult() + if err != nil { + return err + } + + // Do not allow to change the auth plugin more than once + if newPlugin != "" { + return ErrMalformPkt + } + } + + switch plugin { + + // https://dev.mysql.com/blog-archive/preparing-your-community-connector-for-mysql-8-part-2-sha256/ + case "caching_sha2_password": + switch len(authData) { + case 0: + return nil // auth successful + case 1: + switch authData[0] { + case cachingSha2PasswordFastAuthSuccess: + if err = mc.resultUnchanged().readResultOK(); err == nil { + return nil // auth successful + } + + case cachingSha2PasswordPerformFullAuthentication: + if mc.cfg.TLS != nil || mc.cfg.Net == "unix" { + // write cleartext auth packet + err = mc.writeAuthSwitchPacket(append([]byte(mc.cfg.Passwd), 0)) + if err != nil { + return err + } + } else { + pubKey := mc.cfg.pubKey + if pubKey == nil { + // request public key from server + data, err := mc.buf.takeSmallBuffer(4 + 1) + if err != nil { + return err + } + data[4] = cachingSha2PasswordRequestPublicKey + err = mc.writePacket(data) + if err != nil { + return err + } + + if data, err = mc.readPacket(); err != nil { + return err + } + + if data[0] != iAuthMoreData { + return fmt.Errorf("unexpected resp from server for caching_sha2_password, perform full authentication") + } + + // parse public key + block, rest := pem.Decode(data[1:]) + if block == nil { + return fmt.Errorf("no pem data found, data: %s", rest) + } + pkix, err := x509.ParsePKIXPublicKey(block.Bytes) + if err != nil { + return err + } + pubKey = pkix.(*rsa.PublicKey) + } + + // send encrypted password + err = mc.sendEncryptedPassword(oldAuthData, pubKey) + if err != nil { + return err + } + } + return mc.resultUnchanged().readResultOK() + + default: + return ErrMalformPkt + } + default: + return ErrMalformPkt + } + + case "sha256_password": + switch len(authData) { + case 0: + return nil // auth successful + default: + block, _ := pem.Decode(authData) + if block == nil { + return fmt.Errorf("no Pem data found, data: %s", authData) + } + + pub, err := x509.ParsePKIXPublicKey(block.Bytes) + if err != nil { + return err + } + + // send encrypted password + err = mc.sendEncryptedPassword(oldAuthData, pub.(*rsa.PublicKey)) + if err != nil { + return err + } + return mc.resultUnchanged().readResultOK() + } + + default: + return nil // auth successful + } + + return err +} diff --git a/vendor/github.com/go-sql-driver/mysql/buffer.go b/vendor/github.com/go-sql-driver/mysql/buffer.go new file mode 100644 index 0000000000..f895e87b3c --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/buffer.go @@ -0,0 +1,149 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "io" +) + +const defaultBufSize = 4096 +const maxCachedBufSize = 256 * 1024 + +// readerFunc is a function that compatible with io.Reader. +// We use this function type instead of io.Reader because we want to +// just pass mc.readWithTimeout. +type readerFunc func([]byte) (int, error) + +// A buffer which is used for both reading and writing. +// This is possible since communication on each connection is synchronous. +// In other words, we can't write and read simultaneously on the same connection. +// The buffer is similar to bufio.Reader / Writer but zero-copy-ish +// Also highly optimized for this particular use case. +type buffer struct { + buf []byte // read buffer. + cachedBuf []byte // buffer that will be reused. len(cachedBuf) <= maxCachedBufSize. +} + +// newBuffer allocates and returns a new buffer. +func newBuffer() buffer { + return buffer{ + cachedBuf: make([]byte, defaultBufSize), + } +} + +// busy returns true if the read buffer is not empty. +func (b *buffer) busy() bool { + return len(b.buf) > 0 +} + +// len returns how many bytes in the read buffer. +func (b *buffer) len() int { + return len(b.buf) +} + +// fill reads into the read buffer until at least _need_ bytes are in it. +func (b *buffer) fill(need int, r readerFunc) error { + // we'll move the contents of the current buffer to dest before filling it. + dest := b.cachedBuf + + // grow buffer if necessary to fit the whole packet. + if need > len(dest) { + // Round up to the next multiple of the default size + dest = make([]byte, ((need/defaultBufSize)+1)*defaultBufSize) + + // if the allocated buffer is not too large, move it to backing storage + // to prevent extra allocations on applications that perform large reads + if len(dest) <= maxCachedBufSize { + b.cachedBuf = dest + } + } + + // move the existing data to the start of the buffer. + n := len(b.buf) + copy(dest[:n], b.buf) + + for { + nn, err := r(dest[n:]) + n += nn + + if err == nil && n < need { + continue + } + + b.buf = dest[:n] + + if err == io.EOF { + if n < need { + err = io.ErrUnexpectedEOF + } else { + err = nil + } + } + return err + } +} + +// returns next N bytes from buffer. +// The returned slice is only guaranteed to be valid until the next read +func (b *buffer) readNext(need int) []byte { + data := b.buf[:need:need] + b.buf = b.buf[need:] + return data +} + +// takeBuffer returns a buffer with the requested size. +// If possible, a slice from the existing buffer is returned. +// Otherwise a bigger buffer is made. +// Only one buffer (total) can be used at a time. +func (b *buffer) takeBuffer(length int) ([]byte, error) { + if b.busy() { + return nil, ErrBusyBuffer + } + + // test (cheap) general case first + if length <= len(b.cachedBuf) { + return b.cachedBuf[:length], nil + } + + if length < maxCachedBufSize { + b.cachedBuf = make([]byte, length) + return b.cachedBuf, nil + } + + // buffer is larger than we want to store. + return make([]byte, length), nil +} + +// takeSmallBuffer is shortcut which can be used if length is +// known to be smaller than defaultBufSize. +// Only one buffer (total) can be used at a time. +func (b *buffer) takeSmallBuffer(length int) ([]byte, error) { + if b.busy() { + return nil, ErrBusyBuffer + } + return b.cachedBuf[:length], nil +} + +// takeCompleteBuffer returns the complete existing buffer. +// This can be used if the necessary buffer size is unknown. +// cap and len of the returned buffer will be equal. +// Only one buffer (total) can be used at a time. +func (b *buffer) takeCompleteBuffer() ([]byte, error) { + if b.busy() { + return nil, ErrBusyBuffer + } + return b.cachedBuf, nil +} + +// store stores buf, an updated buffer, if its suitable to do so. +func (b *buffer) store(buf []byte) { + if cap(buf) <= maxCachedBufSize && cap(buf) > cap(b.cachedBuf) { + b.cachedBuf = buf[:cap(buf)] + } +} diff --git a/vendor/github.com/go-sql-driver/mysql/collations.go b/vendor/github.com/go-sql-driver/mysql/collations.go new file mode 100644 index 0000000000..29b1aa43f2 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/collations.go @@ -0,0 +1,266 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2014 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +const defaultCollationID = 45 // utf8mb4_general_ci +const binaryCollationID = 63 + +// A list of available collations mapped to the internal ID. +// To update this map use the following MySQL query: +// +// SELECT COLLATION_NAME, ID FROM information_schema.COLLATIONS WHERE ID<256 ORDER BY ID +// +// Handshake packet have only 1 byte for collation_id. So we can't use collations with ID > 255. +// +// ucs2, utf16, and utf32 can't be used for connection charset. +// https://dev.mysql.com/doc/refman/5.7/en/charset-connection.html#charset-connection-impermissible-client-charset +// They are commented out to reduce this map. +var collations = map[string]byte{ + "big5_chinese_ci": 1, + "latin2_czech_cs": 2, + "dec8_swedish_ci": 3, + "cp850_general_ci": 4, + "latin1_german1_ci": 5, + "hp8_english_ci": 6, + "koi8r_general_ci": 7, + "latin1_swedish_ci": 8, + "latin2_general_ci": 9, + "swe7_swedish_ci": 10, + "ascii_general_ci": 11, + "ujis_japanese_ci": 12, + "sjis_japanese_ci": 13, + "cp1251_bulgarian_ci": 14, + "latin1_danish_ci": 15, + "hebrew_general_ci": 16, + "tis620_thai_ci": 18, + "euckr_korean_ci": 19, + "latin7_estonian_cs": 20, + "latin2_hungarian_ci": 21, + "koi8u_general_ci": 22, + "cp1251_ukrainian_ci": 23, + "gb2312_chinese_ci": 24, + "greek_general_ci": 25, + "cp1250_general_ci": 26, + "latin2_croatian_ci": 27, + "gbk_chinese_ci": 28, + "cp1257_lithuanian_ci": 29, + "latin5_turkish_ci": 30, + "latin1_german2_ci": 31, + "armscii8_general_ci": 32, + "utf8_general_ci": 33, + "cp1250_czech_cs": 34, + //"ucs2_general_ci": 35, + "cp866_general_ci": 36, + "keybcs2_general_ci": 37, + "macce_general_ci": 38, + "macroman_general_ci": 39, + "cp852_general_ci": 40, + "latin7_general_ci": 41, + "latin7_general_cs": 42, + "macce_bin": 43, + "cp1250_croatian_ci": 44, + "utf8mb4_general_ci": 45, + "utf8mb4_bin": 46, + "latin1_bin": 47, + "latin1_general_ci": 48, + "latin1_general_cs": 49, + "cp1251_bin": 50, + "cp1251_general_ci": 51, + "cp1251_general_cs": 52, + "macroman_bin": 53, + //"utf16_general_ci": 54, + //"utf16_bin": 55, + //"utf16le_general_ci": 56, + "cp1256_general_ci": 57, + "cp1257_bin": 58, + "cp1257_general_ci": 59, + //"utf32_general_ci": 60, + //"utf32_bin": 61, + //"utf16le_bin": 62, + "binary": 63, + "armscii8_bin": 64, + "ascii_bin": 65, + "cp1250_bin": 66, + "cp1256_bin": 67, + "cp866_bin": 68, + "dec8_bin": 69, + "greek_bin": 70, + "hebrew_bin": 71, + "hp8_bin": 72, + "keybcs2_bin": 73, + "koi8r_bin": 74, + "koi8u_bin": 75, + "utf8_tolower_ci": 76, + "latin2_bin": 77, + "latin5_bin": 78, + "latin7_bin": 79, + "cp850_bin": 80, + "cp852_bin": 81, + "swe7_bin": 82, + "utf8_bin": 83, + "big5_bin": 84, + "euckr_bin": 85, + "gb2312_bin": 86, + "gbk_bin": 87, + "sjis_bin": 88, + "tis620_bin": 89, + //"ucs2_bin": 90, + "ujis_bin": 91, + "geostd8_general_ci": 92, + "geostd8_bin": 93, + "latin1_spanish_ci": 94, + "cp932_japanese_ci": 95, + "cp932_bin": 96, + "eucjpms_japanese_ci": 97, + "eucjpms_bin": 98, + "cp1250_polish_ci": 99, + //"utf16_unicode_ci": 101, + //"utf16_icelandic_ci": 102, + //"utf16_latvian_ci": 103, + //"utf16_romanian_ci": 104, + //"utf16_slovenian_ci": 105, + //"utf16_polish_ci": 106, + //"utf16_estonian_ci": 107, + //"utf16_spanish_ci": 108, + //"utf16_swedish_ci": 109, + //"utf16_turkish_ci": 110, + //"utf16_czech_ci": 111, + //"utf16_danish_ci": 112, + //"utf16_lithuanian_ci": 113, + //"utf16_slovak_ci": 114, + //"utf16_spanish2_ci": 115, + //"utf16_roman_ci": 116, + //"utf16_persian_ci": 117, + //"utf16_esperanto_ci": 118, + //"utf16_hungarian_ci": 119, + //"utf16_sinhala_ci": 120, + //"utf16_german2_ci": 121, + //"utf16_croatian_ci": 122, + //"utf16_unicode_520_ci": 123, + //"utf16_vietnamese_ci": 124, + //"ucs2_unicode_ci": 128, + //"ucs2_icelandic_ci": 129, + //"ucs2_latvian_ci": 130, + //"ucs2_romanian_ci": 131, + //"ucs2_slovenian_ci": 132, + //"ucs2_polish_ci": 133, + //"ucs2_estonian_ci": 134, + //"ucs2_spanish_ci": 135, + //"ucs2_swedish_ci": 136, + //"ucs2_turkish_ci": 137, + //"ucs2_czech_ci": 138, + //"ucs2_danish_ci": 139, + //"ucs2_lithuanian_ci": 140, + //"ucs2_slovak_ci": 141, + //"ucs2_spanish2_ci": 142, + //"ucs2_roman_ci": 143, + //"ucs2_persian_ci": 144, + //"ucs2_esperanto_ci": 145, + //"ucs2_hungarian_ci": 146, + //"ucs2_sinhala_ci": 147, + //"ucs2_german2_ci": 148, + //"ucs2_croatian_ci": 149, + //"ucs2_unicode_520_ci": 150, + //"ucs2_vietnamese_ci": 151, + //"ucs2_general_mysql500_ci": 159, + //"utf32_unicode_ci": 160, + //"utf32_icelandic_ci": 161, + //"utf32_latvian_ci": 162, + //"utf32_romanian_ci": 163, + //"utf32_slovenian_ci": 164, + //"utf32_polish_ci": 165, + //"utf32_estonian_ci": 166, + //"utf32_spanish_ci": 167, + //"utf32_swedish_ci": 168, + //"utf32_turkish_ci": 169, + //"utf32_czech_ci": 170, + //"utf32_danish_ci": 171, + //"utf32_lithuanian_ci": 172, + //"utf32_slovak_ci": 173, + //"utf32_spanish2_ci": 174, + //"utf32_roman_ci": 175, + //"utf32_persian_ci": 176, + //"utf32_esperanto_ci": 177, + //"utf32_hungarian_ci": 178, + //"utf32_sinhala_ci": 179, + //"utf32_german2_ci": 180, + //"utf32_croatian_ci": 181, + //"utf32_unicode_520_ci": 182, + //"utf32_vietnamese_ci": 183, + "utf8_unicode_ci": 192, + "utf8_icelandic_ci": 193, + "utf8_latvian_ci": 194, + "utf8_romanian_ci": 195, + "utf8_slovenian_ci": 196, + "utf8_polish_ci": 197, + "utf8_estonian_ci": 198, + "utf8_spanish_ci": 199, + "utf8_swedish_ci": 200, + "utf8_turkish_ci": 201, + "utf8_czech_ci": 202, + "utf8_danish_ci": 203, + "utf8_lithuanian_ci": 204, + "utf8_slovak_ci": 205, + "utf8_spanish2_ci": 206, + "utf8_roman_ci": 207, + "utf8_persian_ci": 208, + "utf8_esperanto_ci": 209, + "utf8_hungarian_ci": 210, + "utf8_sinhala_ci": 211, + "utf8_german2_ci": 212, + "utf8_croatian_ci": 213, + "utf8_unicode_520_ci": 214, + "utf8_vietnamese_ci": 215, + "utf8_general_mysql500_ci": 223, + "utf8mb4_unicode_ci": 224, + "utf8mb4_icelandic_ci": 225, + "utf8mb4_latvian_ci": 226, + "utf8mb4_romanian_ci": 227, + "utf8mb4_slovenian_ci": 228, + "utf8mb4_polish_ci": 229, + "utf8mb4_estonian_ci": 230, + "utf8mb4_spanish_ci": 231, + "utf8mb4_swedish_ci": 232, + "utf8mb4_turkish_ci": 233, + "utf8mb4_czech_ci": 234, + "utf8mb4_danish_ci": 235, + "utf8mb4_lithuanian_ci": 236, + "utf8mb4_slovak_ci": 237, + "utf8mb4_spanish2_ci": 238, + "utf8mb4_roman_ci": 239, + "utf8mb4_persian_ci": 240, + "utf8mb4_esperanto_ci": 241, + "utf8mb4_hungarian_ci": 242, + "utf8mb4_sinhala_ci": 243, + "utf8mb4_german2_ci": 244, + "utf8mb4_croatian_ci": 245, + "utf8mb4_unicode_520_ci": 246, + "utf8mb4_vietnamese_ci": 247, + "gb18030_chinese_ci": 248, + "gb18030_bin": 249, + "gb18030_unicode_520_ci": 250, + "utf8mb4_0900_ai_ci": 255, +} + +// A denylist of collations which is unsafe to interpolate parameters. +// These multibyte encodings may contains 0x5c (`\`) in their trailing bytes. +var unsafeCollations = map[string]bool{ + "big5_chinese_ci": true, + "sjis_japanese_ci": true, + "gbk_chinese_ci": true, + "big5_bin": true, + "gb2312_bin": true, + "gbk_bin": true, + "sjis_bin": true, + "cp932_japanese_ci": true, + "cp932_bin": true, + "gb18030_chinese_ci": true, + "gb18030_bin": true, + "gb18030_unicode_520_ci": true, +} diff --git a/vendor/github.com/go-sql-driver/mysql/compress.go b/vendor/github.com/go-sql-driver/mysql/compress.go new file mode 100644 index 0000000000..38bfa000e8 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/compress.go @@ -0,0 +1,213 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2024 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "compress/zlib" + "fmt" + "io" + "sync" +) + +var ( + zrPool *sync.Pool // Do not use directly. Use zDecompress() instead. + zwPool *sync.Pool // Do not use directly. Use zCompress() instead. +) + +func init() { + zrPool = &sync.Pool{ + New: func() any { return nil }, + } + zwPool = &sync.Pool{ + New: func() any { + zw, err := zlib.NewWriterLevel(new(bytes.Buffer), 2) + if err != nil { + panic(err) // compress/zlib return non-nil error only if level is invalid + } + return zw + }, + } +} + +func zDecompress(src []byte, dst *bytes.Buffer) (int, error) { + br := bytes.NewReader(src) + var zr io.ReadCloser + var err error + + if a := zrPool.Get(); a == nil { + if zr, err = zlib.NewReader(br); err != nil { + return 0, err + } + } else { + zr = a.(io.ReadCloser) + if err := zr.(zlib.Resetter).Reset(br, nil); err != nil { + return 0, err + } + } + + n, _ := dst.ReadFrom(zr) // ignore err because zr.Close() will return it again. + err = zr.Close() // zr.Close() may return chuecksum error. + zrPool.Put(zr) + return int(n), err +} + +func zCompress(src []byte, dst io.Writer) error { + zw := zwPool.Get().(*zlib.Writer) + zw.Reset(dst) + if _, err := zw.Write(src); err != nil { + return err + } + err := zw.Close() + zwPool.Put(zw) + return err +} + +type compIO struct { + mc *mysqlConn + buff bytes.Buffer +} + +func newCompIO(mc *mysqlConn) *compIO { + return &compIO{ + mc: mc, + } +} + +func (c *compIO) reset() { + c.buff.Reset() +} + +func (c *compIO) readNext(need int) ([]byte, error) { + for c.buff.Len() < need { + if err := c.readCompressedPacket(); err != nil { + return nil, err + } + } + data := c.buff.Next(need) + return data[:need:need], nil // prevent caller writes into c.buff +} + +func (c *compIO) readCompressedPacket() error { + header, err := c.mc.readNext(7) + if err != nil { + return err + } + _ = header[6] // bounds check hint to compiler; guaranteed by readNext + + // compressed header structure + comprLength := getUint24(header[0:3]) + compressionSequence := header[3] + uncompressedLength := getUint24(header[4:7]) + if debug { + fmt.Printf("uncompress cmplen=%v uncomplen=%v pkt_cmp_seq=%v expected_cmp_seq=%v\n", + comprLength, uncompressedLength, compressionSequence, c.mc.sequence) + } + // Do not return ErrPktSync here. + // Server may return error packet (e.g. 1153 Got a packet bigger than 'max_allowed_packet' bytes) + // before receiving all packets from client. In this case, seqnr is younger than expected. + // NOTE: Both of mariadbclient and mysqlclient do not check seqnr. Only server checks it. + if debug && compressionSequence != c.mc.compressSequence { + fmt.Printf("WARN: unexpected cmpress seq nr: expected %v, got %v", + c.mc.compressSequence, compressionSequence) + } + c.mc.compressSequence = compressionSequence + 1 + + comprData, err := c.mc.readNext(comprLength) + if err != nil { + return err + } + + // if payload is uncompressed, its length will be specified as zero, and its + // true length is contained in comprLength + if uncompressedLength == 0 { + c.buff.Write(comprData) + return nil + } + + // use existing capacity in bytesBuf if possible + c.buff.Grow(uncompressedLength) + nread, err := zDecompress(comprData, &c.buff) + if err != nil { + return err + } + if nread != uncompressedLength { + return fmt.Errorf("invalid compressed packet: uncompressed length in header is %d, actual %d", + uncompressedLength, nread) + } + return nil +} + +const minCompressLength = 150 +const maxPayloadLen = maxPacketSize - 4 + +// writePackets sends one or some packets with compression. +// Use this instead of mc.netConn.Write() when mc.compress is true. +func (c *compIO) writePackets(packets []byte) (int, error) { + totalBytes := len(packets) + blankHeader := make([]byte, 7) + buf := &c.buff + + for len(packets) > 0 { + payloadLen := min(maxPayloadLen, len(packets)) + payload := packets[:payloadLen] + uncompressedLen := payloadLen + + buf.Reset() + buf.Write(blankHeader) // Buffer.Write() never returns error + + // If payload is less than minCompressLength, don't compress. + if uncompressedLen < minCompressLength { + buf.Write(payload) + uncompressedLen = 0 + } else { + err := zCompress(payload, buf) + if debug && err != nil { + fmt.Printf("zCompress error: %v", err) + } + // do not compress if compressed data is larger than uncompressed data + // I intentionally miss 7 byte header in the buf; zCompress must compress more than 7 bytes. + if err != nil || buf.Len() >= uncompressedLen { + buf.Reset() + buf.Write(blankHeader) + buf.Write(payload) + uncompressedLen = 0 + } + } + + if n, err := c.writeCompressedPacket(buf.Bytes(), uncompressedLen); err != nil { + // To allow returning ErrBadConn when sending really 0 bytes, we sum + // up compressed bytes that is returned by underlying Write(). + return totalBytes - len(packets) + n, err + } + packets = packets[payloadLen:] + } + + return totalBytes, nil +} + +// writeCompressedPacket writes a compressed packet with header. +// data should start with 7 size space for header followed by payload. +func (c *compIO) writeCompressedPacket(data []byte, uncompressedLen int) (int, error) { + mc := c.mc + comprLength := len(data) - 7 + if debug { + fmt.Printf( + "writeCompressedPacket: comprLength=%v, uncompressedLen=%v, seq=%v\n", + comprLength, uncompressedLen, mc.compressSequence) + } + + // compression header + putUint24(data[0:3], comprLength) + data[3] = mc.compressSequence + putUint24(data[4:7], uncompressedLen) + + mc.compressSequence++ + return mc.writeWithTimeout(data) +} diff --git a/vendor/github.com/go-sql-driver/mysql/conncheck.go b/vendor/github.com/go-sql-driver/mysql/conncheck.go new file mode 100644 index 0000000000..f9c5cb65cb --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/conncheck.go @@ -0,0 +1,54 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2019 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +//go:build linux || darwin || dragonfly || freebsd || netbsd || openbsd || solaris || illumos + +package mysql + +import ( + "errors" + "io" + "net" + "syscall" +) + +var errUnexpectedRead = errors.New("unexpected read from socket") + +func connCheck(conn net.Conn) error { + var sysErr error + + sysConn, ok := conn.(syscall.Conn) + if !ok { + return nil + } + rawConn, err := sysConn.SyscallConn() + if err != nil { + return err + } + + err = rawConn.Read(func(fd uintptr) bool { + var buf [1]byte + n, err := syscall.Read(int(fd), buf[:]) + switch { + case n == 0 && err == nil: + sysErr = io.EOF + case n > 0: + sysErr = errUnexpectedRead + case err == syscall.EAGAIN || err == syscall.EWOULDBLOCK: + sysErr = nil + default: + sysErr = err + } + return true + }) + if err != nil { + return err + } + + return sysErr +} diff --git a/vendor/github.com/go-sql-driver/mysql/conncheck_dummy.go b/vendor/github.com/go-sql-driver/mysql/conncheck_dummy.go new file mode 100644 index 0000000000..0ebf05c213 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/conncheck_dummy.go @@ -0,0 +1,17 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2019 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +//go:build !linux && !darwin && !dragonfly && !freebsd && !netbsd && !openbsd && !solaris && !illumos + +package mysql + +import "net" + +func connCheck(conn net.Conn) error { + return nil +} diff --git a/vendor/github.com/go-sql-driver/mysql/connection.go b/vendor/github.com/go-sql-driver/mysql/connection.go new file mode 100644 index 0000000000..5a25c878ac --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/connection.go @@ -0,0 +1,818 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "context" + "database/sql" + "database/sql/driver" + "encoding/json" + "fmt" + "io" + "net" + "runtime" + "strconv" + "strings" + "sync/atomic" + "time" +) + +type mysqlConn struct { + buf buffer + netConn net.Conn + rawConn net.Conn // underlying connection when netConn is TLS connection. + result mysqlResult // managed by clearResult() and handleOkPacket(). + compIO *compIO + cfg *Config + connector *connector + maxAllowedPacket int + maxWriteSize int + capabilities capabilityFlag + extCapabilities extendedCapabilityFlag + status statusFlag + sequence uint8 + compressSequence uint8 + parseTime bool + compress bool + + // for context support (Go 1.8+) + watching bool + watcher chan<- context.Context + closech chan struct{} + finished chan<- struct{} + canceled atomicError // set non-nil if conn is canceled + closed atomic.Bool // set when conn is closed, before closech is closed +} + +// Helper function to call per-connection logger. +func (mc *mysqlConn) log(v ...any) { + _, filename, lineno, ok := runtime.Caller(1) + if ok { + pos := strings.LastIndexByte(filename, '/') + if pos != -1 { + filename = filename[pos+1:] + } + prefix := fmt.Sprintf("%s:%d ", filename, lineno) + v = append([]any{prefix}, v...) + } + + mc.cfg.Logger.Print(v...) +} + +func (mc *mysqlConn) readWithTimeout(b []byte) (int, error) { + to := mc.cfg.ReadTimeout + if to > 0 { + if err := mc.netConn.SetReadDeadline(time.Now().Add(to)); err != nil { + return 0, err + } + } + return mc.netConn.Read(b) +} + +func (mc *mysqlConn) writeWithTimeout(b []byte) (int, error) { + to := mc.cfg.WriteTimeout + if to > 0 { + if err := mc.netConn.SetWriteDeadline(time.Now().Add(to)); err != nil { + return 0, err + } + } + return mc.netConn.Write(b) +} + +func (mc *mysqlConn) resetSequence() { + mc.sequence = 0 + mc.compressSequence = 0 +} + +// syncSequence must be called when finished writing some packet and before start reading. +func (mc *mysqlConn) syncSequence() { + // Syncs compressionSequence to sequence. + // This is not documented but done in `net_flush()` in MySQL and MariaDB. + // https://github.com/mariadb-corporation/mariadb-connector-c/blob/8228164f850b12353da24df1b93a1e53cc5e85e9/libmariadb/ma_net.c#L170-L171 + // https://github.com/mysql/mysql-server/blob/824e2b4064053f7daf17d7f3f84b7a3ed92e5fb4/sql-common/net_serv.cc#L293 + if mc.compress { + mc.sequence = mc.compressSequence + mc.compIO.reset() + } +} + +// Handles parameters set in DSN after the connection is established +func (mc *mysqlConn) handleParams() (err error) { + var cmdSet strings.Builder + + for param, val := range mc.cfg.Params { + if cmdSet.Len() == 0 { + // Heuristic: 29 chars for each other key=value to reduce reallocations + cmdSet.Grow(4 + len(param) + 3 + len(val) + 30*(len(mc.cfg.Params)-1)) + cmdSet.WriteString("SET ") + } else { + cmdSet.WriteString(", ") + } + cmdSet.WriteString(param) + cmdSet.WriteString(" = ") + cmdSet.WriteString(val) + } + + if cmdSet.Len() > 0 { + err = mc.exec(cmdSet.String()) + } + + return +} + +// markBadConn replaces errBadConnNoWrite with driver.ErrBadConn. +// This function is used to return driver.ErrBadConn only when safe to retry. +func (mc *mysqlConn) markBadConn(err error) error { + if err == errBadConnNoWrite { + return driver.ErrBadConn + } + return err +} + +func (mc *mysqlConn) Begin() (driver.Tx, error) { + return mc.begin(false) +} + +func (mc *mysqlConn) begin(readOnly bool) (driver.Tx, error) { + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + var q string + if readOnly { + q = "START TRANSACTION READ ONLY" + } else { + q = "START TRANSACTION" + } + err := mc.exec(q) + if err == nil { + return &mysqlTx{mc}, err + } + return nil, mc.markBadConn(err) +} + +func (mc *mysqlConn) Close() (err error) { + // Makes Close idempotent + if !mc.closed.Load() { + err = mc.writeCommandPacket(comQuit) + } + mc.close() + return +} + +// close closes the network connection and clear results without sending COM_QUIT. +func (mc *mysqlConn) close() { + mc.cleanup() + mc.clearResult() +} + +// Closes the network connection and unsets internal variables. Do not call this +// function after successful authentication, call Close instead. This function +// is called before auth or on auth failure because MySQL will have already +// closed the network connection. +func (mc *mysqlConn) cleanup() { + if mc.closed.Swap(true) { + return + } + + // Makes cleanup idempotent + close(mc.closech) + conn := mc.rawConn + if conn == nil { + return + } + if err := conn.Close(); err != nil { + mc.log("closing connection:", err) + } + // This function can be called from multiple goroutines. + // So we can not mc.clearResult() here. + // Caller should do it if they are in safe goroutine. +} + +func (mc *mysqlConn) error() error { + if mc.closed.Load() { + if err := mc.canceled.Value(); err != nil { + return err + } + return ErrInvalidConn + } + return nil +} + +func (mc *mysqlConn) Prepare(query string) (driver.Stmt, error) { + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + // Send command + err := mc.writeCommandPacketStr(comStmtPrepare, query) + if err != nil { + // STMT_PREPARE is safe to retry. So we can return ErrBadConn here. + mc.log(err) + return nil, driver.ErrBadConn + } + + stmt := &mysqlStmt{ + mc: mc, + } + + // Read Result + columnCount, err := stmt.readPrepareResultPacket() + if err == nil { + if stmt.paramCount > 0 { + if err = mc.skipColumns(stmt.paramCount); err != nil { + return nil, err + } + } + + if columnCount > 0 { + if mc.extCapabilities&clientCacheMetadata != 0 { + if stmt.columns, err = mc.readColumns(int(columnCount), nil); err != nil { + return nil, err + } + } else { + if err = mc.skipColumns(int(columnCount)); err != nil { + return nil, err + } + } + } + } + + return stmt, err +} + +func (mc *mysqlConn) interpolateParams(query string, args []driver.Value) (string, error) { + noBackslashEscapes := (mc.status & statusNoBackslashEscapes) != 0 + const ( + stateNormal = iota + stateString + stateEscape + stateEOLComment + stateSlashStarComment + stateBacktick + ) + + const ( + QUOTE_BYTE = byte('\'') + DBL_QUOTE_BYTE = byte('"') + BACKSLASH_BYTE = byte('\\') + QUESTION_MARK_BYTE = byte('?') + SLASH_BYTE = byte('/') + STAR_BYTE = byte('*') + HASH_BYTE = byte('#') + MINUS_BYTE = byte('-') + LINE_FEED_BYTE = byte('\n') + BACKTICK_BYTE = byte('`') + ) + + buf, err := mc.buf.takeCompleteBuffer() + if err != nil { + mc.cleanup() + return "", driver.ErrBadConn + } + buf = buf[:0] + state := stateNormal + singleQuotes := false + lastChar := byte(0) + argPos := 0 + lenQuery := len(query) + lastIdx := 0 + + for i := range lenQuery { + currentChar := query[i] + if state == stateEscape && !((currentChar == QUOTE_BYTE && singleQuotes) || (currentChar == DBL_QUOTE_BYTE && !singleQuotes)) { + state = stateString + lastChar = currentChar + continue + } + switch currentChar { + case STAR_BYTE: + if state == stateNormal && lastChar == SLASH_BYTE { + state = stateSlashStarComment + } + case SLASH_BYTE: + if state == stateSlashStarComment && lastChar == STAR_BYTE { + state = stateNormal + // Clear lastChar so the '/' that closed the comment isn't + // reused to start a new comment with a following '*'. + lastChar = 0 + continue + } + case HASH_BYTE: + if state == stateNormal { + state = stateEOLComment + } + case MINUS_BYTE: + if state == stateNormal && lastChar == MINUS_BYTE { + // -- only starts a comment if followed by whitespace or control char + if i+1 < lenQuery { + nextChar := query[i+1] + if nextChar == ' ' || nextChar == '\t' || nextChar == '\n' || nextChar == '\r' { + state = stateEOLComment + } + } else { + state = stateEOLComment + } + } + case LINE_FEED_BYTE: + if state == stateEOLComment { + state = stateNormal + } + case DBL_QUOTE_BYTE: + if state == stateNormal { + state = stateString + singleQuotes = false + } else if state == stateString && !singleQuotes { + state = stateNormal + } else if state == stateEscape { + state = stateString + } + case QUOTE_BYTE: + if state == stateNormal { + state = stateString + singleQuotes = true + } else if state == stateString && singleQuotes { + state = stateNormal + } else if state == stateEscape { + state = stateString + } + case BACKSLASH_BYTE: + if state == stateString && !noBackslashEscapes { + state = stateEscape + } + case QUESTION_MARK_BYTE: + if state == stateNormal { + if argPos >= len(args) { + return "", driver.ErrSkip + } + buf = append(buf, query[lastIdx:i]...) + arg := args[argPos] + argPos++ + + if arg == nil { + buf = append(buf, "NULL"...) + lastIdx = i + 1 + break + } + + switch v := arg.(type) { + case int64: + buf = strconv.AppendInt(buf, v, 10) + case uint64: + buf = strconv.AppendUint(buf, v, 10) + case float64: + buf = strconv.AppendFloat(buf, v, 'g', -1, 64) + case bool: + if v { + buf = append(buf, '1') + } else { + buf = append(buf, '0') + } + case time.Time: + if v.IsZero() { + buf = append(buf, "'0000-00-00'"...) + } else { + buf = append(buf, '\'') + buf, err = appendDateTime(buf, v.In(mc.cfg.Loc), mc.cfg.timeTruncate) + if err != nil { + return "", err + } + buf = append(buf, '\'') + } + case json.RawMessage: + if v == nil { + buf = append(buf, "NULL"...) + } else if noBackslashEscapes { + buf = escapeBytesQuotes(buf, v, false) + } else { + buf = escapeBytesBackslash(buf, v, false) + } + case []byte: + if v == nil { + buf = append(buf, "NULL"...) + } else { + if noBackslashEscapes { + buf = escapeBytesQuotes(buf, v, true) + } else { + buf = escapeBytesBackslash(buf, v, true) + } + } + case string: + if noBackslashEscapes { + buf = escapeStringQuotes(buf, v) + } else { + buf = escapeStringBackslash(buf, v) + } + default: + return "", driver.ErrSkip + } + + if len(buf)+4 > mc.maxAllowedPacket { + return "", driver.ErrSkip + } + lastIdx = i + 1 + } + case BACKTICK_BYTE: + if state == stateBacktick { + state = stateNormal + } else if state == stateNormal { + state = stateBacktick + } + } + lastChar = currentChar + } + buf = append(buf, query[lastIdx:]...) + if argPos != len(args) { + return "", driver.ErrSkip + } + return string(buf), nil +} + +func (mc *mysqlConn) Exec(query string, args []driver.Value) (driver.Result, error) { + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + if len(args) != 0 { + if !mc.cfg.InterpolateParams { + return nil, driver.ErrSkip + } + // try to interpolate the parameters to save extra roundtrips for preparing and closing a statement + prepared, err := mc.interpolateParams(query, args) + if err != nil { + return nil, err + } + query = prepared + } + + err := mc.exec(query) + if err == nil { + copied := mc.result + return &copied, err + } + return nil, mc.markBadConn(err) +} + +// Internal function to execute commands +func (mc *mysqlConn) exec(query string) error { + handleOk := mc.clearResult() + // Send command + if err := mc.writeCommandPacketStr(comQuery, query); err != nil { + return mc.markBadConn(err) + } + + // Read Result + resLen, _, err := handleOk.readResultSetHeaderPacket() + if err != nil { + return err + } + + if resLen > 0 { + // columns + if err := mc.skipColumns(resLen); err != nil { + return err + } + + // rows + if err := mc.skipRows(); err != nil { + return err + } + } + + return handleOk.discardResults() +} + +func (mc *mysqlConn) Query(query string, args []driver.Value) (driver.Rows, error) { + return mc.query(query, args) +} + +func (mc *mysqlConn) query(query string, args []driver.Value) (*textRows, error) { + handleOk := mc.clearResult() + + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + if len(args) != 0 { + if !mc.cfg.InterpolateParams { + return nil, driver.ErrSkip + } + // try client-side prepare to reduce roundtrip + prepared, err := mc.interpolateParams(query, args) + if err != nil { + return nil, err + } + query = prepared + } + // Send command + err := mc.writeCommandPacketStr(comQuery, query) + if err != nil { + return nil, mc.markBadConn(err) + } + + // Read Result + var resLen int + resLen, _, err = handleOk.readResultSetHeaderPacket() + if err != nil { + return nil, err + } + + rows := new(textRows) + rows.mc = mc + + if resLen == 0 { + rows.rs.done = true + + switch err := rows.NextResultSet(); err { + case nil, io.EOF: + return rows, nil + default: + return nil, err + } + } + + // Columns + rows.rs.columns, err = mc.readColumns(resLen, nil) + return rows, err +} + +// Gets the value of the given MySQL System Variable +func (mc *mysqlConn) getSystemVar(name string) (string, error) { + // Send command + handleOk := mc.clearResult() + if err := mc.writeCommandPacketStr(comQuery, "SELECT @@"+name); err != nil { + return "", err + } + + // Read Result + resLen, _, err := handleOk.readResultSetHeaderPacket() + if err == nil { + rows := new(textRows) + rows.mc = mc + rows.rs.columns = []mysqlField{{fieldType: fieldTypeVarChar}} + + if resLen > 0 { + // Columns + if err := mc.skipColumns(resLen); err != nil { + return "", err + } + } + + dest := make([]driver.Value, resLen) + if err = rows.readRow(dest); err == nil { + // Convert to string before skipRows, which may + // overwrite the read buffer that dest[0] points into. + val := string(dest[0].([]byte)) + return val, mc.skipRows() + } + } + return "", err +} + +// cancel is called when the query has canceled. +func (mc *mysqlConn) cancel(err error) { + mc.canceled.Set(err) + mc.cleanup() +} + +// finish is called when the query has succeeded. +func (mc *mysqlConn) finish() { + if !mc.watching || mc.finished == nil { + return + } + select { + case mc.finished <- struct{}{}: + mc.watching = false + case <-mc.closech: + } +} + +// Ping implements driver.Pinger interface +func (mc *mysqlConn) Ping(ctx context.Context) (err error) { + if mc.closed.Load() { + return driver.ErrBadConn + } + + if err = mc.watchCancel(ctx); err != nil { + return + } + defer mc.finish() + + handleOk := mc.clearResult() + if err = mc.writeCommandPacket(comPing); err != nil { + return mc.markBadConn(err) + } + + return handleOk.readResultOK() +} + +// BeginTx implements driver.ConnBeginTx interface +func (mc *mysqlConn) BeginTx(ctx context.Context, opts driver.TxOptions) (driver.Tx, error) { + if mc.closed.Load() { + return nil, driver.ErrBadConn + } + + if err := mc.watchCancel(ctx); err != nil { + return nil, err + } + defer mc.finish() + + if sql.IsolationLevel(opts.Isolation) != sql.LevelDefault { + level, err := mapIsolationLevel(opts.Isolation) + if err != nil { + return nil, err + } + err = mc.exec("SET TRANSACTION ISOLATION LEVEL " + level) + if err != nil { + return nil, err + } + } + + return mc.begin(opts.ReadOnly) +} + +func (mc *mysqlConn) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error) { + dargs, err := namedValueToValue(args) + if err != nil { + return nil, err + } + + if err := mc.watchCancel(ctx); err != nil { + return nil, err + } + + rows, err := mc.query(query, dargs) + if err != nil { + mc.finish() + return nil, err + } + rows.finish = mc.finish + return rows, err +} + +func (mc *mysqlConn) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error) { + dargs, err := namedValueToValue(args) + if err != nil { + return nil, err + } + + if err := mc.watchCancel(ctx); err != nil { + return nil, err + } + defer mc.finish() + + return mc.Exec(query, dargs) +} + +func (mc *mysqlConn) PrepareContext(ctx context.Context, query string) (driver.Stmt, error) { + if err := mc.watchCancel(ctx); err != nil { + return nil, err + } + + stmt, err := mc.Prepare(query) + mc.finish() + if err != nil { + return nil, err + } + + select { + default: + case <-ctx.Done(): + stmt.Close() + return nil, ctx.Err() + } + return stmt, nil +} + +func (stmt *mysqlStmt) QueryContext(ctx context.Context, args []driver.NamedValue) (driver.Rows, error) { + dargs, err := namedValueToValue(args) + if err != nil { + return nil, err + } + + if err := stmt.mc.watchCancel(ctx); err != nil { + return nil, err + } + + rows, err := stmt.query(dargs) + if err != nil { + stmt.mc.finish() + return nil, err + } + rows.finish = stmt.mc.finish + return rows, err +} + +func (stmt *mysqlStmt) ExecContext(ctx context.Context, args []driver.NamedValue) (driver.Result, error) { + dargs, err := namedValueToValue(args) + if err != nil { + return nil, err + } + + if err := stmt.mc.watchCancel(ctx); err != nil { + return nil, err + } + defer stmt.mc.finish() + + return stmt.Exec(dargs) +} + +func (mc *mysqlConn) watchCancel(ctx context.Context) error { + if mc.watching { + // Reach here if canceled, + // so the connection is already invalid + mc.cleanup() + return nil + } + // When ctx is already cancelled, don't watch it. + if err := ctx.Err(); err != nil { + return err + } + // When ctx is not cancellable, don't watch it. + if ctx.Done() == nil { + return nil + } + // When watcher is not alive, can't watch it. + if mc.watcher == nil { + return nil + } + + mc.watching = true + mc.watcher <- ctx + return nil +} + +func (mc *mysqlConn) startWatcher() { + watcher := make(chan context.Context, 1) + mc.watcher = watcher + finished := make(chan struct{}) + mc.finished = finished + go func() { + for { + var ctx context.Context + select { + case ctx = <-watcher: + case <-mc.closech: + return + } + + select { + case <-ctx.Done(): + mc.cancel(ctx.Err()) + case <-finished: + case <-mc.closech: + return + } + } + }() +} + +func (mc *mysqlConn) CheckNamedValue(nv *driver.NamedValue) (err error) { + nv.Value, err = converter{}.ConvertValue(nv.Value) + return +} + +// ResetSession implements driver.SessionResetter. +// (From Go 1.10) +func (mc *mysqlConn) ResetSession(ctx context.Context) error { + if mc.closed.Load() || mc.buf.busy() { + return driver.ErrBadConn + } + + // Perform a stale connection check. We only perform this check for + // the first query on a connection that has been checked out of the + // connection pool: a fresh connection from the pool is more likely + // to be stale, and it has not performed any previous writes that + // could cause data corruption, so it's safe to return ErrBadConn + // if the check fails. + if mc.cfg.CheckConnLiveness { + conn := mc.netConn + if mc.rawConn != nil { + conn = mc.rawConn + } + var err error + if mc.cfg.ReadTimeout != 0 { + err = conn.SetReadDeadline(time.Now().Add(mc.cfg.ReadTimeout)) + } + if err == nil { + err = connCheck(conn) + } + if err != nil { + mc.log("closing bad idle connection: ", err) + return driver.ErrBadConn + } + } + + return nil +} + +// IsValid implements driver.Validator interface +// (From Go 1.15) +func (mc *mysqlConn) IsValid() bool { + return !mc.closed.Load() && !mc.buf.busy() +} + +var _ driver.SessionResetter = &mysqlConn{} +var _ driver.Validator = &mysqlConn{} diff --git a/vendor/github.com/go-sql-driver/mysql/connector.go b/vendor/github.com/go-sql-driver/mysql/connector.go new file mode 100644 index 0000000000..3d37604775 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/connector.go @@ -0,0 +1,229 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2018 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "context" + "database/sql/driver" + "fmt" + "net" + "os" + "strconv" + "strings" +) + +type connector struct { + cfg *Config // immutable private copy. + encodedAttributes string // Encoded connection attributes. +} + +func encodeConnectionAttributes(cfg *Config) string { + connAttrsBuf := make([]byte, 0) + + // default connection attributes + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrClientName) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrClientNameValue) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrOS) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrOSValue) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrPlatform) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrPlatformValue) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrPid) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, strconv.Itoa(os.Getpid())) + serverHost, _, _ := net.SplitHostPort(cfg.Addr) + if serverHost != "" { + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, connAttrServerHost) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, serverHost) + } + + // user-defined connection attributes + for connAttr := range strings.SplitSeq(cfg.ConnectionAttributes, ",") { + k, v, found := strings.Cut(connAttr, ":") + if !found { + continue + } + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, k) + connAttrsBuf = appendLengthEncodedString(connAttrsBuf, v) + } + + return string(connAttrsBuf) +} + +func newConnector(cfg *Config) *connector { + encodedAttributes := encodeConnectionAttributes(cfg) + return &connector{ + cfg: cfg, + encodedAttributes: encodedAttributes, + } +} + +// Connect implements driver.Connector interface. +// Connect returns a connection to the database. +func (c *connector) Connect(ctx context.Context) (driver.Conn, error) { + var err error + + // Invoke beforeConnect if present, with a copy of the configuration + cfg := c.cfg + if c.cfg.beforeConnect != nil { + cfg = c.cfg.Clone() + err = c.cfg.beforeConnect(ctx, cfg) + if err != nil { + return nil, err + } + } + + // New mysqlConn + mc := &mysqlConn{ + maxAllowedPacket: maxPacketSize, + maxWriteSize: maxPacketSize - 1, + closech: make(chan struct{}), + cfg: cfg, + connector: c, + } + mc.parseTime = mc.cfg.ParseTime + + // Connect to Server + dctx := ctx + if mc.cfg.Timeout > 0 { + var cancel context.CancelFunc + dctx, cancel = context.WithTimeout(ctx, c.cfg.Timeout) + defer cancel() + } + + if c.cfg.DialFunc != nil { + mc.netConn, err = c.cfg.DialFunc(dctx, mc.cfg.Net, mc.cfg.Addr) + } else { + dialsLock.RLock() + dial, ok := dials[mc.cfg.Net] + dialsLock.RUnlock() + if ok { + mc.netConn, err = dial(dctx, mc.cfg.Addr) + } else { + nd := net.Dialer{} + mc.netConn, err = nd.DialContext(dctx, mc.cfg.Net, mc.cfg.Addr) + } + } + if err != nil { + return nil, err + } + mc.rawConn = mc.netConn + + // Enable TCP Keepalives on TCP connections + if tc, ok := mc.netConn.(*net.TCPConn); ok { + if err := tc.SetKeepAlive(true); err != nil { + c.cfg.Logger.Print(err) + } + } + + // Call startWatcher for context support (From Go 1.8) + mc.startWatcher() + if err := mc.watchCancel(ctx); err != nil { + mc.cleanup() + return nil, err + } + defer mc.finish() + + mc.buf = newBuffer() + + // Reading Handshake Initialization Packet + authData, serverCapabilities, serverExtCapabilities, plugin, err := mc.readHandshakePacket() + if err != nil { + mc.cleanup() + return nil, err + } + + if plugin == "" { + plugin = defaultAuthPlugin + } + + // Send Client Authentication Packet + authResp, err := mc.auth(authData, plugin) + if err != nil { + // try the default auth plugin, if using the requested plugin failed + c.cfg.Logger.Print("could not use requested auth plugin '"+plugin+"': ", err.Error()) + plugin = defaultAuthPlugin + authResp, err = mc.auth(authData, plugin) + if err != nil { + mc.cleanup() + return nil, err + } + } + mc.initCapabilities(serverCapabilities, serverExtCapabilities, mc.cfg) + if err = mc.writeHandshakeResponsePacket(authResp, plugin); err != nil { + mc.cleanup() + return nil, err + } + + // Handle response to auth packet, switch methods if possible + if err = mc.handleAuthResult(authData, plugin); err != nil { + // Authentication failed and MySQL has already closed the connection + // (https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase.html#sect_protocol_connection_phase_fast_path_fails). + // Do not send COM_QUIT, just cleanup and return the error. + mc.cleanup() + return nil, err + } + + // compression is enabled after auth, not right after sending handshake response. + if mc.capabilities&clientCompress > 0 { + mc.compress = true + mc.compIO = newCompIO(mc) + } + if mc.cfg.MaxAllowedPacket > 0 { + mc.maxAllowedPacket = mc.cfg.MaxAllowedPacket + } else { + // Get max allowed packet size + maxap, err := mc.getSystemVar("max_allowed_packet") + if err != nil { + mc.Close() + return nil, err + } + n, err := strconv.Atoi(maxap) + if err != nil { + mc.Close() + return nil, fmt.Errorf("invalid max_allowed_packet value (%q): %w", maxap, err) + } + mc.maxAllowedPacket = n - 1 + } + if mc.maxAllowedPacket < maxPacketSize { + mc.maxWriteSize = mc.maxAllowedPacket + } + + // Charset: character_set_connection, character_set_client, character_set_results + if len(mc.cfg.charsets) > 0 { + for _, cs := range mc.cfg.charsets { + // ignore errors here - a charset may not exist + if mc.cfg.Collation != "" { + err = mc.exec("SET NAMES " + cs + " COLLATE " + mc.cfg.Collation) + } else { + err = mc.exec("SET NAMES " + cs) + } + if err == nil { + break + } + } + if err != nil { + mc.Close() + return nil, err + } + } + + // Handle DSN Params + err = mc.handleParams() + if err != nil { + mc.Close() + return nil, err + } + + return mc, nil +} + +// Driver implements driver.Connector interface. +// Driver returns &MySQLDriver{}. +func (c *connector) Driver() driver.Driver { + return &MySQLDriver{} +} diff --git a/vendor/github.com/go-sql-driver/mysql/const.go b/vendor/github.com/go-sql-driver/mysql/const.go new file mode 100644 index 0000000000..6f0cdf3032 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/const.go @@ -0,0 +1,205 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import "runtime" + +const ( + debug = false // for debugging. Set true only in development. + + defaultAuthPlugin = "mysql_native_password" + defaultMaxAllowedPacket = 64 << 20 // 64 MiB. See https://github.com/go-sql-driver/mysql/issues/1355 + minProtocolVersion = 10 + maxPacketSize = 1<<24 - 1 + timeFormat = "2006-01-02 15:04:05.999999" + + // Connection attributes + // See https://dev.mysql.com/doc/refman/8.0/en/performance-schema-connection-attribute-tables.html#performance-schema-connection-attributes-available + connAttrClientName = "_client_name" + connAttrClientNameValue = "Go-MySQL-Driver" + connAttrOS = "_os" + connAttrOSValue = runtime.GOOS + connAttrPlatform = "_platform" + connAttrPlatformValue = runtime.GOARCH + connAttrPid = "_pid" + connAttrServerHost = "_server_host" +) + +// MySQL constants documentation: +// https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_PROTOCOL.html + +const ( + iOK byte = 0x00 + iAuthMoreData byte = 0x01 + iLocalInFile byte = 0xfb + iEOF byte = 0xfe + iERR byte = 0xff +) + +// https://dev.mysql.com/doc/dev/mysql-server/latest/group__group__cs__capabilities__flags.html +// https://mariadb.com/kb/en/connection/#capabilities +type capabilityFlag uint32 + +const ( + clientMySQL capabilityFlag = 1 << iota + clientFoundRows + clientLongFlag + clientConnectWithDB + clientNoSchema + clientCompress + clientODBC + clientLocalFiles + clientIgnoreSpace + clientProtocol41 + clientInteractive + clientSSL + clientIgnoreSIGPIPE + clientTransactions + clientReserved + clientSecureConn + clientMultiStatements + clientMultiResults + clientPSMultiResults + clientPluginAuth + clientConnectAttrs + clientPluginAuthLenEncClientData + clientCanHandleExpiredPasswords + clientSessionTrack + clientDeprecateEOF +) + +// https://mariadb.com/kb/en/connection/#capabilities +type extendedCapabilityFlag uint32 + +const ( + progressIndicator extendedCapabilityFlag = 1 << iota + clientComMulti + clientStmtBulkOperations + clientExtendedMetadata + clientCacheMetadata + clientUnitBulkResult +) + +const ( + comQuit byte = iota + 1 + comInitDB + comQuery + comFieldList + comCreateDB + comDropDB + comRefresh + comShutdown + comStatistics + comProcessInfo + comConnect + comProcessKill + comDebug + comPing + comTime + comDelayedInsert + comChangeUser + comBinlogDump + comTableDump + comConnectOut + comRegisterSlave + comStmtPrepare + comStmtExecute + comStmtSendLongData + comStmtClose + comStmtReset + comSetOption + comStmtFetch +) + +// https://dev.mysql.com/doc/internals/en/com-query-response.html#packet-Protocol::ColumnType +type fieldType byte + +const ( + fieldTypeDecimal fieldType = iota + fieldTypeTiny + fieldTypeShort + fieldTypeLong + fieldTypeFloat + fieldTypeDouble + fieldTypeNULL + fieldTypeTimestamp + fieldTypeLongLong + fieldTypeInt24 + fieldTypeDate + fieldTypeTime + fieldTypeDateTime + fieldTypeYear + fieldTypeNewDate + fieldTypeVarChar + fieldTypeBit +) +const ( + fieldTypeVector fieldType = iota + 0xf2 + fieldTypeInvalid + fieldTypeBool + fieldTypeJSON + fieldTypeNewDecimal + fieldTypeEnum + fieldTypeSet + fieldTypeTinyBLOB + fieldTypeMediumBLOB + fieldTypeLongBLOB + fieldTypeBLOB + fieldTypeVarString + fieldTypeString + fieldTypeGeometry +) + +type fieldFlag uint16 + +const ( + flagNotNULL fieldFlag = 1 << iota + flagPriKey + flagUniqueKey + flagMultipleKey + flagBLOB + flagUnsigned + flagZeroFill + flagBinary + flagEnum + flagAutoIncrement + flagTimestamp + flagSet + flagUnknown1 + flagUnknown2 + flagUnknown3 + flagUnknown4 +) + +// http://dev.mysql.com/doc/internals/en/status-flags.html +type statusFlag uint16 + +const ( + statusInTrans statusFlag = 1 << iota + statusInAutocommit + statusReserved // Not in documentation + statusMoreResultsExists + statusNoGoodIndexUsed + statusNoIndexUsed + statusCursorExists + statusLastRowSent + statusDbDropped + statusNoBackslashEscapes + statusMetadataChanged + statusQueryWasSlow + statusPsOutParams + statusInTransReadonly + statusSessionStateChanged +) + +const ( + cachingSha2PasswordRequestPublicKey = 2 + cachingSha2PasswordFastAuthSuccess = 3 + cachingSha2PasswordPerformFullAuthentication = 4 +) diff --git a/vendor/github.com/go-sql-driver/mysql/driver.go b/vendor/github.com/go-sql-driver/mysql/driver.go new file mode 100644 index 0000000000..105316b816 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/driver.go @@ -0,0 +1,118 @@ +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +// Package mysql provides a MySQL driver for Go's database/sql package. +// +// The driver should be used via the database/sql package: +// +// import "database/sql" +// import _ "github.com/go-sql-driver/mysql" +// +// db, err := sql.Open("mysql", "user:password@/dbname") +// +// See https://github.com/go-sql-driver/mysql#usage for details +package mysql + +import ( + "context" + "database/sql" + "database/sql/driver" + "net" + "sync" +) + +// MySQLDriver is exported to make the driver directly accessible. +// In general the driver is used via the database/sql package. +type MySQLDriver struct{} + +// DialFunc is a function which can be used to establish the network connection. +// Custom dial functions must be registered with RegisterDial +// +// Deprecated: users should register a DialContextFunc instead +type DialFunc func(addr string) (net.Conn, error) + +// DialContextFunc is a function which can be used to establish the network connection. +// Custom dial functions must be registered with RegisterDialContext +type DialContextFunc func(ctx context.Context, addr string) (net.Conn, error) + +var ( + dialsLock sync.RWMutex + dials map[string]DialContextFunc +) + +// RegisterDialContext registers a custom dial function. It can then be used by the +// network address mynet(addr), where mynet is the registered new network. +// The current context for the connection and its address is passed to the dial function. +func RegisterDialContext(net string, dial DialContextFunc) { + dialsLock.Lock() + defer dialsLock.Unlock() + if dials == nil { + dials = make(map[string]DialContextFunc) + } + dials[net] = dial +} + +// DeregisterDialContext removes the custom dial function registered with the given net. +func DeregisterDialContext(net string) { + dialsLock.Lock() + defer dialsLock.Unlock() + if dials != nil { + delete(dials, net) + } +} + +// RegisterDial registers a custom dial function. It can then be used by the +// network address mynet(addr), where mynet is the registered new network. +// addr is passed as a parameter to the dial function. +// +// Deprecated: users should call RegisterDialContext instead +func RegisterDial(network string, dial DialFunc) { + RegisterDialContext(network, func(_ context.Context, addr string) (net.Conn, error) { + return dial(addr) + }) +} + +// Open new Connection. +// See https://github.com/go-sql-driver/mysql#dsn-data-source-name for how +// the DSN string is formatted +func (d MySQLDriver) Open(dsn string) (driver.Conn, error) { + cfg, err := ParseDSN(dsn) + if err != nil { + return nil, err + } + c := newConnector(cfg) + return c.Connect(context.Background()) +} + +// This variable can be replaced with -ldflags like below: +// go build "-ldflags=-X github.com/go-sql-driver/mysql.driverName=custom" +var driverName = "mysql" + +func init() { + if driverName != "" { + sql.Register(driverName, &MySQLDriver{}) + } +} + +// NewConnector returns new driver.Connector. +func NewConnector(cfg *Config) (driver.Connector, error) { + cfg = cfg.Clone() + // normalize the contents of cfg so calls to NewConnector have the same + // behavior as MySQLDriver.OpenConnector + if err := cfg.normalize(); err != nil { + return nil, err + } + return newConnector(cfg), nil +} + +// OpenConnector implements driver.DriverContext. +func (d MySQLDriver) OpenConnector(dsn string) (driver.Connector, error) { + cfg, err := ParseDSN(dsn) + if err != nil { + return nil, err + } + return newConnector(cfg), nil +} diff --git a/vendor/github.com/go-sql-driver/mysql/dsn.go b/vendor/github.com/go-sql-driver/mysql/dsn.go new file mode 100644 index 0000000000..41463a503d --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/dsn.go @@ -0,0 +1,706 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2016 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "context" + "crypto/rsa" + "crypto/tls" + "errors" + "fmt" + "maps" + "math/big" + "net" + "net/url" + "sort" + "strconv" + "strings" + "time" +) + +var ( + errInvalidDSNUnescaped = errors.New("invalid DSN: did you forget to escape a param value?") + errInvalidDSNAddr = errors.New("invalid DSN: network address not terminated (missing closing brace)") + errInvalidDSNNoSlash = errors.New("invalid DSN: missing the slash separating the database name") + errInvalidDSNUnsafeCollation = errors.New("invalid DSN: interpolateParams can not be used with unsafe collations") +) + +// Config is a configuration parsed from a DSN string. +// If a new Config is created instead of being parsed from a DSN string, +// the NewConfig function should be used, which sets default values. +type Config struct { + // non boolean fields + + User string // Username + Passwd string // Password (requires User) + Net string // Network (e.g. "tcp", "tcp6", "unix". default: "tcp") + Addr string // Address (default: "127.0.0.1:3306" for "tcp" and "/tmp/mysql.sock" for "unix") + DBName string // Database name + Params map[string]string // Connection parameters + ConnectionAttributes string // Connection Attributes, comma-delimited string of user-defined "key:value" pairs + Collation string // Connection collation. When set, this will be set in SET NAMES COLLATE query + Loc *time.Location // Location for time.Time values + MaxAllowedPacket int // Max packet size allowed + ServerPubKey string // Server public key name + TLSConfig string // TLS configuration name + TLS *tls.Config // TLS configuration, its priority is higher than TLSConfig + Timeout time.Duration // Dial timeout + ReadTimeout time.Duration // I/O read timeout + WriteTimeout time.Duration // I/O write timeout + Logger Logger // Logger + // DialFunc specifies the dial function for creating connections + DialFunc func(ctx context.Context, network, addr string) (net.Conn, error) + + // boolean fields + + AllowAllFiles bool // Allow all files to be used with LOAD DATA LOCAL INFILE + AllowCleartextPasswords bool // Allows the cleartext client side plugin + AllowFallbackToPlaintext bool // Allows fallback to unencrypted connection if server does not support TLS + AllowNativePasswords bool // Allows the native password authentication method + AllowOldPasswords bool // Allows the old insecure password method + CheckConnLiveness bool // Check connections for liveness before using them + ClientFoundRows bool // Return number of matching rows instead of rows changed + ColumnsWithAlias bool // Prepend table alias to column names + InterpolateParams bool // Interpolate placeholders into query string + MultiStatements bool // Allow multiple statements in one query + ParseTime bool // Parse time values to time.Time + RejectReadOnly bool // Reject read-only connections + + // unexported fields. new options should be come here. + // boolean first. alphabetical order. + + compress bool // Enable zlib compression + + beforeConnect func(context.Context, *Config) error // Invoked before a connection is established + pubKey *rsa.PublicKey // Server public key + timeTruncate time.Duration // Truncate time.Time values to the specified duration + charsets []string // Connection charset. When set, this will be set in SET NAMES query +} + +// Functional Options Pattern +// https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis +type Option func(*Config) error + +// NewConfig creates a new Config and sets default values. +func NewConfig() *Config { + cfg := &Config{ + Loc: time.UTC, + MaxAllowedPacket: defaultMaxAllowedPacket, + Logger: defaultLogger, + AllowNativePasswords: true, + CheckConnLiveness: true, + } + return cfg +} + +// Apply applies the given options to the Config object. +func (c *Config) Apply(opts ...Option) error { + for _, opt := range opts { + err := opt(c) + if err != nil { + return err + } + } + return nil +} + +// TimeTruncate sets the time duration to truncate time.Time values in +// query parameters. +func TimeTruncate(d time.Duration) Option { + return func(cfg *Config) error { + cfg.timeTruncate = d + return nil + } +} + +// BeforeConnect sets the function to be invoked before a connection is established. +func BeforeConnect(fn func(context.Context, *Config) error) Option { + return func(cfg *Config) error { + cfg.beforeConnect = fn + return nil + } +} + +// EnableCompress sets the compression mode. +func EnableCompression(yes bool) Option { + return func(cfg *Config) error { + cfg.compress = yes + return nil + } +} + +// Charset sets the connection charset and collation. +// +// charset is the connection charset. +// collation is the connection collation. It can be null or empty string. +// +// When collation is not specified, `SET NAMES ` command is sent when the connection is established. +// When collation is specified, `SET NAMES COLLATE ` command is sent when the connection is established. +func Charset(charset, collation string) Option { + return func(cfg *Config) error { + cfg.charsets = []string{charset} + cfg.Collation = collation + return nil + } +} + +func (cfg *Config) Clone() *Config { + cp := *cfg + if cp.TLS != nil { + cp.TLS = cfg.TLS.Clone() + } + if len(cp.Params) > 0 { + cp.Params = make(map[string]string, len(cfg.Params)) + maps.Copy(cp.Params, cfg.Params) + } + if cfg.pubKey != nil { + cp.pubKey = &rsa.PublicKey{ + N: new(big.Int).Set(cfg.pubKey.N), + E: cfg.pubKey.E, + } + } + return &cp +} + +func (cfg *Config) normalize() error { + if cfg.InterpolateParams && cfg.Collation != "" && unsafeCollations[cfg.Collation] { + return errInvalidDSNUnsafeCollation + } + + // Set default network if empty + if cfg.Net == "" { + cfg.Net = "tcp" + } + + // Set default address if empty + if cfg.Addr == "" { + switch cfg.Net { + case "tcp": + cfg.Addr = "127.0.0.1:3306" + case "unix": + cfg.Addr = "/tmp/mysql.sock" + default: + return errors.New("default addr for network '" + cfg.Net + "' unknown") + } + } else if cfg.Net == "tcp" { + cfg.Addr = ensureHavePort(cfg.Addr) + } + + if cfg.TLS == nil { + switch cfg.TLSConfig { + case "false", "": + // don't set anything + case "true": + cfg.TLS = &tls.Config{} + case "skip-verify": + cfg.TLS = &tls.Config{InsecureSkipVerify: true} + case "preferred": + cfg.TLS = &tls.Config{InsecureSkipVerify: true} + cfg.AllowFallbackToPlaintext = true + default: + cfg.TLS = getTLSConfigClone(cfg.TLSConfig) + if cfg.TLS == nil { + return errors.New("invalid value / unknown config name: " + cfg.TLSConfig) + } + } + } + + if cfg.TLS != nil && cfg.TLS.ServerName == "" && !cfg.TLS.InsecureSkipVerify { + host, _, err := net.SplitHostPort(cfg.Addr) + if err == nil { + cfg.TLS.ServerName = host + } + } + + if cfg.ServerPubKey != "" { + cfg.pubKey = getServerPubKey(cfg.ServerPubKey) + if cfg.pubKey == nil { + return errors.New("invalid value / unknown server pub key name: " + cfg.ServerPubKey) + } + } + + if cfg.Logger == nil { + cfg.Logger = defaultLogger + } + + return nil +} + +func writeDSNParam(buf *bytes.Buffer, hasParam *bool, name, value string) { + buf.Grow(1 + len(name) + 1 + len(value)) + if !*hasParam { + *hasParam = true + buf.WriteByte('?') + } else { + buf.WriteByte('&') + } + buf.WriteString(name) + buf.WriteByte('=') + buf.WriteString(value) +} + +// FormatDSN formats the given Config into a DSN string which can be passed to +// the driver. +// +// Note: use [NewConnector] and [database/sql.OpenDB] to open a connection from a [*Config]. +func (cfg *Config) FormatDSN() string { + var buf bytes.Buffer + + // [username[:password]@] + if len(cfg.User) > 0 { + buf.WriteString(cfg.User) + if len(cfg.Passwd) > 0 { + buf.WriteByte(':') + buf.WriteString(cfg.Passwd) + } + buf.WriteByte('@') + } + + // [protocol[(address)]] + if len(cfg.Addr) > 0 { + net := cfg.Net + if net == "" { + net = "tcp" + } + buf.WriteString(net) + buf.WriteByte('(') + buf.WriteString(cfg.Addr) + buf.WriteByte(')') + } else if cfg.Net != "" && cfg.Net != "tcp" { + // Preserve an explicit non-default protocol when there's no + // address, so e.g. Net="unix" still round-trips. + buf.WriteString(cfg.Net) + } + + // /dbname + buf.WriteByte('/') + buf.WriteString(url.PathEscape(cfg.DBName)) + + // [?param1=value1&...¶mN=valueN] + hasParam := false + + if cfg.AllowAllFiles { + hasParam = true + buf.WriteString("?allowAllFiles=true") + } + + if cfg.AllowCleartextPasswords { + writeDSNParam(&buf, &hasParam, "allowCleartextPasswords", "true") + } + + if cfg.AllowFallbackToPlaintext { + writeDSNParam(&buf, &hasParam, "allowFallbackToPlaintext", "true") + } + + if !cfg.AllowNativePasswords { + writeDSNParam(&buf, &hasParam, "allowNativePasswords", "false") + } + + if cfg.AllowOldPasswords { + writeDSNParam(&buf, &hasParam, "allowOldPasswords", "true") + } + + if !cfg.CheckConnLiveness { + writeDSNParam(&buf, &hasParam, "checkConnLiveness", "false") + } + + if cfg.ClientFoundRows { + writeDSNParam(&buf, &hasParam, "clientFoundRows", "true") + } + + if charsets := cfg.charsets; len(charsets) > 0 { + writeDSNParam(&buf, &hasParam, "charset", strings.Join(charsets, ",")) + } + + if col := cfg.Collation; col != "" { + writeDSNParam(&buf, &hasParam, "collation", col) + } + + if cfg.ColumnsWithAlias { + writeDSNParam(&buf, &hasParam, "columnsWithAlias", "true") + } + + if cfg.ConnectionAttributes != "" { + writeDSNParam(&buf, &hasParam, "connectionAttributes", url.QueryEscape(cfg.ConnectionAttributes)) + } + + if cfg.compress { + writeDSNParam(&buf, &hasParam, "compress", "true") + } + + if cfg.InterpolateParams { + writeDSNParam(&buf, &hasParam, "interpolateParams", "true") + } + + if cfg.Loc != time.UTC && cfg.Loc != nil { + writeDSNParam(&buf, &hasParam, "loc", url.QueryEscape(cfg.Loc.String())) + } + + if cfg.MultiStatements { + writeDSNParam(&buf, &hasParam, "multiStatements", "true") + } + + if cfg.ParseTime { + writeDSNParam(&buf, &hasParam, "parseTime", "true") + } + + if cfg.timeTruncate > 0 { + writeDSNParam(&buf, &hasParam, "timeTruncate", cfg.timeTruncate.String()) + } + + if cfg.ReadTimeout > 0 { + writeDSNParam(&buf, &hasParam, "readTimeout", cfg.ReadTimeout.String()) + } + + if cfg.RejectReadOnly { + writeDSNParam(&buf, &hasParam, "rejectReadOnly", "true") + } + + if len(cfg.ServerPubKey) > 0 { + writeDSNParam(&buf, &hasParam, "serverPubKey", url.QueryEscape(cfg.ServerPubKey)) + } + + if cfg.Timeout > 0 { + writeDSNParam(&buf, &hasParam, "timeout", cfg.Timeout.String()) + } + + if len(cfg.TLSConfig) > 0 { + writeDSNParam(&buf, &hasParam, "tls", url.QueryEscape(cfg.TLSConfig)) + } + + if cfg.WriteTimeout > 0 { + writeDSNParam(&buf, &hasParam, "writeTimeout", cfg.WriteTimeout.String()) + } + + if cfg.MaxAllowedPacket != defaultMaxAllowedPacket { + writeDSNParam(&buf, &hasParam, "maxAllowedPacket", strconv.Itoa(cfg.MaxAllowedPacket)) + } + + // other params + if cfg.Params != nil { + var params []string + for param := range cfg.Params { + params = append(params, param) + } + sort.Strings(params) + for _, param := range params { + writeDSNParam(&buf, &hasParam, param, url.QueryEscape(cfg.Params[param])) + } + } + + return buf.String() +} + +// ParseDSN parses the DSN string to a Config +func ParseDSN(dsn string) (cfg *Config, err error) { + // New config with some default values + cfg = NewConfig() + + // [user[:password]@][net[(addr)]]/dbname[?param1=value1¶mN=valueN] + // Find the last '/' (since the password or the net addr might contain a '/') + foundSlash := false + for i := len(dsn) - 1; i >= 0; i-- { + if dsn[i] == '/' { + foundSlash = true + var j, k int + + // left part is empty if i <= 0 + if i > 0 { + // [username[:password]@][protocol[(address)]] + // Find the last '@' in dsn[:i] + for j = i; j >= 0; j-- { + if dsn[j] == '@' { + // username[:password] + // Find the first ':' in dsn[:j] + for k = 0; k < j; k++ { // We cannot use k = range j here, because we use dsn[:k] below + if dsn[k] == ':' { + cfg.Passwd = dsn[k+1 : j] + break + } + } + cfg.User = dsn[:k] + + break + } + } + + // [protocol[(address)]] + // Find the first '(' in dsn[j+1:i] + for k = j + 1; k < i; k++ { + if dsn[k] == '(' { + // dsn[i-1] must be == ')' if an address is specified + if dsn[i-1] != ')' { + if strings.ContainsRune(dsn[k+1:i], ')') { + return nil, errInvalidDSNUnescaped + } + return nil, errInvalidDSNAddr + } + cfg.Addr = dsn[k+1 : i-1] + break + } + } + cfg.Net = dsn[j+1 : k] + } + + // dbname[?param1=value1&...¶mN=valueN] + // Find the first '?' in dsn[i+1:] + for j = i + 1; j < len(dsn); j++ { + if dsn[j] == '?' { + if err = parseDSNParams(cfg, dsn[j+1:]); err != nil { + return + } + break + } + } + + dbname := dsn[i+1 : j] + if cfg.DBName, err = url.PathUnescape(dbname); err != nil { + return nil, fmt.Errorf("invalid dbname %q: %w", dbname, err) + } + + break + } + } + + if !foundSlash && len(dsn) > 0 { + return nil, errInvalidDSNNoSlash + } + + if err = cfg.normalize(); err != nil { + return nil, err + } + return +} + +// parseDSNParams parses the DSN "query string" +// Values must be url.QueryEscape'ed +func parseDSNParams(cfg *Config, params string) (err error) { + for v := range strings.SplitSeq(params, "&") { + key, value, found := strings.Cut(v, "=") + if !found { + continue + } + + // cfg params + switch key { + // Disable INFILE allowlist / enable all files + case "allowAllFiles": + var isBool bool + cfg.AllowAllFiles, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Use cleartext authentication mode (MySQL 5.5.10+) + case "allowCleartextPasswords": + var isBool bool + cfg.AllowCleartextPasswords, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Allow fallback to unencrypted connection if server does not support TLS + case "allowFallbackToPlaintext": + var isBool bool + cfg.AllowFallbackToPlaintext, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Use native password authentication + case "allowNativePasswords": + var isBool bool + cfg.AllowNativePasswords, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Use old authentication mode (pre MySQL 4.1) + case "allowOldPasswords": + var isBool bool + cfg.AllowOldPasswords, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Check connections for Liveness before using them + case "checkConnLiveness": + var isBool bool + cfg.CheckConnLiveness, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Switch "rowsAffected" mode + case "clientFoundRows": + var isBool bool + cfg.ClientFoundRows, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // charset + case "charset": + cfg.charsets = strings.Split(value, ",") + + // Collation + case "collation": + cfg.Collation = value + + case "columnsWithAlias": + var isBool bool + cfg.ColumnsWithAlias, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Compression + case "compress": + var isBool bool + cfg.compress, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Enable client side placeholder substitution + case "interpolateParams": + var isBool bool + cfg.InterpolateParams, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Time Location + case "loc": + if value, err = url.QueryUnescape(value); err != nil { + return + } + cfg.Loc, err = time.LoadLocation(value) + if err != nil { + return + } + + // multiple statements in one query + case "multiStatements": + var isBool bool + cfg.MultiStatements, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // time.Time parsing + case "parseTime": + var isBool bool + cfg.ParseTime, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // time.Time truncation + case "timeTruncate": + cfg.timeTruncate, err = time.ParseDuration(value) + if err != nil { + return fmt.Errorf("invalid timeTruncate value: %v, error: %w", value, err) + } + + // I/O read Timeout + case "readTimeout": + cfg.ReadTimeout, err = time.ParseDuration(value) + if err != nil { + return + } + + // Reject read-only connections + case "rejectReadOnly": + var isBool bool + cfg.RejectReadOnly, isBool = readBool(value) + if !isBool { + return errors.New("invalid bool value: " + value) + } + + // Server public key + case "serverPubKey": + name, err := url.QueryUnescape(value) + if err != nil { + return fmt.Errorf("invalid value for server pub key name: %v", err) + } + cfg.ServerPubKey = name + + // Strict mode + case "strict": + panic("strict mode has been removed. See https://github.com/go-sql-driver/mysql/wiki/strict-mode") + + // Dial Timeout + case "timeout": + cfg.Timeout, err = time.ParseDuration(value) + if err != nil { + return + } + + // TLS-Encryption + case "tls": + boolValue, isBool := readBool(value) + if isBool { + if boolValue { + cfg.TLSConfig = "true" + } else { + cfg.TLSConfig = "false" + } + } else if vl := strings.ToLower(value); vl == "skip-verify" || vl == "preferred" { + cfg.TLSConfig = vl + } else { + name, err := url.QueryUnescape(value) + if err != nil { + return fmt.Errorf("invalid value for TLS config name: %v", err) + } + cfg.TLSConfig = name + } + + // I/O write Timeout + case "writeTimeout": + cfg.WriteTimeout, err = time.ParseDuration(value) + if err != nil { + return + } + case "maxAllowedPacket": + cfg.MaxAllowedPacket, err = strconv.Atoi(value) + if err != nil { + return + } + + // Connection attributes + case "connectionAttributes": + connectionAttributes, err := url.QueryUnescape(value) + if err != nil { + return fmt.Errorf("invalid connectionAttributes value: %v", err) + } + cfg.ConnectionAttributes = connectionAttributes + + default: + // lazy init + if cfg.Params == nil { + cfg.Params = make(map[string]string) + } + + if cfg.Params[key], err = url.QueryUnescape(value); err != nil { + return + } + } + } + + return +} + +func ensureHavePort(addr string) string { + if _, _, err := net.SplitHostPort(addr); err != nil { + return net.JoinHostPort(addr, "3306") + } + return addr +} diff --git a/vendor/github.com/go-sql-driver/mysql/errors.go b/vendor/github.com/go-sql-driver/mysql/errors.go new file mode 100644 index 0000000000..584617b118 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/errors.go @@ -0,0 +1,83 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "errors" + "fmt" + "log" + "os" +) + +// Various errors the driver might return. Can change between driver versions. +var ( + ErrInvalidConn = errors.New("invalid connection") + ErrMalformPkt = errors.New("malformed packet") + ErrNoTLS = errors.New("TLS requested but server does not support TLS") + ErrCleartextPassword = errors.New("this user requires clear text authentication. If you still want to use it, please add 'allowCleartextPasswords=1' to your DSN") + ErrNativePassword = errors.New("this user requires mysql native password authentication") + ErrOldPassword = errors.New("this user requires old password authentication. If you still want to use it, please add 'allowOldPasswords=1' to your DSN. See also https://github.com/go-sql-driver/mysql/wiki/old_passwords") + ErrUnknownPlugin = errors.New("this authentication plugin is not supported") + ErrOldProtocol = errors.New("MySQL server does not support required protocol 41+") + ErrPktSync = errors.New("commands out of sync. You can't run this command now") + ErrPktSyncMul = errors.New("commands out of sync. Did you run multiple statements at once?") + ErrPktTooLarge = errors.New("packet for query is too large. Try adjusting the `Config.MaxAllowedPacket`") + ErrBusyBuffer = errors.New("busy buffer") + + // errBadConnNoWrite is used for connection errors where nothing was sent to the database yet. + // If this happens first in a function starting a database interaction, it should be replaced by driver.ErrBadConn + // to trigger a resend. Use mc.markBadConn(err) to do this. + // See https://github.com/go-sql-driver/mysql/pull/302 + errBadConnNoWrite = errors.New("bad connection") +) + +var defaultLogger = Logger(log.New(os.Stderr, "[mysql] ", log.Ldate|log.Ltime)) + +// Logger is used to log critical error messages. +type Logger interface { + Print(v ...any) +} + +// NopLogger is a nop implementation of the Logger interface. +type NopLogger struct{} + +// Print implements Logger interface. +func (nl *NopLogger) Print(_ ...any) {} + +// SetLogger is used to set the default logger for critical errors. +// The initial logger is os.Stderr. +func SetLogger(logger Logger) error { + if logger == nil { + return errors.New("logger is nil") + } + defaultLogger = logger + return nil +} + +// MySQLError is an error type which represents a single MySQL error +type MySQLError struct { + Number uint16 + SQLState [5]byte + Message string +} + +func (me *MySQLError) Error() string { + if me.SQLState != [5]byte{} { + return fmt.Sprintf("Error %d (%s): %s", me.Number, me.SQLState, me.Message) + } + + return fmt.Sprintf("Error %d: %s", me.Number, me.Message) +} + +func (me *MySQLError) Is(err error) bool { + if merr, ok := err.(*MySQLError); ok { + return merr.Number == me.Number + } + return false +} diff --git a/vendor/github.com/go-sql-driver/mysql/fields.go b/vendor/github.com/go-sql-driver/mysql/fields.go new file mode 100644 index 0000000000..ee9d964171 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/fields.go @@ -0,0 +1,228 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2017 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql" + "reflect" +) + +func (mf *mysqlField) typeDatabaseName() string { + switch mf.fieldType { + case fieldTypeBit: + return "BIT" + case fieldTypeBLOB: + if mf.charSet != binaryCollationID { + return "TEXT" + } + return "BLOB" + case fieldTypeDate: + return "DATE" + case fieldTypeDateTime: + return "DATETIME" + case fieldTypeDecimal: + return "DECIMAL" + case fieldTypeDouble: + return "DOUBLE" + case fieldTypeEnum: + return "ENUM" + case fieldTypeFloat: + return "FLOAT" + case fieldTypeGeometry: + return "GEOMETRY" + case fieldTypeInt24: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED MEDIUMINT" + } + return "MEDIUMINT" + case fieldTypeJSON: + return "JSON" + case fieldTypeLong: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED INT" + } + return "INT" + case fieldTypeLongBLOB: + if mf.charSet != binaryCollationID { + return "LONGTEXT" + } + return "LONGBLOB" + case fieldTypeLongLong: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED BIGINT" + } + return "BIGINT" + case fieldTypeMediumBLOB: + if mf.charSet != binaryCollationID { + return "MEDIUMTEXT" + } + return "MEDIUMBLOB" + case fieldTypeNewDate: + return "DATE" + case fieldTypeNewDecimal: + return "DECIMAL" + case fieldTypeNULL: + return "NULL" + case fieldTypeSet: + return "SET" + case fieldTypeShort: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED SMALLINT" + } + return "SMALLINT" + case fieldTypeString: + if mf.flags&flagEnum != 0 { + return "ENUM" + } else if mf.flags&flagSet != 0 { + return "SET" + } + if mf.charSet == binaryCollationID { + return "BINARY" + } + return "CHAR" + case fieldTypeTime: + return "TIME" + case fieldTypeTimestamp: + return "TIMESTAMP" + case fieldTypeTiny: + if mf.flags&flagUnsigned != 0 { + return "UNSIGNED TINYINT" + } + return "TINYINT" + case fieldTypeTinyBLOB: + if mf.charSet != binaryCollationID { + return "TINYTEXT" + } + return "TINYBLOB" + case fieldTypeVarChar: + if mf.charSet == binaryCollationID { + return "VARBINARY" + } + return "VARCHAR" + case fieldTypeVarString: + if mf.charSet == binaryCollationID { + return "VARBINARY" + } + return "VARCHAR" + case fieldTypeYear: + return "YEAR" + case fieldTypeVector: + return "VECTOR" + default: + return "" + } +} + +var ( + scanTypeFloat32 = reflect.TypeFor[float32]() + scanTypeFloat64 = reflect.TypeFor[float64]() + scanTypeInt8 = reflect.TypeFor[int8]() + scanTypeInt16 = reflect.TypeFor[int16]() + scanTypeInt32 = reflect.TypeFor[int32]() + scanTypeInt64 = reflect.TypeFor[int64]() + scanTypeNullFloat = reflect.TypeFor[sql.NullFloat64]() + scanTypeNullInt = reflect.TypeFor[sql.NullInt64]() + scanTypeNullUint = reflect.TypeFor[sql.Null[uint64]]() + scanTypeNullTime = reflect.TypeFor[sql.NullTime]() + scanTypeUint8 = reflect.TypeFor[uint8]() + scanTypeUint16 = reflect.TypeFor[uint16]() + scanTypeUint32 = reflect.TypeFor[uint32]() + scanTypeUint64 = reflect.TypeFor[uint64]() + scanTypeString = reflect.TypeFor[string]() + scanTypeNullString = reflect.TypeFor[sql.NullString]() + scanTypeBytes = reflect.TypeFor[[]byte]() + scanTypeUnknown = reflect.TypeFor[*any]() +) + +type mysqlField struct { + tableName string + name string + length uint32 + flags fieldFlag + fieldType fieldType + decimals byte + charSet uint8 +} + +func (mf *mysqlField) scanType() reflect.Type { + switch mf.fieldType { + case fieldTypeTiny: + if mf.flags&flagNotNULL != 0 { + if mf.flags&flagUnsigned != 0 { + return scanTypeUint8 + } + return scanTypeInt8 + } + return scanTypeNullInt + + case fieldTypeShort, fieldTypeYear: + if mf.flags&flagNotNULL != 0 { + if mf.flags&flagUnsigned != 0 { + return scanTypeUint16 + } + return scanTypeInt16 + } + return scanTypeNullInt + + case fieldTypeInt24, fieldTypeLong: + if mf.flags&flagNotNULL != 0 { + if mf.flags&flagUnsigned != 0 { + return scanTypeUint32 + } + return scanTypeInt32 + } + return scanTypeNullInt + + case fieldTypeLongLong: + if mf.flags&flagNotNULL != 0 { + if mf.flags&flagUnsigned != 0 { + return scanTypeUint64 + } + return scanTypeInt64 + } + if mf.flags&flagUnsigned != 0 { + return scanTypeNullUint + } + return scanTypeNullInt + + case fieldTypeFloat: + if mf.flags&flagNotNULL != 0 { + return scanTypeFloat32 + } + return scanTypeNullFloat + + case fieldTypeDouble: + if mf.flags&flagNotNULL != 0 { + return scanTypeFloat64 + } + return scanTypeNullFloat + + case fieldTypeBit, fieldTypeTinyBLOB, fieldTypeMediumBLOB, fieldTypeLongBLOB, + fieldTypeBLOB, fieldTypeVarString, fieldTypeString, fieldTypeGeometry, fieldTypeVector: + if mf.charSet == binaryCollationID { + return scanTypeBytes + } + fallthrough + case fieldTypeDecimal, fieldTypeNewDecimal, fieldTypeVarChar, + fieldTypeEnum, fieldTypeSet, fieldTypeJSON, fieldTypeTime: + if mf.flags&flagNotNULL != 0 { + return scanTypeString + } + return scanTypeNullString + + case fieldTypeDate, fieldTypeNewDate, + fieldTypeTimestamp, fieldTypeDateTime: + // NullTime is always returned for more consistent behavior as it can + // handle both cases of parseTime regardless if the field is nullable. + return scanTypeNullTime + + default: + return scanTypeUnknown + } +} diff --git a/vendor/github.com/go-sql-driver/mysql/infile.go b/vendor/github.com/go-sql-driver/mysql/infile.go new file mode 100644 index 0000000000..597b5e7f67 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/infile.go @@ -0,0 +1,181 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "fmt" + "io" + "os" + "strings" + "sync" +) + +var ( + fileRegister map[string]struct{} + fileRegisterLock sync.RWMutex + readerRegister map[string]func() io.Reader + readerRegisterLock sync.RWMutex +) + +// RegisterLocalFile adds the given file to the file allowlist, +// so that it can be used by "LOAD DATA LOCAL INFILE ". +// Alternatively you can allow the use of all local files with +// the DSN parameter 'allowAllFiles=true' +// +// filePath := "/home/gopher/data.csv" +// mysql.RegisterLocalFile(filePath) +// err := db.Exec("LOAD DATA LOCAL INFILE '" + filePath + "' INTO TABLE foo") +// if err != nil { +// ... +func RegisterLocalFile(filePath string) { + fileRegisterLock.Lock() + // lazy map init + if fileRegister == nil { + fileRegister = make(map[string]struct{}) + } + + fileRegister[strings.Trim(filePath, `"`)] = struct{}{} + fileRegisterLock.Unlock() +} + +// DeregisterLocalFile removes the given filepath from the allowlist. +func DeregisterLocalFile(filePath string) { + fileRegisterLock.Lock() + delete(fileRegister, strings.Trim(filePath, `"`)) + fileRegisterLock.Unlock() +} + +// RegisterReaderHandler registers a handler function which is used +// to receive a io.Reader. +// The Reader can be used by "LOAD DATA LOCAL INFILE Reader::". +// If the handler returns a io.ReadCloser Close() is called when the +// request is finished. +// +// mysql.RegisterReaderHandler("data", func() io.Reader { +// var csvReader io.Reader // Some Reader that returns CSV data +// ... // Open Reader here +// return csvReader +// }) +// err := db.Exec("LOAD DATA LOCAL INFILE 'Reader::data' INTO TABLE foo") +// if err != nil { +// ... +func RegisterReaderHandler(name string, handler func() io.Reader) { + readerRegisterLock.Lock() + // lazy map init + if readerRegister == nil { + readerRegister = make(map[string]func() io.Reader) + } + + readerRegister[name] = handler + readerRegisterLock.Unlock() +} + +// DeregisterReaderHandler removes the ReaderHandler function with +// the given name from the registry. +func DeregisterReaderHandler(name string) { + readerRegisterLock.Lock() + delete(readerRegister, name) + readerRegisterLock.Unlock() +} + +func deferredClose(err *error, closer io.Closer) { + closeErr := closer.Close() + if *err == nil { + *err = closeErr + } +} + +const defaultPacketSize = 16 * 1024 // 16KB is small enough for disk readahead and large enough for TCP + +func (mc *okHandler) handleInFileRequest(name string) (err error) { + var rdr io.Reader + packetSize := min(mc.maxWriteSize, defaultPacketSize) + + if idx := strings.Index(name, "Reader::"); idx == 0 || (idx > 0 && name[idx-1] == '/') { // io.Reader + // The server might return an an absolute path. See issue #355. + name = name[idx+8:] + + readerRegisterLock.RLock() + handler, inMap := readerRegister[name] + readerRegisterLock.RUnlock() + + if inMap { + rdr = handler() + if rdr != nil { + if cl, ok := rdr.(io.Closer); ok { + defer deferredClose(&err, cl) + } + } else { + err = fmt.Errorf("reader '%s' is ", name) + } + } else { + err = fmt.Errorf("reader '%s' is not registered", name) + } + } else { // File + name = strings.Trim(name, `"`) + fileRegisterLock.RLock() + _, exists := fileRegister[name] + fileRegisterLock.RUnlock() + if mc.cfg.AllowAllFiles || exists { + var file *os.File + var fi os.FileInfo + + if file, err = os.Open(name); err == nil { + defer deferredClose(&err, file) + + // get file size + if fi, err = file.Stat(); err == nil { + rdr = file + if fileSize := int(fi.Size()); fileSize < packetSize { + packetSize = fileSize + } + } + } + } else { + err = fmt.Errorf("local file '%s' is not registered", name) + } + } + + // send content packets + var data []byte + + // if packetSize == 0, the Reader contains no data + if err == nil && packetSize > 0 { + data = make([]byte, 4+packetSize) + var n int + for err == nil { + n, err = rdr.Read(data[4:]) + if n > 0 { + if ioErr := mc.conn().writePacket(data[:4+n]); ioErr != nil { + return ioErr + } + } + } + if err == io.EOF { + err = nil + } + } + + // send empty packet (termination) + if data == nil { + data = make([]byte, 4) + } + if ioErr := mc.conn().writePacket(data[:4]); ioErr != nil { + return ioErr + } + mc.conn().syncSequence() + + // read OK packet + if err == nil { + return mc.readResultOK() + } + + mc.conn().readPacket() + return err +} diff --git a/vendor/github.com/go-sql-driver/mysql/nulltime.go b/vendor/github.com/go-sql-driver/mysql/nulltime.go new file mode 100644 index 0000000000..316a48aae1 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/nulltime.go @@ -0,0 +1,71 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql" + "database/sql/driver" + "fmt" + "time" +) + +// NullTime represents a time.Time that may be NULL. +// NullTime implements the Scanner interface so +// it can be used as a scan destination: +// +// var nt NullTime +// err := db.QueryRow("SELECT time FROM foo WHERE id=?", id).Scan(&nt) +// ... +// if nt.Valid { +// // use nt.Time +// } else { +// // NULL value +// } +// +// # This NullTime implementation is not driver-specific +// +// Deprecated: NullTime doesn't honor the loc DSN parameter. +// NullTime.Scan interprets a time as UTC, not the loc DSN parameter. +// Use sql.NullTime instead. +type NullTime sql.NullTime + +// Scan implements the Scanner interface. +// The value type must be time.Time or string / []byte (formatted time-string), +// otherwise Scan fails. +func (nt *NullTime) Scan(value any) (err error) { + if value == nil { + nt.Time, nt.Valid = time.Time{}, false + return + } + + switch v := value.(type) { + case time.Time: + nt.Time, nt.Valid = v, true + return + case []byte: + nt.Time, err = parseDateTime(v, time.UTC) + nt.Valid = (err == nil) + return + case string: + nt.Time, err = parseDateTime([]byte(v), time.UTC) + nt.Valid = (err == nil) + return + } + + nt.Valid = false + return fmt.Errorf("can't convert %T to time.Time", value) +} + +// Value implements the driver Valuer interface. +func (nt NullTime) Value() (driver.Value, error) { + if !nt.Valid { + return nil, nil + } + return nt.Time, nil +} diff --git a/vendor/github.com/go-sql-driver/mysql/packets.go b/vendor/github.com/go-sql-driver/mysql/packets.go new file mode 100644 index 0000000000..d0b21b06c9 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/packets.go @@ -0,0 +1,1451 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "bytes" + "crypto/tls" + "database/sql/driver" + "encoding/binary" + "encoding/json" + "fmt" + "io" + "math" + "os" + "strconv" + "time" +) + +// MySQL client/server protocol documentations. +// https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_PROTOCOL.html +// https://mariadb.com/kb/en/clientserver-protocol/ + +// read n bytes from mc.buf +func (mc *mysqlConn) readNext(n int) ([]byte, error) { + if mc.buf.len() < n { + err := mc.buf.fill(n, mc.readWithTimeout) + if err != nil { + return nil, err + } + } + return mc.buf.readNext(n), nil +} + +// Read packet to buffer 'data' +func (mc *mysqlConn) readPacket() ([]byte, error) { + var prevData []byte + invalidSequence := false + + readNext := mc.readNext + if mc.compress { + readNext = mc.compIO.readNext + } + + for { + // read packet header + data, err := readNext(4) + if err != nil { + mc.close() + if cerr := mc.canceled.Value(); cerr != nil { + return nil, cerr + } + mc.log(err) + return nil, ErrInvalidConn + } + + // packet length [24 bit] + pktLen := getUint24(data[:3]) + seq := data[3] + + // check packet sync [8 bit] + if seq != mc.sequence { + mc.log(fmt.Sprintf("[warn] unexpected sequence nr: expected %v, got %v", mc.sequence, seq)) + // MySQL and MariaDB doesn't check packet nr in compressed packet. + if !mc.compress { + // For large packets, we stop reading as soon as sync error. + if len(prevData) > 0 { + mc.close() + return nil, ErrPktSyncMul + } + invalidSequence = true + } + } + mc.sequence = seq + 1 + + // packets with length 0 terminate a previous packet which is a + // multiple of (2^24)-1 bytes long + if pktLen == 0 { + // there was no previous packet + if prevData == nil { + mc.log(ErrMalformPkt) + mc.close() + return nil, ErrInvalidConn + } + return prevData, nil + } + + // read packet body [pktLen bytes] + data, err = readNext(pktLen) + if err != nil { + mc.close() + if cerr := mc.canceled.Value(); cerr != nil { + return nil, cerr + } + mc.log(err) + return nil, ErrInvalidConn + } + + // return data if this was the last packet + if pktLen < maxPacketSize { + // zero allocations for non-split packets + if prevData != nil { + data = append(prevData, data...) + } + if invalidSequence { + mc.close() + // return sync error only for regular packet. + // error packets may have wrong sequence number. + if data[0] != iERR { + return nil, ErrPktSync + } + } + return data, nil + } + + prevData = append(prevData, data...) + } +} + +// Write packet buffer 'data' +func (mc *mysqlConn) writePacket(data []byte) error { + pktLen := len(data) - 4 + if pktLen > mc.maxAllowedPacket { + return ErrPktTooLarge + } + + writeFunc := mc.writeWithTimeout + if mc.compress { + writeFunc = mc.compIO.writePackets + } + + for { + size := min(maxPacketSize, pktLen) + putUint24(data[:3], size) + data[3] = mc.sequence + + // Write packet + if debug { + fmt.Fprintf(os.Stderr, "writePacket: size=%v seq=%v\n", size, mc.sequence) + } + + n, err := writeFunc(data[:4+size]) + if err != nil { + mc.cleanup() + if cerr := mc.canceled.Value(); cerr != nil { + return cerr + } + if n == 0 && pktLen == len(data)-4 { + // only for the first loop iteration when nothing was written yet + mc.log(err) + return errBadConnNoWrite + } else { + return err + } + } + if n != 4+size { + // io.Writer(b) must return a non-nil error if it cannot write len(b) bytes. + // The io.ErrShortWrite error is used to indicate that this rule has not been followed. + mc.cleanup() + return io.ErrShortWrite + } + + mc.sequence++ + if size != maxPacketSize { + return nil + } + pktLen -= size + data = data[size:] + } +} + +/****************************************************************************** +* Initialization Process * +******************************************************************************/ + +// Handshake Initialization Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_handshake_v10.html +// https://mariadb.com/kb/en/connection/#initial-handshake-packet +func (mc *mysqlConn) readHandshakePacket() (data []byte, capabilities capabilityFlag, extendedCapabilities extendedCapabilityFlag, plugin string, err error) { + data, err = mc.readPacket() + if err != nil { + return + } + + if data[0] == iERR { + err = mc.handleErrorPacket(data) + return + } + + // protocol version [1 byte] + if data[0] < minProtocolVersion { + return nil, 0, 0, "", fmt.Errorf( + "unsupported protocol version %d. Version %d or higher is required", + data[0], + minProtocolVersion, + ) + } + + // server version [null terminated string] + // connection id [4 bytes] + pos := 1 + bytes.IndexByte(data[1:], 0x00) + 1 + 4 + + // first part of the password cipher [8 bytes] + authData := data[pos : pos+8] + + // (filler) always 0x00 [1 byte] + pos += 8 + 1 + + // capability flags (lower 2 bytes) [2 bytes] + capabilities = capabilityFlag(binary.LittleEndian.Uint16(data[pos : pos+2])) + if capabilities&clientProtocol41 == 0 { + return nil, capabilities, 0, "", ErrOldProtocol + } + if capabilities&clientSSL == 0 && mc.cfg.TLS != nil { + if mc.cfg.AllowFallbackToPlaintext { + mc.cfg.TLS = nil + } else { + return nil, capabilities, 0, "", ErrNoTLS + } + } + pos += 2 + + if len(data) > pos { + // character set [1 byte] + // status flags [2 bytes] + pos += 3 + // capability flags (upper 2 bytes) [2 bytes] + capabilities |= capabilityFlag(binary.LittleEndian.Uint16(data[pos:pos+2])) << 16 + pos += 2 + // length of auth-plugin-data [1 byte] + // reserved (all [00]) [6 bytes] + pos += 7 + if capabilities&clientMySQL == 0 { + // MariaDB server extended flag + extendedCapabilities = extendedCapabilityFlag(binary.LittleEndian.Uint32(data[pos : pos+4])) + } + pos += 4 + + // second part of the password cipher [minimum 13 bytes], + // where len=MAX(13, length of auth-plugin-data - 8) + // + // The web documentation is ambiguous about the length. However, + // according to mysql-5.7/sql/auth/sql_authentication.cc line 538, + // the 13th byte is "\0 byte, terminating the second part of + // a scramble". So the second part of the password cipher is + // a NULL terminated string that's at least 13 bytes with the + // last byte being NULL. + // + // The official Python library uses the fixed length 12 + // which seems to work but technically could have a hidden bug. + authData = append(authData, data[pos:pos+12]...) + pos += 13 + + // EOF if version (>= 5.5.7 and < 5.5.10) or (>= 5.6.0 and < 5.6.2) + // \NUL otherwise + if end := bytes.IndexByte(data[pos:], 0x00); end != -1 { + plugin = string(data[pos : pos+end]) + } else { + plugin = string(data[pos:]) + } + + // make a memory safe copy of the cipher slice + var b [20]byte + copy(b[:], authData) + return b[:], capabilities, extendedCapabilities, plugin, nil + } + + // make a memory safe copy of the cipher slice + var b [8]byte + copy(b[:], authData) + return b[:], capabilities, 0, plugin, nil +} + +// initCapabilities initializes the capabilities based on server support and configuration +func (mc *mysqlConn) initCapabilities(serverCapabilities capabilityFlag, serverExtCapabilities extendedCapabilityFlag, cfg *Config) { + clientCapabilities := + clientMySQL | + clientLongFlag | + clientProtocol41 | + clientSecureConn | + clientTransactions | + clientPluginAuthLenEncClientData | + clientLocalFiles | + clientPluginAuth | + clientMultiResults | + clientConnectAttrs | + clientDeprecateEOF + + if cfg.ClientFoundRows { + clientCapabilities |= clientFoundRows + } + if cfg.compress { + clientCapabilities |= clientCompress + } + // To enable TLS / SSL + if mc.cfg.TLS != nil { + clientCapabilities |= clientSSL + } + + if mc.cfg.MultiStatements { + clientCapabilities |= clientMultiStatements + } + if n := len(cfg.DBName); n > 0 { + clientCapabilities |= clientConnectWithDB + } + + // only keep client capabilities that server have + mc.capabilities = clientCapabilities & serverCapabilities + + // set MariaDB extended clientCacheMetadata capability if server support it + mc.extCapabilities = clientCacheMetadata & serverExtCapabilities +} + +// Client Authentication Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_handshake_response.html +func (mc *mysqlConn) writeHandshakeResponsePacket(authResp []byte, plugin string) error { + // packet header 4 + // capabilities 4 + // maxPacketSize 4 + // collation id 1 + // filler 23 + data, err := mc.buf.takeSmallBuffer(4*3 + 24) + if err != nil { + mc.cleanup() + return err + } + _ = data[4*3+23] // boundery check + + // clientCapabilities [32 bit] + binary.LittleEndian.PutUint32(data[4:], uint32(mc.capabilities)) + + // MaxPacketSize [32 bit] (none) + binary.LittleEndian.PutUint32(data[8:], 0) + + // Collation ID [1 byte] + data[12] = defaultCollationID + if cname := mc.cfg.Collation; cname != "" { + colID, ok := collations[cname] + if ok { + data[12] = colID + } else if len(mc.cfg.charsets) > 0 { + // When cfg.charset is set, the collation is set by `SET NAMES COLLATE `. + return fmt.Errorf("unknown collation: %q", cname) + } + } + + // Filler [23 bytes] (all 0x00) + // or filler 19bytes + mariadb extCapabilities + pos := 13 + if mc.capabilities&clientMySQL == 0 { + for ; pos < 13+19; pos++ { + data[pos] = 0 + } + // MariaDB Extended Capabilities + binary.LittleEndian.PutUint32(data[13+19:], uint32(mc.extCapabilities)) + } else { + for ; pos < 13+23; pos++ { + data[pos] = 0 + } + } + + // SSL Connection Request Packet + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_ssl_request.html + // https://mariadb.com/kb/en/connection/#sslrequest-packet + if mc.cfg.TLS != nil { + // Send TLS / SSL request packet + if err := mc.writePacket(data); err != nil { + return err + } + + // Switch to TLS + tlsConn := tls.Client(mc.netConn, mc.cfg.TLS) + if err := tlsConn.Handshake(); err != nil { + if cerr := mc.canceled.Value(); cerr != nil { + return cerr + } + return err + } + mc.netConn = tlsConn + } + + // User [null terminated string] + if len(mc.cfg.User) > 0 { + data = append(data, mc.cfg.User...) + } + data = append(data, 0) + + // Auth Data [length encoded integer] + data = appendLengthEncodedInteger(data, uint64(len(authResp))) + data = append(data, authResp...) + + // Database name [null terminated string] + if mc.capabilities&clientConnectWithDB != 0 { + data = append(data, mc.cfg.DBName...) + data = append(data, 0) + } + + data = append(data, plugin...) + data = append(data, 0) + + // Connection Attributes + if mc.capabilities&clientConnectAttrs != 0 { + connAttrsLen := len(mc.connector.encodedAttributes) + data = appendLengthEncodedInteger(data, uint64(connAttrsLen)) + data = append(data, mc.connector.encodedAttributes...) + } + + // Send Auth packet + return mc.writePacket(data) +} + +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_auth_switch_response.html +func (mc *mysqlConn) writeAuthSwitchPacket(authData []byte) error { + pktLen := 4 + len(authData) + data, err := mc.buf.takeBuffer(pktLen) + if err != nil { + mc.cleanup() + return err + } + + // Add the auth data [EOF] + copy(data[4:], authData) + return mc.writePacket(data) +} + +/****************************************************************************** +* Command Packets * +******************************************************************************/ + +func (mc *mysqlConn) writeCommandPacket(command byte) error { + // Reset Packet Sequence + mc.resetSequence() + + data, err := mc.buf.takeSmallBuffer(4 + 1) + if err != nil { + return err + } + + // Add command byte + data[4] = command + + // Send CMD packet + err = mc.writePacket(data) + mc.syncSequence() + return err +} + +func (mc *mysqlConn) writeCommandPacketStr(command byte, arg string) error { + // Reset Packet Sequence + mc.resetSequence() + + pktLen := 1 + len(arg) + data, err := mc.buf.takeBuffer(pktLen + 4) + if err != nil { + return err + } + + // Add command byte + data[4] = command + + // Add arg + copy(data[5:], arg) + + // Send CMD packet + err = mc.writePacket(data) + mc.syncSequence() + return err +} + +func (mc *mysqlConn) writeCommandPacketUint32(command byte, arg uint32) error { + // Reset Packet Sequence + mc.resetSequence() + + data, err := mc.buf.takeSmallBuffer(4 + 1 + 4) + if err != nil { + return err + } + + // Add command byte + data[4] = command + + // Add arg [32 bit] + binary.LittleEndian.PutUint32(data[5:], arg) + + // Send CMD packet + err = mc.writePacket(data) + mc.syncSequence() + return err +} + +/****************************************************************************** +* Result Packets * +******************************************************************************/ + +func (mc *mysqlConn) readAuthResult() ([]byte, string, error) { + data, err := mc.readPacket() + if err != nil { + return nil, "", err + } + + // packet indicator + switch data[0] { + + case iOK: + // resultUnchanged, since auth happens before any queries or + // commands have been executed. + return nil, "", mc.resultUnchanged().handleOkPacket(data) + + case iAuthMoreData: + return data[1:], "", err + + case iEOF: + if len(data) == 1 { + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_old_auth_switch_request.html + return nil, "mysql_old_password", nil + } + pluginEndIndex := bytes.IndexByte(data, 0x00) + if pluginEndIndex < 0 { + return nil, "", ErrMalformPkt + } + plugin := string(data[1:pluginEndIndex]) + authData := data[pluginEndIndex+1:] + if len(authData) > 0 && authData[len(authData)-1] == 0 { + authData = authData[:len(authData)-1] + } + return authData, plugin, nil + + default: // Error otherwise + return nil, "", mc.handleErrorPacket(data) + } +} + +// Returns error if Packet is not a 'Result OK'-Packet +func (mc *okHandler) readResultOK() error { + data, err := mc.conn().readPacket() + if err != nil { + return err + } + + if data[0] == iOK { + return mc.handleOkPacket(data) + } + return mc.conn().handleErrorPacket(data) +} + +// Result Set Header Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response.html +func (mc *okHandler) readResultSetHeaderPacket() (int, bool, error) { + // handleOkPacket replaces both values; other cases leave the values unchanged. + mc.result.affectedRows = append(mc.result.affectedRows, 0) + mc.result.insertIds = append(mc.result.insertIds, 0) + + data, err := mc.conn().readPacket() + if err != nil { + return 0, false, err + } + + switch data[0] { + case iOK: + return 0, false, mc.handleOkPacket(data) + + case iERR: + return 0, false, mc.conn().handleErrorPacket(data) + + case iLocalInFile: + return 0, false, mc.handleInFileRequest(string(data[1:])) + } + + // column count + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset.html + // https://mariadb.com/kb/en/result-set-packets/#column-count-packet + num, _, len := readLengthEncodedInteger(data) + + if mc.extCapabilities&clientCacheMetadata != 0 { + return int(num), data[len] == 0x01, nil + } + // ignore remaining data in the packet. see #1478. + return int(num), true, nil +} + +// Error Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_err_packet.html +func (mc *mysqlConn) handleErrorPacket(data []byte) error { + if data[0] != iERR { + return ErrMalformPkt + } + + // 0xff [1 byte] + + // Error Number [16 bit uint] + errno := binary.LittleEndian.Uint16(data[1:3]) + + // 1792: ER_CANT_EXECUTE_IN_READ_ONLY_TRANSACTION + // 1290: ER_OPTION_PREVENTS_STATEMENT (returned by Aurora during failover) + // 1836: ER_READ_ONLY_MODE + if (errno == 1792 || errno == 1290 || errno == 1836) && mc.cfg.RejectReadOnly { + // Oops; we are connected to a read-only connection, and won't be able + // to issue any write statements. Since RejectReadOnly is configured, + // we throw away this connection hoping this one would have write + // permission. This is specifically for a possible race condition + // during failover (e.g. on AWS Aurora). See README.md for more. + // + // We explicitly close the connection before returning + // driver.ErrBadConn to ensure that `database/sql` purges this + // connection and initiates a new one for next statement next time. + mc.Close() + return driver.ErrBadConn + } + + me := &MySQLError{Number: errno} + + pos := 3 + + // SQL State [optional: # + 5bytes string] + if data[3] == 0x23 { + copy(me.SQLState[:], data[4:4+5]) + pos = 9 + } + + // Error Message [string] + me.Message = string(data[pos:]) + + return me +} + +func readStatus(b []byte) statusFlag { + return statusFlag(b[0]) | statusFlag(b[1])<<8 +} + +// Returns an instance of okHandler for codepaths where mysqlConn.result doesn't +// need to be cleared first (e.g. during authentication, or while additional +// resultsets are being fetched.) +func (mc *mysqlConn) resultUnchanged() *okHandler { + return (*okHandler)(mc) +} + +// okHandler represents the state of the connection when mysqlConn.result has +// been prepared for processing of OK packets. +// +// To correctly populate mysqlConn.result (updated by handleOkPacket()), all +// callpaths must either: +// +// 1. first clear it using clearResult(), or +// 2. confirm that they don't need to (by calling resultUnchanged()). +// +// Both return an instance of type *okHandler. +type okHandler mysqlConn + +// Exposes the underlying type's methods. +func (mc *okHandler) conn() *mysqlConn { + return (*mysqlConn)(mc) +} + +// clearResult clears the connection's stored affectedRows and insertIds +// fields. +// +// It returns a handler that can process OK responses. +func (mc *mysqlConn) clearResult() *okHandler { + mc.result = mysqlResult{} + return (*okHandler)(mc) +} + +// Ok Packet +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_ok_packet.html +func (mc *okHandler) handleOkPacket(data []byte) error { + var n, m int + var affectedRows, insertId uint64 + + // 0x00 [1 byte] + + // Affected rows [Length Coded Binary] + affectedRows, _, n = readLengthEncodedInteger(data[1:]) + + // Insert id [Length Coded Binary] + insertId, _, m = readLengthEncodedInteger(data[1+n:]) + + // Update for the current statement result (only used by + // readResultSetHeaderPacket). + if len(mc.result.affectedRows) > 0 { + mc.result.affectedRows[len(mc.result.affectedRows)-1] = int64(affectedRows) + } + if len(mc.result.insertIds) > 0 { + mc.result.insertIds[len(mc.result.insertIds)-1] = int64(insertId) + } + + // server_status [2 bytes] + mc.status = readStatus(data[1+n+m : 1+n+m+2]) + if mc.status&statusMoreResultsExists != 0 { + return nil + } + + // warning count [2 bytes] + + return nil +} + +// Read Packets as Field Packets until EOF-Packet or an Error appears +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset_column_definition.html#sect_protocol_com_query_response_text_resultset_column_definition_41 +func (mc *mysqlConn) readColumns(count int, old []mysqlField) ([]mysqlField, error) { + columns := make([]mysqlField, count) + if len(old) != count { + old = nil + } + + for i := range count { + data, err := mc.readPacket() + if err != nil { + return nil, err + } + + // Catalog + pos, err := skipLengthEncodedString(data) + if err != nil { + return nil, err + } + + // Database [len coded string] + n, err := skipLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + + // Table [len coded string] + if mc.cfg.ColumnsWithAlias { + tableName, _, n, err := readLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + if old != nil && old[i].tableName == string(tableName) { + // avoid allocating new string + columns[i].tableName = old[i].tableName + } else { + columns[i].tableName = string(tableName) + } + } else { + n, err = skipLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + } + + // Original table [len coded string] + n, err = skipLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + + // Name [len coded string] + name, _, n, err := readLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + if old != nil && old[i].name == string(name) { + // avoid allocating new string + columns[i].name = old[i].name + } else { + columns[i].name = string(name) + } + pos += n + + // Original name [len coded string] + n, err = skipLengthEncodedString(data[pos:]) + if err != nil { + return nil, err + } + pos += n + + // Filler [uint8] + pos++ + + // Charset [charset, collation uint8] + columns[i].charSet = data[pos] + pos += 2 + + // Length [uint32] + columns[i].length = binary.LittleEndian.Uint32(data[pos : pos+4]) + pos += 4 + + // Field type [uint8] + columns[i].fieldType = fieldType(data[pos]) + pos++ + + // Flags [uint16] + columns[i].flags = fieldFlag(binary.LittleEndian.Uint16(data[pos : pos+2])) + pos += 2 + + // Decimals [uint8] + columns[i].decimals = data[pos] + } + + // skip EOF packet if client does not support deprecateEOF + if err := mc.skipEof(); err != nil { + return nil, err + } + return columns, nil +} + +// Read Packets as Field Packets until EOF-Packet or an Error appears +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset_row.html +func (rows *textRows) readRow(dest []driver.Value) error { + mc := rows.mc + + if rows.rs.done { + return io.EOF + } + + data, err := mc.readPacket() + if err != nil { + return err + } + + // EOF Packet + // text row packets may starts with LengthEncodedString. + // In such case, 0xFE can mean string larger than 0xffffff. + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_dt_integers.html#sect_protocol_basic_dt_int_le + if data[0] == iEOF && len(data) <= 0xffffff { + if mc.capabilities&clientDeprecateEOF == 0 { + // Deprecated EOF packet + // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_eof_packet.html + mc.status = readStatus(data[3:]) + } else { + // Ok Packet with an 0xFE header + _, _, n := readLengthEncodedInteger(data[1:]) // affected_rows + _, _, m := readLengthEncodedInteger(data[1+n:]) // last_insert_id + mc.status = readStatus(data[1+n+m:]) + } + rows.rs.done = true + if !rows.HasNextResultSet() { + rows.mc = nil + } + return io.EOF + } + if data[0] == iERR { + rows.mc = nil + return mc.handleErrorPacket(data) + } + + // RowSet Packet + var ( + n int + isNull bool + pos int = 0 + ) + + for i := range dest { + // Read bytes and convert to string + var buf []byte + buf, isNull, n, err = readLengthEncodedString(data[pos:]) + pos += n + + if err != nil { + return err + } + + if isNull { + dest[i] = nil + continue + } + + switch rows.rs.columns[i].fieldType { + case fieldTypeTimestamp, + fieldTypeDateTime, + fieldTypeDate, + fieldTypeNewDate: + if mc.parseTime { + dest[i], err = parseDateTime(buf, mc.cfg.Loc) + } else { + dest[i] = buf + } + + case fieldTypeTiny, fieldTypeShort, fieldTypeInt24, fieldTypeYear, fieldTypeLong: + dest[i], err = strconv.ParseInt(string(buf), 10, 64) + + case fieldTypeLongLong: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + dest[i], err = strconv.ParseUint(string(buf), 10, 64) + } else { + dest[i], err = strconv.ParseInt(string(buf), 10, 64) + } + + case fieldTypeFloat: + var d float64 + d, err = strconv.ParseFloat(string(buf), 32) + dest[i] = float32(d) + + case fieldTypeDouble: + dest[i], err = strconv.ParseFloat(string(buf), 64) + + default: + dest[i] = buf + } + if err != nil { + return err + } + } + + return nil +} + +func (mc *mysqlConn) skipPackets(n int) error { + for range n { + if _, err := mc.readPacket(); err != nil { + return err + } + } + return nil +} + +// skips EOF packet after n * ColumnDefinition packets when clientDeprecateEOF is not set +func (mc *mysqlConn) skipEof() error { + if mc.capabilities&clientDeprecateEOF == 0 { + if _, err := mc.readPacket(); err != nil { + return err + } + } + return nil +} + +func (mc *mysqlConn) skipColumns(n int) error { + if err := mc.skipPackets(n); err != nil { + return err + } + return mc.skipEof() +} + +// Reads Packets until EOF-Packet or an Error appears. +func (mc *mysqlConn) skipRows() error { + for { + data, err := mc.readPacket() + if err != nil { + return err + } + + switch data[0] { + case iERR: + return mc.handleErrorPacket(data) + case iEOF: + // text row packets may starts with LengthEncodedString. + // In such case, 0xFE can mean string larger than 0xffffff. + if len(data) <= 0xffffff { + if mc.capabilities&clientDeprecateEOF == 0 { + // EOF packet + mc.status = readStatus(data[3:]) + } else { + // OK packet with an 0xFE header + _, _, n := readLengthEncodedInteger(data[1:]) // affected_rows + _, _, m := readLengthEncodedInteger(data[1+n:]) // last_insert_id + mc.status = readStatus(data[1+n+m:]) + } + return nil + } + } + } +} + +/****************************************************************************** +* Prepared Statements * +******************************************************************************/ + +// Prepare Result Packets +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_prepare.html#sect_protocol_com_stmt_prepare_response +func (stmt *mysqlStmt) readPrepareResultPacket() (uint16, error) { + data, err := stmt.mc.readPacket() + if err == nil { + // packet indicator [1 byte] + if data[0] != iOK { + return 0, stmt.mc.handleErrorPacket(data) + } + + // statement id [4 bytes] + stmt.id = binary.LittleEndian.Uint32(data[1:5]) + + // Column count [16 bit uint] + columnCount := binary.LittleEndian.Uint16(data[5:7]) + + // Param count [16 bit uint] + stmt.paramCount = int(binary.LittleEndian.Uint16(data[7:9])) + + // Reserved [8 bit] + + // Warning count [16 bit uint] + + return columnCount, nil + } + return 0, err +} + +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_send_long_data.html +func (stmt *mysqlStmt) writeCommandLongData(paramID int, arg []byte) error { + maxLen := stmt.mc.maxAllowedPacket - 1 + pktLen := maxLen + + // After the header (bytes 0-3) follows before the data: + // 1 byte command + // 4 bytes stmtID + // 2 bytes paramID + const dataOffset = 1 + 4 + 2 + + // Cannot use the write buffer since + // a) the buffer is too small + // b) it is in use + data := make([]byte, 4+1+4+2+len(arg)) + + copy(data[4+dataOffset:], arg) + + for argLen := len(arg); argLen > 0; argLen -= pktLen - dataOffset { + if dataOffset+argLen < maxLen { + pktLen = dataOffset + argLen + } + + // Add command byte [1 byte] + data[4] = comStmtSendLongData + + // Add stmtID [32 bit] + binary.LittleEndian.PutUint32(data[5:], stmt.id) + + // Add paramID [16 bit] + binary.LittleEndian.PutUint16(data[9:], uint16(paramID)) + + // Send CMD packet + err := stmt.mc.writePacket(data[:4+pktLen]) + // Every COM_LONG_DATA packet reset Packet Sequence + stmt.mc.resetSequence() + if err == nil { + data = data[pktLen-dataOffset:] + continue + } + return err + } + + return nil +} + +// Execute Prepared Statement +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_execute.html +func (stmt *mysqlStmt) writeExecutePacket(args []driver.Value) error { + if len(args) != stmt.paramCount { + return fmt.Errorf( + "argument count mismatch (got: %d; has: %d)", + len(args), + stmt.paramCount, + ) + } + + const minPktLen = 4 + 1 + 4 + 1 + 4 + mc := stmt.mc + + // Determine threshold dynamically to avoid packet size shortage. + longDataSize := max(mc.maxAllowedPacket/(stmt.paramCount+1), 64) + + // Reset packet-sequence + mc.resetSequence() + + var data []byte + var err error + + if len(args) == 0 { + data, err = mc.buf.takeBuffer(minPktLen) + } else { + data, err = mc.buf.takeCompleteBuffer() + // In this case the len(data) == cap(data) which is used to optimise the flow below. + } + if err != nil { + return err + } + + // command [1 byte] + data[4] = comStmtExecute + + // statement_id [4 bytes] + binary.LittleEndian.PutUint32(data[5:], stmt.id) + + // flags (0: CURSOR_TYPE_NO_CURSOR) [1 byte] + data[9] = 0x00 + + // iteration_count (uint32(1)) [4 bytes] + binary.LittleEndian.PutUint32(data[10:], 1) + + if len(args) > 0 { + pos := minPktLen + + var nullMask []byte + if maskLen, typesLen := (len(args)+7)/8, 1+2*len(args); pos+maskLen+typesLen >= cap(data) { + // buffer has to be extended but we don't know by how much so + // we depend on append after all data with known sizes fit. + // We stop at that because we deal with a lot of columns here + // which makes the required allocation size hard to guess. + tmp := make([]byte, pos+maskLen+typesLen) + copy(tmp[:pos], data[:pos]) + data = tmp + nullMask = data[pos : pos+maskLen] + // No need to clean nullMask as make ensures that. + pos += maskLen + } else { + nullMask = data[pos : pos+maskLen] + for i := range nullMask { + nullMask[i] = 0 + } + pos += maskLen + } + + // newParameterBoundFlag 1 [1 byte] + data[pos] = 0x01 + pos++ + + // type of each parameter [len(args)*2 bytes] + paramTypes := data[pos:] + pos += len(args) * 2 + + // value of each parameter [n bytes] + paramValues := data[pos:pos] + valuesCap := cap(paramValues) + + for i, arg := range args { + // build NULL-bitmap + if arg == nil { + nullMask[i/8] |= 1 << (uint(i) & 7) + paramTypes[i+i] = byte(fieldTypeNULL) + paramTypes[i+i+1] = 0x00 + continue + } + + if v, ok := arg.(json.RawMessage); ok { + arg = []byte(v) + } + // cache types and values + switch v := arg.(type) { + case int64: + paramTypes[i+i] = byte(fieldTypeLongLong) + paramTypes[i+i+1] = 0x00 + paramValues = binary.LittleEndian.AppendUint64(paramValues, uint64(v)) + + case uint64: + paramTypes[i+i] = byte(fieldTypeLongLong) + paramTypes[i+i+1] = 0x80 // type is unsigned + paramValues = binary.LittleEndian.AppendUint64(paramValues, uint64(v)) + + case float64: + paramTypes[i+i] = byte(fieldTypeDouble) + paramTypes[i+i+1] = 0x00 + paramValues = binary.LittleEndian.AppendUint64(paramValues, math.Float64bits(v)) + + case bool: + paramTypes[i+i] = byte(fieldTypeTiny) + paramTypes[i+i+1] = 0x00 + + if v { + paramValues = append(paramValues, 0x01) + } else { + paramValues = append(paramValues, 0x00) + } + + case []byte: + // Common case (non-nil value) first + if v != nil { + paramTypes[i+i] = byte(fieldTypeString) + paramTypes[i+i+1] = 0x00 + + if len(v) < longDataSize { + paramValues = appendLengthEncodedInteger(paramValues, + uint64(len(v)), + ) + paramValues = append(paramValues, v...) + } else { + if err := stmt.writeCommandLongData(i, v); err != nil { + return err + } + } + continue + } + + // Handle []byte(nil) as a NULL value + nullMask[i/8] |= 1 << (uint(i) & 7) + paramTypes[i+i] = byte(fieldTypeNULL) + paramTypes[i+i+1] = 0x00 + + case string: + paramTypes[i+i] = byte(fieldTypeString) + paramTypes[i+i+1] = 0x00 + + if len(v) < longDataSize { + paramValues = appendLengthEncodedInteger(paramValues, + uint64(len(v)), + ) + paramValues = append(paramValues, v...) + } else { + if err := stmt.writeCommandLongData(i, []byte(v)); err != nil { + return err + } + } + + case time.Time: + paramTypes[i+i] = byte(fieldTypeString) + paramTypes[i+i+1] = 0x00 + + var a [64]byte + var b = a[:0] + + if v.IsZero() { + b = append(b, "0000-00-00"...) + } else { + b, err = appendDateTime(b, v.In(mc.cfg.Loc), mc.cfg.timeTruncate) + if err != nil { + return err + } + } + + paramValues = appendLengthEncodedInteger(paramValues, + uint64(len(b)), + ) + paramValues = append(paramValues, b...) + + default: + return fmt.Errorf("cannot convert type: %T", arg) + } + } + + // Check if param values exceeded the available buffer + // In that case we must build the data packet with the new values buffer + if valuesCap != cap(paramValues) { + data = append(data[:pos], paramValues...) + mc.buf.store(data) // allow this buffer to be reused + } + + pos += len(paramValues) + data = data[:pos] + } + + err = mc.writePacket(data) + mc.syncSequence() + return err +} + +// For each remaining resultset in the stream, discards its rows and updates +// mc.affectedRows and mc.insertIds. +func (mc *okHandler) discardResults() error { + for mc.status&statusMoreResultsExists != 0 { + resLen, _, err := mc.readResultSetHeaderPacket() + if err != nil { + return err + } + if resLen > 0 { + // columns + if err := mc.conn().skipColumns(resLen); err != nil { + return err + } + // rows + if err := mc.conn().skipRows(); err != nil { + return err + } + } + } + return nil +} + +// https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_binary_resultset.html#sect_protocol_binary_resultset_row +func (rows *binaryRows) readRow(dest []driver.Value) error { + data, err := rows.mc.readPacket() + if err != nil { + return err + } + + // packet indicator [1 byte] + if data[0] != iOK { + // EOF/OK Packet + if data[0] == iEOF { + if rows.mc.capabilities&clientDeprecateEOF == 0 { + // EOF packet + rows.mc.status = readStatus(data[3:]) + } else { + // OK Packet with an 0xFE header + _, _, n := readLengthEncodedInteger(data[1:]) + _, _, m := readLengthEncodedInteger(data[1+n:]) + rows.mc.status = readStatus(data[1+n+m:]) + } + rows.rs.done = true + if !rows.HasNextResultSet() { + rows.mc = nil + } + return io.EOF + } + mc := rows.mc + rows.mc = nil + + // Error otherwise + return mc.handleErrorPacket(data) + } + + // NULL-bitmap, [(column-count + 7 + 2) / 8 bytes] + pos := 1 + (len(dest)+7+2)>>3 + nullMask := data[1:pos] + + for i := range dest { + // Field is NULL + // (byte >> bit-pos) % 2 == 1 + if ((nullMask[(i+2)>>3] >> uint((i+2)&7)) & 1) == 1 { + dest[i] = nil + continue + } + + // Convert to byte-coded string + switch rows.rs.columns[i].fieldType { + case fieldTypeNULL: + dest[i] = nil + continue + + // Numeric Types + case fieldTypeTiny: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + dest[i] = int64(data[pos]) + } else { + dest[i] = int64(int8(data[pos])) + } + pos++ + continue + + case fieldTypeShort, fieldTypeYear: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + dest[i] = int64(binary.LittleEndian.Uint16(data[pos : pos+2])) + } else { + dest[i] = int64(int16(binary.LittleEndian.Uint16(data[pos : pos+2]))) + } + pos += 2 + continue + + case fieldTypeInt24, fieldTypeLong: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + dest[i] = int64(binary.LittleEndian.Uint32(data[pos : pos+4])) + } else { + dest[i] = int64(int32(binary.LittleEndian.Uint32(data[pos : pos+4]))) + } + pos += 4 + continue + + case fieldTypeLongLong: + if rows.rs.columns[i].flags&flagUnsigned != 0 { + val := binary.LittleEndian.Uint64(data[pos : pos+8]) + if val > math.MaxInt64 { + dest[i] = uint64ToString(val) + } else { + dest[i] = int64(val) + } + } else { + dest[i] = int64(binary.LittleEndian.Uint64(data[pos : pos+8])) + } + pos += 8 + continue + + case fieldTypeFloat: + dest[i] = math.Float32frombits(binary.LittleEndian.Uint32(data[pos : pos+4])) + pos += 4 + continue + + case fieldTypeDouble: + dest[i] = math.Float64frombits(binary.LittleEndian.Uint64(data[pos : pos+8])) + pos += 8 + continue + + // Length coded Binary Strings + case fieldTypeDecimal, fieldTypeNewDecimal, fieldTypeVarChar, + fieldTypeBit, fieldTypeEnum, fieldTypeSet, fieldTypeTinyBLOB, + fieldTypeMediumBLOB, fieldTypeLongBLOB, fieldTypeBLOB, + fieldTypeVarString, fieldTypeString, fieldTypeGeometry, fieldTypeJSON, + fieldTypeVector: + var isNull bool + var n int + dest[i], isNull, n, err = readLengthEncodedString(data[pos:]) + pos += n + if err == nil { + if !isNull { + continue + } else { + dest[i] = nil + continue + } + } + return err + + case + fieldTypeDate, fieldTypeNewDate, // Date YYYY-MM-DD + fieldTypeTime, // Time [-][H]HH:MM:SS[.fractal] + fieldTypeTimestamp, fieldTypeDateTime: // Timestamp YYYY-MM-DD HH:MM:SS[.fractal] + + num, isNull, n := readLengthEncodedInteger(data[pos:]) + pos += n + + switch { + case isNull: + dest[i] = nil + continue + case rows.rs.columns[i].fieldType == fieldTypeTime: + // database/sql does not support an equivalent to TIME, return a string + var dstlen uint8 + switch decimals := rows.rs.columns[i].decimals; decimals { + case 0x00, 0x1f: + dstlen = 8 + case 1, 2, 3, 4, 5, 6: + dstlen = 8 + 1 + decimals + default: + return fmt.Errorf( + "protocol error, illegal decimals value %d", + rows.rs.columns[i].decimals, + ) + } + dest[i], err = formatBinaryTime(data[pos:pos+int(num)], dstlen) + case rows.mc.parseTime: + dest[i], err = parseBinaryDateTime(num, data[pos:], rows.mc.cfg.Loc) + default: + var dstlen uint8 + if rows.rs.columns[i].fieldType == fieldTypeDate { + dstlen = 10 + } else { + switch decimals := rows.rs.columns[i].decimals; decimals { + case 0x00, 0x1f: + dstlen = 19 + case 1, 2, 3, 4, 5, 6: + dstlen = 19 + 1 + decimals + default: + return fmt.Errorf( + "protocol error, illegal decimals value %d", + rows.rs.columns[i].decimals, + ) + } + } + dest[i], err = formatBinaryDateTime(data[pos:pos+int(num)], dstlen) + } + + if err == nil { + pos += int(num) + continue + } else { + return err + } + + // Please report if this happens! + default: + return fmt.Errorf("unknown field type %d", rows.rs.columns[i].fieldType) + } + } + + return nil +} diff --git a/vendor/github.com/go-sql-driver/mysql/result.go b/vendor/github.com/go-sql-driver/mysql/result.go new file mode 100644 index 0000000000..82dc0f9b6c --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/result.go @@ -0,0 +1,52 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import "slices" + +import "database/sql/driver" + +// Result exposes data not available through *connection.Result. +// +// This is accessible by executing statements using sql.Conn.Raw() and +// downcasting the returned result: +// +// res, err := rawConn.Exec(...) +// res.(mysql.Result).AllRowsAffected() +type Result interface { + driver.Result + // AllRowsAffected returns a slice containing the affected rows for each + // executed statement. + AllRowsAffected() []int64 + // AllLastInsertIds returns a slice containing the last inserted ID for each + // executed statement. + AllLastInsertIds() []int64 +} + +type mysqlResult struct { + // One entry in both slices is created for every executed statement result. + affectedRows []int64 + insertIds []int64 +} + +func (res *mysqlResult) LastInsertId() (int64, error) { + return res.insertIds[len(res.insertIds)-1], nil +} + +func (res *mysqlResult) RowsAffected() (int64, error) { + return res.affectedRows[len(res.affectedRows)-1], nil +} + +func (res *mysqlResult) AllLastInsertIds() []int64 { + return slices.Clone(res.insertIds) // defensive copy +} + +func (res *mysqlResult) AllRowsAffected() []int64 { + return slices.Clone(res.affectedRows) // defensive copy +} diff --git a/vendor/github.com/go-sql-driver/mysql/rows.go b/vendor/github.com/go-sql-driver/mysql/rows.go new file mode 100644 index 0000000000..190e75f9bf --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/rows.go @@ -0,0 +1,225 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql/driver" + "io" + "math" + "reflect" +) + +type resultSet struct { + columns []mysqlField + columnNames []string + done bool +} + +type mysqlRows struct { + mc *mysqlConn + rs resultSet + finish func() +} + +type binaryRows struct { + mysqlRows +} + +type textRows struct { + mysqlRows +} + +func (rows *mysqlRows) Columns() []string { + if rows.rs.columnNames != nil { + return rows.rs.columnNames + } + + columns := make([]string, len(rows.rs.columns)) + if rows.mc != nil && rows.mc.cfg.ColumnsWithAlias { + for i := range columns { + if tableName := rows.rs.columns[i].tableName; len(tableName) > 0 { + columns[i] = tableName + "." + rows.rs.columns[i].name + } else { + columns[i] = rows.rs.columns[i].name + } + } + } else { + for i := range columns { + columns[i] = rows.rs.columns[i].name + } + } + + rows.rs.columnNames = columns + return columns +} + +func (rows *mysqlRows) ColumnTypeDatabaseTypeName(i int) string { + return rows.rs.columns[i].typeDatabaseName() +} + +// func (rows *mysqlRows) ColumnTypeLength(i int) (length int64, ok bool) { +// return int64(rows.rs.columns[i].length), true +// } + +func (rows *mysqlRows) ColumnTypeNullable(i int) (nullable, ok bool) { + return rows.rs.columns[i].flags&flagNotNULL == 0, true +} + +func (rows *mysqlRows) ColumnTypePrecisionScale(i int) (int64, int64, bool) { + column := rows.rs.columns[i] + decimals := int64(column.decimals) + + switch column.fieldType { + case fieldTypeDecimal, fieldTypeNewDecimal: + if decimals > 0 { + return int64(column.length) - 2, decimals, true + } + return int64(column.length) - 1, decimals, true + case fieldTypeTimestamp, fieldTypeDateTime, fieldTypeTime: + return decimals, decimals, true + case fieldTypeFloat, fieldTypeDouble: + if decimals == 0x1f { + return math.MaxInt64, math.MaxInt64, true + } + return math.MaxInt64, decimals, true + } + + return 0, 0, false +} + +func (rows *mysqlRows) ColumnTypeScanType(i int) reflect.Type { + return rows.rs.columns[i].scanType() +} + +func (rows *mysqlRows) Close() (err error) { + if f := rows.finish; f != nil { + f() + rows.finish = nil + } + + mc := rows.mc + if mc == nil { + return nil + } + if err := mc.error(); err != nil { + return err + } + + // Remove unread packets from stream + if !rows.rs.done { + err = mc.skipRows() + } + if err == nil { + handleOk := mc.clearResult() + if err = handleOk.discardResults(); err != nil { + return err + } + } + + rows.mc = nil + return err +} + +func (rows *mysqlRows) HasNextResultSet() (b bool) { + if rows.mc == nil { + return false + } + return rows.mc.status&statusMoreResultsExists != 0 +} + +func (rows *mysqlRows) nextResultSet() (int, error) { + if rows.mc == nil { + return 0, io.EOF + } + if err := rows.mc.error(); err != nil { + return 0, err + } + + // Remove unread packets from stream + if !rows.rs.done { + if err := rows.mc.skipRows(); err != nil { + return 0, err + } + rows.rs.done = true + } + + if !rows.HasNextResultSet() { + rows.mc = nil + return 0, io.EOF + } + rows.rs = resultSet{} + // rows.mc.affectedRows and rows.mc.insertIds accumulate on each call to + // nextResultSet. + resLen, _, err := rows.mc.resultUnchanged().readResultSetHeaderPacket() + if err != nil { + // Clean up about multi-results flag + rows.rs.done = true + rows.mc.status = rows.mc.status & (^statusMoreResultsExists) + } + return resLen, err +} + +func (rows *mysqlRows) nextNotEmptyResultSet() (int, error) { + for { + resLen, err := rows.nextResultSet() + if err != nil { + return 0, err + } + + if resLen > 0 { + return resLen, nil + } + + rows.rs.done = true + } +} + +func (rows *binaryRows) NextResultSet() error { + resLen, err := rows.nextNotEmptyResultSet() + if err != nil { + return err + } + + rows.rs.columns, err = rows.mc.readColumns(resLen, nil) + return err +} + +func (rows *binaryRows) Next(dest []driver.Value) error { + if mc := rows.mc; mc != nil { + if err := mc.error(); err != nil { + return err + } + + // Fetch next row from stream + return rows.readRow(dest) + } + return io.EOF +} + +func (rows *textRows) NextResultSet() (err error) { + resLen, err := rows.nextNotEmptyResultSet() + if err != nil { + return err + } + + rows.rs.columns, err = rows.mc.readColumns(resLen, nil) + return err +} + +func (rows *textRows) Next(dest []driver.Value) error { + if mc := rows.mc; mc != nil { + if err := mc.error(); err != nil { + return err + } + + // Fetch next row from stream + return rows.readRow(dest) + } + return io.EOF +} diff --git a/vendor/github.com/go-sql-driver/mysql/statement.go b/vendor/github.com/go-sql-driver/mysql/statement.go new file mode 100644 index 0000000000..0261903b9d --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/statement.go @@ -0,0 +1,234 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "database/sql/driver" + "encoding/json" + "fmt" + "io" + "reflect" +) + +type mysqlStmt struct { + mc *mysqlConn + id uint32 + paramCount int + columns []mysqlField +} + +func (stmt *mysqlStmt) Close() error { + if stmt.mc == nil || stmt.mc.closed.Load() { + // driver.Stmt.Close could be called more than once, thus this function + // had to be idempotent. See also Issue #450 and golang/go#16019. + // This bug has been fixed in Go 1.8. + // https://github.com/golang/go/commit/90b8a0ca2d0b565c7c7199ffcf77b15ea6b6db3a + // But we keep this function idempotent because it is safer. + return nil + } + + err := stmt.mc.writeCommandPacketUint32(comStmtClose, stmt.id) + stmt.mc = nil + return err +} + +func (stmt *mysqlStmt) NumInput() int { + return stmt.paramCount +} + +func (stmt *mysqlStmt) ColumnConverter(idx int) driver.ValueConverter { + return converter{} +} + +func (stmt *mysqlStmt) CheckNamedValue(nv *driver.NamedValue) (err error) { + nv.Value, err = converter{}.ConvertValue(nv.Value) + return +} + +func (stmt *mysqlStmt) Exec(args []driver.Value) (driver.Result, error) { + if stmt.mc.closed.Load() { + return nil, driver.ErrBadConn + } + // Send command + err := stmt.writeExecutePacket(args) + if err != nil { + return nil, stmt.mc.markBadConn(err) + } + + mc := stmt.mc + handleOk := stmt.mc.clearResult() + + // Read Result + resLen, metadataFollows, err := handleOk.readResultSetHeaderPacket() + if err != nil { + return nil, err + } + + if resLen > 0 { + // Columns + if metadataFollows && stmt.mc.extCapabilities&clientCacheMetadata != 0 { + // we can not skip column metadata because next stmt.Query() may use it. + if stmt.columns, err = mc.readColumns(resLen, stmt.columns); err != nil { + return nil, err + } + } else { + if err = mc.skipColumns(resLen); err != nil { + return nil, err + } + } + + // Rows + if err = mc.skipRows(); err != nil { + return nil, err + } + } + + if err := handleOk.discardResults(); err != nil { + return nil, err + } + + copied := mc.result + return &copied, nil +} + +func (stmt *mysqlStmt) Query(args []driver.Value) (driver.Rows, error) { + return stmt.query(args) +} + +func (stmt *mysqlStmt) query(args []driver.Value) (*binaryRows, error) { + if stmt.mc.closed.Load() { + return nil, driver.ErrBadConn + } + // Send command + err := stmt.writeExecutePacket(args) + if err != nil { + return nil, stmt.mc.markBadConn(err) + } + + mc := stmt.mc + + // Read Result + handleOk := stmt.mc.clearResult() + resLen, metadataFollows, err := handleOk.readResultSetHeaderPacket() + if err != nil { + return nil, err + } + + rows := new(binaryRows) + + if resLen > 0 { + rows.mc = mc + if metadataFollows { + if rows.rs.columns, err = mc.readColumns(resLen, stmt.columns); err != nil { + return nil, err + } + stmt.columns = rows.rs.columns + } else { + if err = mc.skipEof(); err != nil { + return nil, err + } + rows.rs.columns = stmt.columns + } + } else { + rows.rs.done = true + + switch err := rows.NextResultSet(); err { + case nil, io.EOF: + return rows, nil + default: + return nil, err + } + } + + return rows, err +} + +var jsonType = reflect.TypeFor[json.RawMessage]() + +type converter struct{} + +// ConvertValue mirrors the reference/default converter in database/sql/driver +// with _one_ exception. We support uint64 with their high bit and the default +// implementation does not. This function should be kept in sync with +// database/sql/driver defaultConverter.ConvertValue() except for that +// deliberate difference. +func (c converter) ConvertValue(v any) (driver.Value, error) { + if driver.IsValue(v) { + return v, nil + } + + if vr, ok := v.(driver.Valuer); ok { + sv, err := callValuerValue(vr) + if err != nil { + return nil, err + } + if driver.IsValue(sv) { + return sv, nil + } + // A value returned from the Valuer interface can be "a type handled by + // a database driver's NamedValueChecker interface" so we should accept + // uint64 here as well. + if u, ok := sv.(uint64); ok { + return u, nil + } + return nil, fmt.Errorf("non-Value type %T returned from Value", sv) + } + rv := reflect.ValueOf(v) + switch rv.Kind() { + case reflect.Ptr: + // indirect pointers + if rv.IsNil() { + return nil, nil + } else { + return c.ConvertValue(rv.Elem().Interface()) + } + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + return rv.Int(), nil + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + return rv.Uint(), nil + case reflect.Float32, reflect.Float64: + return rv.Float(), nil + case reflect.Bool: + return rv.Bool(), nil + case reflect.Slice: + switch t := rv.Type(); { + case t == jsonType: + return v, nil + case t.Elem().Kind() == reflect.Uint8: + return rv.Bytes(), nil + default: + return nil, fmt.Errorf("unsupported type %T, a slice of %s", v, t.Elem().Kind()) + } + case reflect.String: + return rv.String(), nil + } + return nil, fmt.Errorf("unsupported type %T, a %s", v, rv.Kind()) +} + +var valuerReflectType = reflect.TypeFor[driver.Valuer]() + +// callValuerValue returns vr.Value(), with one exception: +// If vr.Value is an auto-generated method on a pointer type and the +// pointer is nil, it would panic at runtime in the panicwrap +// method. Treat it like nil instead. +// +// This is so people can implement driver.Value on value types and +// still use nil pointers to those types to mean nil/NULL, just like +// string/*string. +// +// This is an exact copy of the same-named unexported function from the +// database/sql package. +func callValuerValue(vr driver.Valuer) (v driver.Value, err error) { + if rv := reflect.ValueOf(vr); rv.Kind() == reflect.Ptr && + rv.IsNil() && + rv.Type().Elem().Implements(valuerReflectType) { + return nil, nil + } + return vr.Value() +} diff --git a/vendor/github.com/go-sql-driver/mysql/transaction.go b/vendor/github.com/go-sql-driver/mysql/transaction.go new file mode 100644 index 0000000000..8c502f49e6 --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/transaction.go @@ -0,0 +1,45 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +type mysqlTx struct { + mc *mysqlConn +} + +func (tx *mysqlTx) Commit() (err error) { + if tx.mc == nil { + return ErrInvalidConn + } + if tx.mc.closed.Load() { + err = tx.mc.error() + if err == nil { + err = ErrInvalidConn + } + return + } + err = tx.mc.exec("COMMIT") + tx.mc = nil + return +} + +func (tx *mysqlTx) Rollback() (err error) { + if tx.mc == nil { + return ErrInvalidConn + } + if tx.mc.closed.Load() { + err = tx.mc.error() + if err == nil { + err = ErrInvalidConn + } + return + } + err = tx.mc.exec("ROLLBACK") + tx.mc = nil + return +} diff --git a/vendor/github.com/go-sql-driver/mysql/utils.go b/vendor/github.com/go-sql-driver/mysql/utils.go new file mode 100644 index 0000000000..2dccb7d53e --- /dev/null +++ b/vendor/github.com/go-sql-driver/mysql/utils.go @@ -0,0 +1,805 @@ +// Go MySQL Driver - A MySQL-Driver for Go's database/sql package +// +// Copyright 2012 The Go-MySQL-Driver Authors. All rights reserved. +// +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this file, +// You can obtain one at http://mozilla.org/MPL/2.0/. + +package mysql + +import ( + "crypto/tls" + "database/sql" + "database/sql/driver" + "encoding/binary" + "errors" + "fmt" + "io" + "strconv" + "strings" + "sync" + "sync/atomic" + "time" +) + +// Registry for custom tls.Configs +var ( + tlsConfigLock sync.RWMutex + tlsConfigRegistry map[string]*tls.Config +) + +// RegisterTLSConfig registers a custom tls.Config to be used with sql.Open. +// Use the key as a value in the DSN where tls=value. +// +// Note: The provided tls.Config is exclusively owned by the driver after +// registering it. +// +// rootCertPool := x509.NewCertPool() +// pem, err := os.ReadFile("/path/ca-cert.pem") +// if err != nil { +// log.Fatal(err) +// } +// if ok := rootCertPool.AppendCertsFromPEM(pem); !ok { +// log.Fatal("Failed to append PEM.") +// } +// clientCert := make([]tls.Certificate, 0, 1) +// certs, err := tls.LoadX509KeyPair("/path/client-cert.pem", "/path/client-key.pem") +// if err != nil { +// log.Fatal(err) +// } +// clientCert = append(clientCert, certs) +// mysql.RegisterTLSConfig("custom", &tls.Config{ +// RootCAs: rootCertPool, +// Certificates: clientCert, +// }) +// db, err := sql.Open("mysql", "user@tcp(localhost:3306)/test?tls=custom") +func RegisterTLSConfig(key string, config *tls.Config) error { + if _, isBool := readBool(key); isBool || strings.ToLower(key) == "skip-verify" || strings.ToLower(key) == "preferred" { + return fmt.Errorf("key '%s' is reserved", key) + } + + tlsConfigLock.Lock() + if tlsConfigRegistry == nil { + tlsConfigRegistry = make(map[string]*tls.Config) + } + + tlsConfigRegistry[key] = config + tlsConfigLock.Unlock() + return nil +} + +// DeregisterTLSConfig removes the tls.Config associated with key. +func DeregisterTLSConfig(key string) { + tlsConfigLock.Lock() + if tlsConfigRegistry != nil { + delete(tlsConfigRegistry, key) + } + tlsConfigLock.Unlock() +} + +func getTLSConfigClone(key string) (config *tls.Config) { + tlsConfigLock.RLock() + if v, ok := tlsConfigRegistry[key]; ok { + config = v.Clone() + } + tlsConfigLock.RUnlock() + return +} + +// Returns the bool value of the input. +// The 2nd return value indicates if the input was a valid bool value +func readBool(input string) (value bool, valid bool) { + switch input { + case "1", "true", "TRUE", "True": + return true, true + case "0", "false", "FALSE", "False": + return false, true + } + + // Not a valid bool value + return +} + +/****************************************************************************** +* Time related utils * +******************************************************************************/ + +func parseDateTime(b []byte, loc *time.Location) (time.Time, error) { + const base = "0000-00-00 00:00:00.000000" + switch len(b) { + case 10, 19, 21, 22, 23, 24, 25, 26: // up to "YYYY-MM-DD HH:MM:SS.MMMMMM" + if string(b) == base[:len(b)] { + return time.Time{}, nil + } + + year, err := parseByteYear(b) + if err != nil { + return time.Time{}, err + } + if b[4] != '-' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[4]) + } + + m, err := parseByte2Digits(b[5], b[6]) + if err != nil { + return time.Time{}, err + } + month := time.Month(m) + + if b[7] != '-' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[7]) + } + + day, err := parseByte2Digits(b[8], b[9]) + if err != nil { + return time.Time{}, err + } + if len(b) == 10 { + return time.Date(year, month, day, 0, 0, 0, 0, loc), nil + } + + if b[10] != ' ' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[10]) + } + + hour, err := parseByte2Digits(b[11], b[12]) + if err != nil { + return time.Time{}, err + } + if b[13] != ':' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[13]) + } + + min, err := parseByte2Digits(b[14], b[15]) + if err != nil { + return time.Time{}, err + } + if b[16] != ':' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[16]) + } + + sec, err := parseByte2Digits(b[17], b[18]) + if err != nil { + return time.Time{}, err + } + if len(b) == 19 { + return time.Date(year, month, day, hour, min, sec, 0, loc), nil + } + + if b[19] != '.' { + return time.Time{}, fmt.Errorf("bad value for field: `%c`", b[19]) + } + nsec, err := parseByteNanoSec(b[20:]) + if err != nil { + return time.Time{}, err + } + return time.Date(year, month, day, hour, min, sec, nsec, loc), nil + default: + return time.Time{}, fmt.Errorf("invalid time bytes: %s", b) + } +} + +func parseByteYear(b []byte) (int, error) { + year, n := 0, 1000 + for i := range 4 { + v, err := bToi(b[i]) + if err != nil { + return 0, err + } + year += v * n + n /= 10 + } + return year, nil +} + +func parseByte2Digits(b1, b2 byte) (int, error) { + d1, err := bToi(b1) + if err != nil { + return 0, err + } + d2, err := bToi(b2) + if err != nil { + return 0, err + } + return d1*10 + d2, nil +} + +func parseByteNanoSec(b []byte) (int, error) { + ns, digit := 0, 100000 // max is 6-digits + for i := range b { + v, err := bToi(b[i]) + if err != nil { + return 0, err + } + ns += v * digit + digit /= 10 + } + // nanoseconds has 10-digits. (needs to scale digits) + // 10 - 6 = 4, so we have to multiple 1000. + return ns * 1000, nil +} + +func bToi(b byte) (int, error) { + if b < '0' || b > '9' { + return 0, errors.New("not [0-9]") + } + return int(b - '0'), nil +} + +func parseBinaryDateTime(num uint64, data []byte, loc *time.Location) (driver.Value, error) { + switch num { + case 0: + return time.Time{}, nil + case 4: + return time.Date( + int(binary.LittleEndian.Uint16(data[:2])), // year + time.Month(data[2]), // month + int(data[3]), // day + 0, 0, 0, 0, + loc, + ), nil + case 7: + return time.Date( + int(binary.LittleEndian.Uint16(data[:2])), // year + time.Month(data[2]), // month + int(data[3]), // day + int(data[4]), // hour + int(data[5]), // minutes + int(data[6]), // seconds + 0, + loc, + ), nil + case 11: + return time.Date( + int(binary.LittleEndian.Uint16(data[:2])), // year + time.Month(data[2]), // month + int(data[3]), // day + int(data[4]), // hour + int(data[5]), // minutes + int(data[6]), // seconds + int(binary.LittleEndian.Uint32(data[7:11]))*1000, // nanoseconds + loc, + ), nil + } + return nil, fmt.Errorf("invalid DATETIME packet length %d", num) +} + +func appendDateTime(buf []byte, t time.Time, timeTruncate time.Duration) ([]byte, error) { + if timeTruncate > 0 { + t = t.Truncate(timeTruncate) + } + + year, month, day := t.Date() + hour, min, sec := t.Clock() + nsec := t.Nanosecond() + + if year < 1 || year > 9999 { + return buf, errors.New("year is not in the range [1, 9999]: " + strconv.Itoa(year)) // use errors.New instead of fmt.Errorf to avoid year escape to heap + } + year100 := year / 100 + year1 := year % 100 + + var localBuf [len("2006-01-02T15:04:05.999999999")]byte // does not escape + localBuf[0], localBuf[1], localBuf[2], localBuf[3] = digits10[year100], digits01[year100], digits10[year1], digits01[year1] + localBuf[4] = '-' + localBuf[5], localBuf[6] = digits10[month], digits01[month] + localBuf[7] = '-' + localBuf[8], localBuf[9] = digits10[day], digits01[day] + + if hour == 0 && min == 0 && sec == 0 && nsec == 0 { + return append(buf, localBuf[:10]...), nil + } + + localBuf[10] = ' ' + localBuf[11], localBuf[12] = digits10[hour], digits01[hour] + localBuf[13] = ':' + localBuf[14], localBuf[15] = digits10[min], digits01[min] + localBuf[16] = ':' + localBuf[17], localBuf[18] = digits10[sec], digits01[sec] + + if nsec == 0 { + return append(buf, localBuf[:19]...), nil + } + nsec100000000 := nsec / 100000000 + nsec1000000 := (nsec / 1000000) % 100 + nsec10000 := (nsec / 10000) % 100 + nsec100 := (nsec / 100) % 100 + nsec1 := nsec % 100 + localBuf[19] = '.' + + // milli second + localBuf[20], localBuf[21], localBuf[22] = + digits01[nsec100000000], digits10[nsec1000000], digits01[nsec1000000] + // micro second + localBuf[23], localBuf[24], localBuf[25] = + digits10[nsec10000], digits01[nsec10000], digits10[nsec100] + // nano second + localBuf[26], localBuf[27], localBuf[28] = + digits01[nsec100], digits10[nsec1], digits01[nsec1] + + // trim trailing zeros + n := len(localBuf) + for n > 0 && localBuf[n-1] == '0' { + n-- + } + + return append(buf, localBuf[:n]...), nil +} + +// zeroDateTime is used in formatBinaryDateTime to avoid an allocation +// if the DATE or DATETIME has the zero value. +// It must never be changed. +// The current behavior depends on database/sql copying the result. +var zeroDateTime = []byte("0000-00-00 00:00:00.000000") + +const digits01 = "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789" +const digits10 = "0000000000111111111122222222223333333333444444444455555555556666666666777777777788888888889999999999" + +func appendMicrosecs(dst, src []byte, decimals int) []byte { + if decimals <= 0 { + return dst + } + if len(src) == 0 { + return append(dst, ".000000"[:decimals+1]...) + } + + microsecs := binary.LittleEndian.Uint32(src[:4]) + p1 := byte(microsecs / 10000) + microsecs -= 10000 * uint32(p1) + p2 := byte(microsecs / 100) + microsecs -= 100 * uint32(p2) + p3 := byte(microsecs) + + switch decimals { + default: + return append(dst, '.', + digits10[p1], digits01[p1], + digits10[p2], digits01[p2], + digits10[p3], digits01[p3], + ) + case 1: + return append(dst, '.', + digits10[p1], + ) + case 2: + return append(dst, '.', + digits10[p1], digits01[p1], + ) + case 3: + return append(dst, '.', + digits10[p1], digits01[p1], + digits10[p2], + ) + case 4: + return append(dst, '.', + digits10[p1], digits01[p1], + digits10[p2], digits01[p2], + ) + case 5: + return append(dst, '.', + digits10[p1], digits01[p1], + digits10[p2], digits01[p2], + digits10[p3], + ) + } +} + +func formatBinaryDateTime(src []byte, length uint8) (driver.Value, error) { + // length expects the deterministic length of the zero value, + // negative time and 100+ hours are automatically added if needed + if len(src) == 0 { + return zeroDateTime[:length], nil + } + var dst []byte // return value + var p1, p2, p3 byte // current digit pair + + switch length { + case 10, 19, 21, 22, 23, 24, 25, 26: + default: + t := "DATE" + if length > 10 { + t += "TIME" + } + return nil, fmt.Errorf("illegal %s length %d", t, length) + } + switch len(src) { + case 4, 7, 11: + default: + t := "DATE" + if length > 10 { + t += "TIME" + } + return nil, fmt.Errorf("illegal %s packet length %d", t, len(src)) + } + dst = make([]byte, 0, length) + // start with the date + year := binary.LittleEndian.Uint16(src[:2]) + pt := year / 100 + p1 = byte(year - 100*uint16(pt)) + p2, p3 = src[2], src[3] + dst = append(dst, + digits10[pt], digits01[pt], + digits10[p1], digits01[p1], '-', + digits10[p2], digits01[p2], '-', + digits10[p3], digits01[p3], + ) + if length == 10 { + return dst, nil + } + if len(src) == 4 { + return append(dst, zeroDateTime[10:length]...), nil + } + dst = append(dst, ' ') + p1 = src[4] // hour + src = src[5:] + + // p1 is 2-digit hour, src is after hour + p2, p3 = src[0], src[1] + dst = append(dst, + digits10[p1], digits01[p1], ':', + digits10[p2], digits01[p2], ':', + digits10[p3], digits01[p3], + ) + return appendMicrosecs(dst, src[2:], int(length)-20), nil +} + +func formatBinaryTime(src []byte, length uint8) (driver.Value, error) { + // length expects the deterministic length of the zero value, + // negative time and 100+ hours are automatically added if needed + if len(src) == 0 { + return zeroDateTime[11 : 11+length], nil + } + var dst []byte // return value + + switch length { + case + 8, // time (can be up to 10 when negative and 100+ hours) + 10, 11, 12, 13, 14, 15: // time with fractional seconds + default: + return nil, fmt.Errorf("illegal TIME length %d", length) + } + switch len(src) { + case 8, 12: + default: + return nil, fmt.Errorf("invalid TIME packet length %d", len(src)) + } + // +2 to enable negative time and 100+ hours + dst = make([]byte, 0, length+2) + if src[0] == 1 { + dst = append(dst, '-') + } + days := binary.LittleEndian.Uint32(src[1:5]) + hours := int64(days)*24 + int64(src[5]) + + if hours >= 100 { + dst = strconv.AppendInt(dst, hours, 10) + } else { + dst = append(dst, digits10[hours], digits01[hours]) + } + + min, sec := src[6], src[7] + dst = append(dst, ':', + digits10[min], digits01[min], ':', + digits10[sec], digits01[sec], + ) + return appendMicrosecs(dst, src[8:], int(length)-9), nil +} + +/****************************************************************************** +* Convert from and to bytes * +******************************************************************************/ + +// 24bit integer: used for packet headers. + +func putUint24(data []byte, n int) { + data[2] = byte(n >> 16) + data[1] = byte(n >> 8) + data[0] = byte(n) +} + +func getUint24(data []byte) int { + return int(data[2])<<16 | int(data[1])<<8 | int(data[0]) +} + +func uint64ToString(n uint64) []byte { + var a [20]byte + i := 20 + + // U+0030 = 0 + // ... + // U+0039 = 9 + + var q uint64 + for n >= 10 { + i-- + q = n / 10 + a[i] = uint8(n-q*10) + 0x30 + n = q + } + + i-- + a[i] = uint8(n) + 0x30 + + return a[i:] +} + +// returns the string read as a bytes slice, whether the value is NULL, +// the number of bytes read and an error, in case the string is longer than +// the input slice +func readLengthEncodedString(b []byte) ([]byte, bool, int, error) { + // Get length + num, isNull, n := readLengthEncodedInteger(b) + if num < 1 { + return b[n:n], isNull, n, nil + } + + n += int(num) + + // Check data length + if len(b) >= n { + return b[n-int(num) : n : n], false, n, nil + } + return nil, false, n, io.EOF +} + +// returns the number of bytes skipped and an error, in case the string is +// longer than the input slice +func skipLengthEncodedString(b []byte) (int, error) { + // Get length + num, _, n := readLengthEncodedInteger(b) + if num < 1 { + return n, nil + } + + n += int(num) + + // Check data length + if len(b) >= n { + return n, nil + } + return n, io.EOF +} + +// returns the number read, whether the value is NULL and the number of bytes read +func readLengthEncodedInteger(b []byte) (uint64, bool, int) { + // See issue #349 + if len(b) == 0 { + return 0, true, 1 + } + + switch b[0] { + // 251: NULL + case 0xfb: + return 0, true, 1 + + // 252: value of following 2 + case 0xfc: + return uint64(binary.LittleEndian.Uint16(b[1:])), false, 3 + + // 253: value of following 3 + case 0xfd: + return uint64(getUint24(b[1:])), false, 4 + + // 254: value of following 8 + case 0xfe: + return uint64(binary.LittleEndian.Uint64(b[1:])), false, 9 + } + + // 0-250: value of first byte + return uint64(b[0]), false, 1 +} + +// encodes a uint64 value and appends it to the given bytes slice +func appendLengthEncodedInteger(b []byte, n uint64) []byte { + switch { + case n <= 250: + return append(b, byte(n)) + + case n <= 0xffff: + b = append(b, 0xfc) + return binary.LittleEndian.AppendUint16(b, uint16(n)) + + case n <= 0xffffff: + return append(b, 0xfd, byte(n), byte(n>>8), byte(n>>16)) + } + b = append(b, 0xfe) + return binary.LittleEndian.AppendUint64(b, n) +} + +func appendLengthEncodedString(b []byte, s string) []byte { + b = appendLengthEncodedInteger(b, uint64(len(s))) + return append(b, s...) +} + +// reserveBuffer checks cap(buf) and expand buffer to len(buf) + appendSize. +// If cap(buf) is not enough, reallocate new buffer. +func reserveBuffer(buf []byte, appendSize int) []byte { + newSize := len(buf) + appendSize + if cap(buf) < newSize { + // Grow buffer exponentially + newBuf := make([]byte, len(buf)*2+appendSize) + copy(newBuf, buf) + buf = newBuf + } + return buf[:newSize] +} + +// Lookup table for backslash escapes (used for both string and bytes) +var backslashEscapeTable [256]byte + +func init() { + backslashEscapeTable['\x00'] = '0' + backslashEscapeTable['\n'] = 'n' + backslashEscapeTable['\r'] = 'r' + backslashEscapeTable['\x1a'] = 'Z' + backslashEscapeTable['\''] = '\'' + backslashEscapeTable['"'] = '"' + backslashEscapeTable['\\'] = '\\' +} + +// escapeStringBackslash is similar to escapeBytesBackslash but for string. +func escapeStringBackslash(buf []byte, v string) []byte { + pos := len(buf) + buf = reserveBuffer(buf, len(v)*2+2) + buf[pos] = '\'' + pos++ + for i := 0; i < len(v); i++ { + c := v[i] + if esc := backslashEscapeTable[c]; esc != 0 { + buf[pos+1] = esc + buf[pos] = '\\' + pos += 2 + } else { + buf[pos] = c + pos++ + } + } + buf[pos] = '\'' + pos++ + return buf[:pos] +} + +// escapeBytesBackslash appends _binary'...' or '...' with backslash escaping for bytes. +func escapeBytesBackslash(buf, v []byte, binary bool) []byte { + pos := len(buf) + if binary { + buf = reserveBuffer(buf, len(v)*2+9) + copy(buf[pos:], []byte("_binary'")) + pos += 8 + } else { + buf = reserveBuffer(buf, len(v)*2+2) + buf[pos] = '\'' + pos++ + } + for _, c := range v { + if esc := backslashEscapeTable[c]; esc != 0 { + buf[pos+1] = esc + buf[pos] = '\\' + pos += 2 + } else { + buf[pos] = c + pos++ + } + } + buf[pos] = '\'' + pos++ + return buf[:pos] +} + +// escapeBytesQuotes appends _binary'...' or '...' with single-quote escaping for bytes. +func escapeBytesQuotes(buf, v []byte, binary bool) []byte { + pos := len(buf) + if binary { + buf = reserveBuffer(buf, len(v)*2+9) + copy(buf[pos:], []byte("_binary'")) + pos += 8 + } else { + buf = reserveBuffer(buf, len(v)*2+2) + buf[pos] = '\'' + pos++ + } + for _, c := range v { + if c == '\'' { + buf[pos+1] = '\'' + buf[pos] = '\'' + pos += 2 + } else { + buf[pos] = c + pos++ + } + } + buf[pos] = '\'' + pos++ + return buf[:pos] +} + +// escapeStringQuotes is similar to escapeBytesQuotes but for string. +func escapeStringQuotes(buf []byte, v string) []byte { + pos := len(buf) + buf = reserveBuffer(buf, len(v)*2+2) + buf[pos] = '\'' + pos++ + for i := range len(v) { + c := v[i] + if c == '\'' { + buf[pos+1] = '\'' + buf[pos] = '\'' + pos += 2 + } else { + buf[pos] = c + pos++ + } + } + buf[pos] = '\'' + pos++ + return buf[:pos] +} + +/****************************************************************************** +* Sync utils * +******************************************************************************/ + +// noCopy may be embedded into structs which must not be copied +// after the first use. +// +// See https://github.com/golang/go/issues/8005#issuecomment-190753527 +// for details. +type noCopy struct{} + +// Lock is a no-op used by -copylocks checker from `go vet`. +func (*noCopy) Lock() {} + +// Unlock is a no-op used by -copylocks checker from `go vet`. +// noCopy should implement sync.Locker from Go 1.11 +// https://github.com/golang/go/commit/c2eba53e7f80df21d51285879d51ab81bcfbf6bc +// https://github.com/golang/go/issues/26165 +func (*noCopy) Unlock() {} + +// atomicError is a wrapper for atomically accessed error values +type atomicError struct { + _ noCopy + value atomic.Value +} + +// Set sets the error value regardless of the previous value. +// The value must not be nil +func (ae *atomicError) Set(value error) { + ae.value.Store(value) +} + +// Value returns the current error value +func (ae *atomicError) Value() error { + if v := ae.value.Load(); v != nil { + // this will panic if the value doesn't implement the error interface + return v.(error) + } + return nil +} + +func namedValueToValue(named []driver.NamedValue) ([]driver.Value, error) { + dargs := make([]driver.Value, len(named)) + for n, param := range named { + if len(param.Name) > 0 { + // TODO: support the use of Named Parameters #561 + return nil, errors.New("mysql: driver does not support the use of Named Parameters") + } + dargs[n] = param.Value + } + return dargs, nil +} + +func mapIsolationLevel(level driver.IsolationLevel) (string, error) { + switch sql.IsolationLevel(level) { + case sql.LevelRepeatableRead: + return "REPEATABLE READ", nil + case sql.LevelReadCommitted: + return "READ COMMITTED", nil + case sql.LevelReadUncommitted: + return "READ UNCOMMITTED", nil + case sql.LevelSerializable: + return "SERIALIZABLE", nil + default: + return "", fmt.Errorf("mysql: unsupported isolation level: %v", level) + } +} diff --git a/vendor/github.com/openfga/openfga/pkg/storage/mysql/doc.go b/vendor/github.com/openfga/openfga/pkg/storage/mysql/doc.go new file mode 100644 index 0000000000..9a8a31248e --- /dev/null +++ b/vendor/github.com/openfga/openfga/pkg/storage/mysql/doc.go @@ -0,0 +1,2 @@ +// Package mysql contains an implementation of the storage interface that works with MySQL. +package mysql diff --git a/vendor/github.com/openfga/openfga/pkg/storage/mysql/mysql.go b/vendor/github.com/openfga/openfga/pkg/storage/mysql/mysql.go new file mode 100644 index 0000000000..04657f345b --- /dev/null +++ b/vendor/github.com/openfga/openfga/pkg/storage/mysql/mysql.go @@ -0,0 +1,845 @@ +package mysql + +import ( + "context" + "database/sql" + "errors" + "fmt" + "strings" + "time" + + sq "github.com/Masterminds/squirrel" + "github.com/cenkalti/backoff/v4" + "github.com/go-sql-driver/mysql" + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/collectors" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/trace" + "go.uber.org/zap" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/known/structpb" + "google.golang.org/protobuf/types/known/timestamppb" + + openfgav1 "github.com/openfga/api/proto/openfga/v1" + + "github.com/openfga/openfga/pkg/logger" + "github.com/openfga/openfga/pkg/storage" + "github.com/openfga/openfga/pkg/storage/sqlcommon" + tupleUtils "github.com/openfga/openfga/pkg/tuple" +) + +var tracer = otel.Tracer("openfga/pkg/storage/mysql") + +func startTrace(ctx context.Context, name string) (context.Context, trace.Span) { + return tracer.Start(ctx, "mysql."+name) +} + +// Datastore provides a MySQL based implementation of [storage.OpenFGADatastore]. +type Datastore struct { + stbl sq.StatementBuilderType + db *sql.DB + dbInfo *sqlcommon.DBInfo + logger logger.Logger + dbStatsCollector prometheus.Collector + maxTuplesPerWriteField int + maxTypesPerModelField int + versionReady bool +} + +// Ensures that Datastore implements the OpenFGADatastore interface. +var _ storage.OpenFGADatastore = (*Datastore)(nil) + +// New creates a new [Datastore] storage. +func New(uri string, cfg *sqlcommon.Config) (*Datastore, error) { + if cfg.Username != "" || cfg.Password != "" { + dsnCfg, err := mysql.ParseDSN(uri) + if err != nil { + return nil, fmt.Errorf("parse mysql connection dsn: %w", err) + } + + if cfg.Username != "" { + dsnCfg.User = cfg.Username + } + if cfg.Password != "" { + dsnCfg.Passwd = cfg.Password + } + dsnCfg.AllowNativePasswords = true + uri = dsnCfg.FormatDSN() + } + + db, err := sql.Open("mysql", uri) + if err != nil { + return nil, fmt.Errorf("initialize mysql connection: %w", err) + } + return NewWithDB(db, cfg) +} + +// NewWithDB creates a new [Datastore] storage with the provided database connection. +func NewWithDB(db *sql.DB, cfg *sqlcommon.Config) (*Datastore, error) { + if cfg.MaxIdleConns != 0 { + db.SetMaxIdleConns(cfg.MaxIdleConns) // default is 2, not retaining connections(0) would be detrimental for performance + } + + db.SetMaxOpenConns(cfg.MaxOpenConns) + db.SetConnMaxIdleTime(cfg.ConnMaxIdleTime) + db.SetConnMaxLifetime(cfg.ConnMaxLifetime) + + policy := backoff.NewExponentialBackOff(backoff.WithMaxElapsedTime(cfg.PingRetryMaxElapsedTime)) + attempt := 1 + err := backoff.Retry(func() error { + ctx, cancel := context.WithTimeout(context.Background(), cfg.PingTimeout) + defer cancel() + + err := db.PingContext(ctx) + if err != nil { + cfg.Logger.Info("waiting for database", zap.Int("attempt", attempt)) + attempt++ + return err + } + return nil + }, policy) + if err != nil { + return nil, fmt.Errorf("ping db: %w", err) + } + + var collector prometheus.Collector + if cfg.ExportMetrics { + collector = collectors.NewDBStatsCollector(db, "openfga") + if err := prometheus.Register(collector); err != nil { + return nil, fmt.Errorf("initialize metrics: %w", err) + } + } + + stbl := sq.StatementBuilder.RunWith(db) + dbInfo := sqlcommon.NewDBInfo(stbl, HandleSQLError, "mysql") + + return &Datastore{ + stbl: stbl, + db: db, + dbInfo: dbInfo, + logger: cfg.Logger, + dbStatsCollector: collector, + maxTuplesPerWriteField: cfg.MaxTuplesPerWriteField, + maxTypesPerModelField: cfg.MaxTypesPerModelField, + versionReady: false, + }, nil +} + +// Close see [storage.OpenFGADatastore].Close. +func (s *Datastore) Close() { + if s.dbStatsCollector != nil { + prometheus.Unregister(s.dbStatsCollector) + } + s.db.Close() +} + +// Read see [storage.RelationshipTupleReader].Read. +func (s *Datastore) Read( + ctx context.Context, + store string, + filter storage.ReadFilter, + _ storage.ReadOptions, +) (storage.TupleIterator, error) { + ctx, span := startTrace(ctx, "Read") + defer span.End() + + return s.read(ctx, store, filter, nil) +} + +// ReadPage see [storage.RelationshipTupleReader].ReadPage. +func (s *Datastore) ReadPage(ctx context.Context, store string, filter storage.ReadFilter, options storage.ReadPageOptions) ([]*openfgav1.Tuple, string, error) { + ctx, span := startTrace(ctx, "ReadPage") + defer span.End() + + iter, err := s.read(ctx, store, filter, &options) + if err != nil { + return nil, "", err + } + defer iter.Stop() + + return iter.ToArray(ctx, options.Pagination) +} + +func (s *Datastore) read(ctx context.Context, store string, filter storage.ReadFilter, options *storage.ReadPageOptions) (*sqlcommon.SQLTupleIterator, error) { + _, span := startTrace(ctx, "read") + defer span.End() + + sb := s.stbl. + Select( + "store", "object_type", "object_id", "relation", + "_user", + "condition_name", "condition_context", "ulid", "inserted_at", + ). + From("tuple"). + Where(sq.Eq{"store": store}) + if options != nil { + sb = sb.OrderBy("ulid") + } + + objectType, objectID := tupleUtils.SplitObject(filter.Object) + if objectType != "" { + sb = sb.Where(sq.Eq{"object_type": objectType}) + } + if objectID != "" { + sb = sb.Where(sq.Eq{"object_id": objectID}) + } + if filter.Relation != "" { + sb = sb.Where(sq.Eq{"relation": filter.Relation}) + } + if filter.User != "" { + userType, userID, _ := tupleUtils.ToUserParts(filter.User) + if userID != "" { + sb = sb.Where(sq.Eq{"_user": filter.User}) + } else { + sb = sb.Where(sq.Like{"_user": userType + ":%"}) + } + } + + if len(filter.Conditions) > 0 { + // Use COALESCE to treat NULL and '' as the same value (empty string). + // This allows filtering for "no condition" (e.g., filter.Conditions = [""]) + // to correctly match rows where condition_name is either '' OR NULL. + sb = sb.Where(sq.Eq{"COALESCE(condition_name, '')": filter.Conditions}) + } + + if options != nil && options.Pagination.From != "" { + token := options.Pagination.From + sb = sb.Where(sq.GtOrEq{"ulid": token}) + } + if options != nil && options.Pagination.PageSize != 0 { + sb = sb.Limit(uint64(options.Pagination.PageSize + 1)) // + 1 is used to determine whether to return a continuation token. + } + + rg, err := sqlcommon.NewRowGetter(sqlcommon.NewSQLConnector(s.db), sb) + if err != nil { + return nil, err + } + return sqlcommon.NewSQLTupleIterator(rg, HandleSQLError), nil +} + +// Write see [storage.RelationshipTupleWriter].Write. +func (s *Datastore) Write( + ctx context.Context, + store string, + deletes storage.Deletes, + writes storage.Writes, + opts ...storage.TupleWriteOption, +) error { + ctx, span := startTrace(ctx, "Write") + defer span.End() + + return sqlcommon.Write(ctx, s.dbInfo, s.db, store, + sqlcommon.WriteData{ + Deletes: deletes, + Writes: writes, + Opts: storage.NewTupleWriteOptions(opts...), + Now: time.Now().UTC(), + }) +} + +// ReadUserTuple see [storage.RelationshipTupleReader].ReadUserTuple. +func (s *Datastore) ReadUserTuple(ctx context.Context, store string, filter storage.ReadUserTupleFilter, _ storage.ReadUserTupleOptions) (*openfgav1.Tuple, error) { + ctx, span := startTrace(ctx, "ReadUserTuple") + defer span.End() + + objectType, objectID := tupleUtils.SplitObject(filter.Object) + userType := tupleUtils.GetUserTypeFromUser(filter.User) + + var conditionName sql.NullString + var conditionContext []byte + var record storage.TupleRecord + + sb := s.stbl. + Select( + "object_type", "object_id", "relation", + "_user", + "condition_name", "condition_context", + ). + From("tuple"). + Where(sq.Eq{ + "store": store, + "object_type": objectType, + "object_id": objectID, + "relation": filter.Relation, + "_user": filter.User, + "user_type": userType, + }) + + if len(filter.Conditions) > 0 { + sb = sb.Where(sq.Eq{"COALESCE(condition_name, '')": filter.Conditions}) + } + + err := sb.QueryRowContext(ctx). + Scan( + &record.ObjectType, + &record.ObjectID, + &record.Relation, + &record.User, + &conditionName, + &conditionContext, + ) + if err != nil { + return nil, HandleSQLError(err) + } + + if conditionName.String != "" { + record.ConditionName = conditionName.String + + if conditionContext != nil { + var conditionContextStruct structpb.Struct + if err := proto.Unmarshal(conditionContext, &conditionContextStruct); err != nil { + return nil, err + } + record.ConditionContext = &conditionContextStruct + } + } + + return record.AsTuple(), nil +} + +// ReadUsersetTuples see [storage.RelationshipTupleReader].ReadUsersetTuples. +func (s *Datastore) ReadUsersetTuples( + ctx context.Context, + store string, + filter storage.ReadUsersetTuplesFilter, + _ storage.ReadUsersetTuplesOptions, +) (storage.TupleIterator, error) { + _, span := startTrace(ctx, "ReadUsersetTuples") + defer span.End() + + sb := s.stbl. + Select( + "store", "object_type", "object_id", "relation", + "_user", + "condition_name", "condition_context", "ulid", "inserted_at", + ). + From("tuple"). + Where(sq.Eq{"store": store}). + Where(sq.Eq{"user_type": tupleUtils.UserSet}) + + objectType, objectID := tupleUtils.SplitObject(filter.Object) + if objectType != "" { + sb = sb.Where(sq.Eq{"object_type": objectType}) + } + if objectID != "" { + sb = sb.Where(sq.Eq{"object_id": objectID}) + } + if filter.Relation != "" { + sb = sb.Where(sq.Eq{"relation": filter.Relation}) + } + if len(filter.AllowedUserTypeRestrictions) > 0 { + orConditions := sq.Or{} + for _, userset := range filter.AllowedUserTypeRestrictions { + if _, ok := userset.GetRelationOrWildcard().(*openfgav1.RelationReference_Relation); ok { + orConditions = append(orConditions, sq.Like{ + "_user": userset.GetType() + ":%#" + userset.GetRelation(), + }) + } + if _, ok := userset.GetRelationOrWildcard().(*openfgav1.RelationReference_Wildcard); ok { + orConditions = append(orConditions, sq.Eq{ + "_user": userset.GetType() + ":*", + }) + } + } + sb = sb.Where(orConditions) + } + if len(filter.Conditions) > 0 { + sb = sb.Where(sq.Eq{"COALESCE(condition_name, '')": filter.Conditions}) + } + + rg, err := sqlcommon.NewRowGetter(sqlcommon.NewSQLConnector(s.db), sb) + if err != nil { + return nil, err + } + return sqlcommon.NewSQLTupleIterator(rg, HandleSQLError), nil +} + +// ReadStartingWithUser see [storage.RelationshipTupleReader].ReadStartingWithUser. +func (s *Datastore) ReadStartingWithUser( + ctx context.Context, + store string, + filter storage.ReadStartingWithUserFilter, + _ storage.ReadStartingWithUserOptions, +) (storage.TupleIterator, error) { + _, span := startTrace(ctx, "ReadStartingWithUser") + defer span.End() + + var targetUsersArg []string + for _, u := range filter.UserFilter { + targetUser := u.GetObject() + if u.GetRelation() != "" { + targetUser = strings.Join([]string{u.GetObject(), u.GetRelation()}, "#") + } + targetUsersArg = append(targetUsersArg, targetUser) + } + + builder := s.stbl. + Select( + "store", "object_type", "object_id", "relation", + "_user", + "condition_name", "condition_context", "ulid", "inserted_at", + ). + From("tuple"). + Where(sq.Eq{ + "store": store, + "object_type": filter.ObjectType, + "relation": filter.Relation, + "_user": targetUsersArg, + }).OrderBy("object_id") + + if filter.ObjectIDs != nil && filter.ObjectIDs.Size() > 0 { + builder = builder.Where(sq.Eq{"object_id": filter.ObjectIDs.Values()}) + } + if len(filter.Conditions) > 0 { + builder = builder.Where(sq.Eq{"COALESCE(condition_name, '')": filter.Conditions}) + } + + rg, err := sqlcommon.NewRowGetter(sqlcommon.NewSQLConnector(s.db), builder) + if err != nil { + return nil, err + } + return sqlcommon.NewSQLTupleIterator(rg, HandleSQLError), nil +} + +// MaxTuplesPerWrite see [storage.RelationshipTupleWriter].MaxTuplesPerWrite. +func (s *Datastore) MaxTuplesPerWrite() int { + return s.maxTuplesPerWriteField +} + +// ReadAuthorizationModel see [storage.AuthorizationModelReadBackend].ReadAuthorizationModel. +func (s *Datastore) ReadAuthorizationModel(ctx context.Context, store string, modelID string) (*openfgav1.AuthorizationModel, error) { + ctx, span := startTrace(ctx, "ReadAuthorizationModel") + defer span.End() + + return sqlcommon.ReadAuthorizationModel(ctx, s.dbInfo, store, modelID) +} + +// ReadAuthorizationModels see [storage.AuthorizationModelReadBackend].ReadAuthorizationModels. +func (s *Datastore) ReadAuthorizationModels(ctx context.Context, store string, options storage.ReadAuthorizationModelsOptions) ([]*openfgav1.AuthorizationModel, string, error) { + ctx, span := startTrace(ctx, "ReadAuthorizationModels") + defer span.End() + + sb := s.stbl. + Select("authorization_model_id"). + Distinct(). + From("authorization_model"). + Where(sq.Eq{"store": store}). + OrderBy("authorization_model_id desc") + + if options.Pagination.From != "" { + token := options.Pagination.From + sb = sb.Where(sq.LtOrEq{"authorization_model_id": token}) + } + if options.Pagination.PageSize > 0 { + sb = sb.Limit(uint64(options.Pagination.PageSize + 1)) // + 1 is used to determine whether to return a continuation token. + } + + rows, err := sb.QueryContext(ctx) + if err != nil { + return nil, "", HandleSQLError(err) + } + defer rows.Close() + + var modelIDs []string + var modelID string + + for rows.Next() { + err = rows.Scan(&modelID) + if err != nil { + return nil, "", HandleSQLError(err) + } + + modelIDs = append(modelIDs, modelID) + } + + if err := rows.Err(); err != nil { + return nil, "", HandleSQLError(err) + } + + var token string + numModelIDs := len(modelIDs) + if len(modelIDs) > options.Pagination.PageSize { + numModelIDs = options.Pagination.PageSize + token = modelID + } + + // TODO: make this concurrent with a maximum of 5 goroutines. This may be helpful: + // https://stackoverflow.com/questions/25306073/always-have-x-number-of-goroutines-running-at-any-time + models := make([]*openfgav1.AuthorizationModel, 0, numModelIDs) + // We use numModelIDs here to avoid retrieving possibly one extra model. + for i := 0; i < numModelIDs; i++ { + model, err := s.ReadAuthorizationModel(ctx, store, modelIDs[i]) + if err != nil { + return nil, "", err + } + models = append(models, model) + } + + return models, token, nil +} + +// FindLatestAuthorizationModel see [storage.AuthorizationModelReadBackend].FindLatestAuthorizationModel. +func (s *Datastore) FindLatestAuthorizationModel(ctx context.Context, store string) (*openfgav1.AuthorizationModel, error) { + ctx, span := startTrace(ctx, "FindLatestAuthorizationModel") + defer span.End() + + return sqlcommon.FindLatestAuthorizationModel(ctx, s.dbInfo, store) +} + +// MaxTypesPerAuthorizationModel see [storage.TypeDefinitionWriteBackend].MaxTypesPerAuthorizationModel. +func (s *Datastore) MaxTypesPerAuthorizationModel() int { + return s.maxTypesPerModelField +} + +// WriteAuthorizationModel see [storage.TypeDefinitionWriteBackend].WriteAuthorizationModel. +func (s *Datastore) WriteAuthorizationModel(ctx context.Context, store string, model *openfgav1.AuthorizationModel) error { + ctx, span := startTrace(ctx, "WriteAuthorizationModel") + defer span.End() + + return sqlcommon.WriteAuthorizationModel(ctx, s.dbInfo, store, model) +} + +// CreateStore adds a new store to storage. +func (s *Datastore) CreateStore(ctx context.Context, store *openfgav1.Store) (*openfgav1.Store, error) { + ctx, span := startTrace(ctx, "CreateStore") + defer span.End() + + var id, name string + var createdAt, updatedAt time.Time + + txn, err := s.db.BeginTx(ctx, &sql.TxOptions{}) + if err != nil { + return nil, HandleSQLError(err) + } + defer func() { + _ = txn.Rollback() + }() + + _, err = s.stbl. + Insert("store"). + Columns("id", "name", "created_at", "updated_at"). + Values(store.GetId(), store.GetName(), sq.Expr("NOW()"), sq.Expr("NOW()")). + RunWith(txn). + ExecContext(ctx) + if err != nil { + return nil, HandleSQLError(err) + } + + err = s.stbl. + Select("id", "name", "created_at", "updated_at"). + From("store"). + Where(sq.Eq{"id": store.GetId()}). + RunWith(txn). + QueryRowContext(ctx). + Scan(&id, &name, &createdAt, &updatedAt) + if err != nil { + return nil, HandleSQLError(err) + } + + err = txn.Commit() + if err != nil { + return nil, HandleSQLError(err) + } + + return &openfgav1.Store{ + Id: id, + Name: name, + CreatedAt: timestamppb.New(createdAt), + UpdatedAt: timestamppb.New(updatedAt), + }, nil +} + +// GetStore retrieves the details of a specific store using its storeID. +func (s *Datastore) GetStore(ctx context.Context, id string) (*openfgav1.Store, error) { + ctx, span := startTrace(ctx, "GetStore") + defer span.End() + + row := s.stbl. + Select("id", "name", "created_at", "updated_at"). + From("store"). + Where(sq.Eq{ + "id": id, + "deleted_at": nil, + }). + QueryRowContext(ctx) + + var storeID, name string + var createdAt, updatedAt time.Time + err := row.Scan(&storeID, &name, &createdAt, &updatedAt) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return nil, storage.ErrNotFound + } + return nil, HandleSQLError(err) + } + + return &openfgav1.Store{ + Id: storeID, + Name: name, + CreatedAt: timestamppb.New(createdAt), + UpdatedAt: timestamppb.New(updatedAt), + }, nil +} + +// ListStores provides a paginated list of all stores present in the storage. +func (s *Datastore) ListStores(ctx context.Context, options storage.ListStoresOptions) ([]*openfgav1.Store, string, error) { + ctx, span := startTrace(ctx, "ListStores") + defer span.End() + + whereClause := sq.And{ + sq.Eq{"deleted_at": nil}, + } + + if len(options.IDs) > 0 { + whereClause = append(whereClause, sq.Eq{"id": options.IDs}) + } + + if options.Name != "" { + whereClause = append(whereClause, sq.Eq{"name": options.Name}) + } + + if options.Pagination.From != "" { + whereClause = append(whereClause, sq.GtOrEq{"id": options.Pagination.From}) + } + + sb := s.stbl. + Select("id", "name", "created_at", "updated_at"). + From("store"). + Where(whereClause). + OrderBy("id") + + if options.Pagination.PageSize > 0 { + sb = sb.Limit(uint64(options.Pagination.PageSize + 1)) // + 1 is used to determine whether to return a continuation token. + } + + rows, err := sb.QueryContext(ctx) + if err != nil { + return nil, "", HandleSQLError(err) + } + defer rows.Close() + + var stores []*openfgav1.Store + var id string + for rows.Next() { + var name string + var createdAt, updatedAt time.Time + err := rows.Scan(&id, &name, &createdAt, &updatedAt) + if err != nil { + return nil, "", HandleSQLError(err) + } + + stores = append(stores, &openfgav1.Store{ + Id: id, + Name: name, + CreatedAt: timestamppb.New(createdAt), + UpdatedAt: timestamppb.New(updatedAt), + }) + } + + if err := rows.Err(); err != nil { + return nil, "", HandleSQLError(err) + } + + if len(stores) > options.Pagination.PageSize { + return stores[:options.Pagination.PageSize], id, nil + } + + return stores, "", nil +} + +// DeleteStore removes a store from storage. +func (s *Datastore) DeleteStore(ctx context.Context, id string) error { + ctx, span := startTrace(ctx, "DeleteStore") + defer span.End() + + _, err := s.stbl. + Update("store"). + Set("deleted_at", sq.Expr("NOW()")). + Where(sq.Eq{"id": id}). + ExecContext(ctx) + if err != nil { + return HandleSQLError(err) + } + + return nil +} + +// WriteAssertions see [storage.AssertionsBackend].WriteAssertions. +func (s *Datastore) WriteAssertions(ctx context.Context, store, modelID string, assertions []*openfgav1.Assertion) error { + ctx, span := startTrace(ctx, "WriteAssertions") + defer span.End() + + marshalledAssertions, err := proto.Marshal(&openfgav1.Assertions{Assertions: assertions}) + if err != nil { + return err + } + + _, err = s.stbl. + Insert("assertion"). + Columns("store", "authorization_model_id", "assertions"). + Values(store, modelID, marshalledAssertions). + Suffix("ON DUPLICATE KEY UPDATE assertions = ?", marshalledAssertions). + ExecContext(ctx) + if err != nil { + return HandleSQLError(err) + } + + return nil +} + +// ReadAssertions see [storage.AssertionsBackend].ReadAssertions. +func (s *Datastore) ReadAssertions(ctx context.Context, store, modelID string) ([]*openfgav1.Assertion, error) { + ctx, span := startTrace(ctx, "ReadAssertions") + defer span.End() + + var marshalledAssertions []byte + err := s.stbl. + Select("assertions"). + From("assertion"). + Where(sq.Eq{ + "store": store, + "authorization_model_id": modelID, + }). + QueryRowContext(ctx). + Scan(&marshalledAssertions) + if err != nil { + if errors.Is(err, sql.ErrNoRows) { + return []*openfgav1.Assertion{}, nil + } + return nil, HandleSQLError(err) + } + + var assertions openfgav1.Assertions + err = proto.Unmarshal(marshalledAssertions, &assertions) + if err != nil { + return nil, err + } + + return assertions.GetAssertions(), nil +} + +// ReadChanges see [storage.ChangelogBackend].ReadChanges. +func (s *Datastore) ReadChanges(ctx context.Context, store string, filter storage.ReadChangesFilter, options storage.ReadChangesOptions) ([]*openfgav1.TupleChange, string, error) { + ctx, span := startTrace(ctx, "ReadChanges") + defer span.End() + + objectTypeFilter := filter.ObjectType + horizonOffset := filter.HorizonOffset + + orderBy := "ulid asc" + if options.SortDesc { + orderBy = "ulid desc" + } + + sb := s.stbl. + Select( + "ulid", "object_type", "object_id", "relation", + "_user", + "operation", + "condition_name", "condition_context", "inserted_at", + ). + From("changelog"). + Where(sq.Eq{"store": store}). + Where(fmt.Sprintf("inserted_at <= NOW() - INTERVAL %d MICROSECOND", horizonOffset.Microseconds())). + OrderBy(orderBy) + + if objectTypeFilter != "" { + sb = sb.Where(sq.Eq{"object_type": objectTypeFilter}) + } + if options.Pagination.From != "" { + sb = sqlcommon.AddFromUlid(sb, options.Pagination.From, options.SortDesc) + } + if options.Pagination.PageSize > 0 { + sb = sb.Limit(uint64(options.Pagination.PageSize)) // + 1 is NOT used here as we always return a continuation token. + } + + rows, err := sb.QueryContext(ctx) + if err != nil { + return nil, "", HandleSQLError(err) + } + defer rows.Close() + + var changes []*openfgav1.TupleChange + var ulid string + for rows.Next() { + var objectType, objectID, relation, user string + var operation int + var insertedAt time.Time + var conditionName sql.NullString + var conditionContext []byte + + err = rows.Scan( + &ulid, + &objectType, + &objectID, + &relation, + &user, + &operation, + &conditionName, + &conditionContext, + &insertedAt, + ) + if err != nil { + return nil, "", HandleSQLError(err) + } + + var conditionContextStruct structpb.Struct + if conditionName.String != "" { + if conditionContext != nil { + if err := proto.Unmarshal(conditionContext, &conditionContextStruct); err != nil { + return nil, "", err + } + } + } + + tk := tupleUtils.NewTupleKeyWithCondition( + tupleUtils.BuildObject(objectType, objectID), + relation, + user, + conditionName.String, + &conditionContextStruct, + ) + + changes = append(changes, &openfgav1.TupleChange{ + TupleKey: tk, + Operation: openfgav1.TupleOperation(operation), + Timestamp: timestamppb.New(insertedAt.UTC()), + }) + } + + if len(changes) == 0 { + return nil, "", storage.ErrNotFound + } + + return changes, ulid, nil +} + +// IsReady see [sqlcommon.IsReady]. +func (s *Datastore) IsReady(ctx context.Context) (storage.ReadinessStatus, error) { + versionReady, err := sqlcommon.IsReady(ctx, s.versionReady, s.db) + if err != nil { + return versionReady, err + } + s.versionReady = versionReady.IsReady + return versionReady, nil +} + +// HandleSQLError processes an SQL error and converts it into a more +// specific error type based on the nature of the SQL error. +func HandleSQLError(err error, args ...interface{}) error { + if errors.Is(err, sql.ErrNoRows) { + return storage.ErrNotFound + } + + var me *mysql.MySQLError + if errors.As(err, &me) && me.Number == 1062 { + if len(args) > 0 { + if tk, ok := args[0].(*openfgav1.TupleKey); ok { + return storage.InvalidWriteInputError(tk, openfgav1.TupleOperation_TUPLE_OPERATION_WRITE) + } + } + return storage.ErrCollision + } + + return fmt.Errorf("sql error: %w", err) +} diff --git a/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/LICENSE b/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/LICENSE new file mode 100644 index 0000000000..607a9c3c46 --- /dev/null +++ b/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/LICENSE @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2017-2019 Gianluca Arbezzano + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/Makefile b/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/Makefile new file mode 100644 index 0000000000..07c4087e8c --- /dev/null +++ b/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/Makefile @@ -0,0 +1,5 @@ +include ../../commons-test.mk + +.PHONY: test +test: + $(MAKE) test-mysql diff --git a/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/mysql.go b/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/mysql.go new file mode 100644 index 0000000000..adfe2c935c --- /dev/null +++ b/vendor/github.com/testcontainers/testcontainers-go/modules/mysql/mysql.go @@ -0,0 +1,202 @@ +package mysql + +import ( + "context" + "errors" + "fmt" + "path/filepath" + "strings" + + "github.com/testcontainers/testcontainers-go" + "github.com/testcontainers/testcontainers-go/wait" +) + +const ( + rootUser = "root" + defaultUser = "test" + defaultPassword = "test" + defaultDatabaseName = "test" +) + +// MySQLContainer represents the MySQL container type used in the module +type MySQLContainer struct { + testcontainers.Container + username string + password string + database string +} + +func WithDefaultCredentials() testcontainers.CustomizeRequestOption { + return func(req *testcontainers.GenericContainerRequest) error { + username := req.Env["MYSQL_USER"] + password := req.Env["MYSQL_PASSWORD"] + if strings.EqualFold(rootUser, username) { + delete(req.Env, "MYSQL_USER") + } + if len(password) != 0 && password != "" { + req.Env["MYSQL_ROOT_PASSWORD"] = password + } else if strings.EqualFold(rootUser, username) { + req.Env["MYSQL_ALLOW_EMPTY_PASSWORD"] = "yes" + delete(req.Env, "MYSQL_PASSWORD") + } + + return nil + } +} + +// Deprecated: use Run instead +// RunContainer creates an instance of the MySQL container type +func RunContainer(ctx context.Context, opts ...testcontainers.ContainerCustomizer) (*MySQLContainer, error) { + return Run(ctx, "mysql:8.0.36", opts...) +} + +// Run creates an instance of the MySQL container type +func Run(ctx context.Context, img string, opts ...testcontainers.ContainerCustomizer) (*MySQLContainer, error) { + moduleOpts := []testcontainers.ContainerCustomizer{ + testcontainers.WithExposedPorts("3306/tcp", "33060/tcp"), + testcontainers.WithEnv(map[string]string{ + "MYSQL_USER": defaultUser, + "MYSQL_PASSWORD": defaultPassword, + "MYSQL_DATABASE": defaultDatabaseName, + }), + testcontainers.WithWaitStrategy(wait.ForLog("port: 3306 MySQL Community Server")), + } + + moduleOpts = append(moduleOpts, opts...) + moduleOpts = append(moduleOpts, WithDefaultCredentials()) + + // Validate credentials after applying all options + validateCreds := func(req *testcontainers.GenericContainerRequest) error { + username, ok := req.Env["MYSQL_USER"] + if !ok { + username = rootUser + } + password := req.Env["MYSQL_PASSWORD"] + + if len(password) == 0 && password == "" && !strings.EqualFold(rootUser, username) { + return errors.New("empty password can be used only with the root user") + } + return nil + } + + moduleOpts = append(moduleOpts, testcontainers.CustomizeRequestOption(validateCreds)) + + ctr, err := testcontainers.Run(ctx, img, moduleOpts...) + var c *MySQLContainer + if ctr != nil { + c = &MySQLContainer{ + Container: ctr, + username: rootUser, // default to root, will be overridden if MYSQL_USER is set + } + } + + if err != nil { + return c, fmt.Errorf("run mysql: %w", err) + } + + // Retrieve credentials from container environment + inspect, err := ctr.Inspect(ctx) + if err != nil { + return c, fmt.Errorf("inspect mysql: %w", err) + } + + var foundUser, foundPass, foundDB bool + for _, env := range inspect.Config.Env { + if v, ok := strings.CutPrefix(env, "MYSQL_USER="); ok { + c.username, foundUser = v, true + } + if v, ok := strings.CutPrefix(env, "MYSQL_PASSWORD="); ok { + c.password, foundPass = v, true + } + if v, ok := strings.CutPrefix(env, "MYSQL_DATABASE="); ok { + c.database, foundDB = v, true + } + + if foundUser && foundPass && foundDB { + break + } + } + + return c, nil +} + +// MustConnectionString panics if the address cannot be determined. +func (c *MySQLContainer) MustConnectionString(ctx context.Context, args ...string) string { + addr, err := c.ConnectionString(ctx, args...) + if err != nil { + panic(err) + } + return addr +} + +func (c *MySQLContainer) ConnectionString(ctx context.Context, args ...string) (string, error) { + endpoint, err := c.PortEndpoint(ctx, "3306/tcp", "") + if err != nil { + return "", err + } + + extraArgs := "" + if len(args) > 0 { + extraArgs = strings.Join(args, "&") + } + if extraArgs != "" { + extraArgs = "?" + extraArgs + } + + connectionString := fmt.Sprintf("%s:%s@tcp(%s)/%s%s", c.username, c.password, endpoint, c.database, extraArgs) + return connectionString, nil +} + +func WithUsername(username string) testcontainers.CustomizeRequestOption { + return func(req *testcontainers.GenericContainerRequest) error { + req.Env["MYSQL_USER"] = username + + return nil + } +} + +func WithPassword(password string) testcontainers.CustomizeRequestOption { + return func(req *testcontainers.GenericContainerRequest) error { + req.Env["MYSQL_PASSWORD"] = password + + return nil + } +} + +func WithDatabase(database string) testcontainers.CustomizeRequestOption { + return func(req *testcontainers.GenericContainerRequest) error { + req.Env["MYSQL_DATABASE"] = database + + return nil + } +} + +func WithConfigFile(configFile string) testcontainers.CustomizeRequestOption { + return func(req *testcontainers.GenericContainerRequest) error { + cf := testcontainers.ContainerFile{ + HostFilePath: configFile, + ContainerFilePath: "/etc/mysql/conf.d/my.cnf", + FileMode: 0o755, + } + req.Files = append(req.Files, cf) + + return nil + } +} + +func WithScripts(scripts ...string) testcontainers.CustomizeRequestOption { + return func(req *testcontainers.GenericContainerRequest) error { + initScripts := make([]testcontainers.ContainerFile, 0, len(scripts)) + for _, script := range scripts { + cf := testcontainers.ContainerFile{ + HostFilePath: script, + ContainerFilePath: "/docker-entrypoint-initdb.d/" + filepath.Base(script), + FileMode: 0o755, + } + initScripts = append(initScripts, cf) + } + req.Files = append(req.Files, initScripts...) + + return nil + } +} diff --git a/vendor/modules.txt b/vendor/modules.txt index 1f54626e3b..04af30ed53 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -78,6 +78,10 @@ cloud.google.com/go/storage/internal/apiv2/storagepb # dario.cat/mergo v1.0.2 ## explicit; go 1.13 dario.cat/mergo +# filippo.io/edwards25519 v1.2.0 +## explicit; go 1.24.0 +filippo.io/edwards25519 +filippo.io/edwards25519/field # github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c ## explicit; go 1.16 github.com/Azure/go-ansiterm @@ -508,6 +512,9 @@ github.com/go-openapi/swag/typeutils # github.com/go-openapi/swag/yamlutils v0.28.0 ## explicit; go 1.25.0 github.com/go-openapi/swag/yamlutils +# github.com/go-sql-driver/mysql v1.10.1 +## explicit; go 1.24.0 +github.com/go-sql-driver/mysql # github.com/go-viper/mapstructure/v2 v2.5.0 ## explicit; go 1.18 github.com/go-viper/mapstructure/v2 @@ -903,6 +910,7 @@ github.com/openfga/openfga/pkg/server/config github.com/openfga/openfga/pkg/server/errors github.com/openfga/openfga/pkg/storage github.com/openfga/openfga/pkg/storage/cache/keys +github.com/openfga/openfga/pkg/storage/mysql github.com/openfga/openfga/pkg/storage/postgres github.com/openfga/openfga/pkg/storage/sqlcommon github.com/openfga/openfga/pkg/storage/storagewrappers @@ -1059,6 +1067,9 @@ github.com/testcontainers/testcontainers-go/internal/core/bootstrap github.com/testcontainers/testcontainers-go/internal/core/network github.com/testcontainers/testcontainers-go/log github.com/testcontainers/testcontainers-go/wait +# github.com/testcontainers/testcontainers-go/modules/mysql v0.44.0 +## explicit; go 1.25.0 +github.com/testcontainers/testcontainers-go/modules/mysql # github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0 ## explicit; go 1.25.0 github.com/testcontainers/testcontainers-go/modules/postgres