Skip to content

testing.md

Latest commit

 

History

History
337 lines (258 loc) · 20 KB

testing.md

File metadata and controls

337 lines (258 loc) · 20 KB
Raw

(testing)=

Testing

PyAthena's main test suites use real AWS services. Read the contribution guide for the validation required with a pull request. Prepare an environment you control before running tests; the project does not provide contributors with AWS credentials or test resources.

Development tools

Use a Python version supported by pyproject.toml and install uv and just. From the repository root, install the development dependencies and check the code:

uv sync --group dev
just format
just lint

Run just lint before tests, including when invoking pytest directly. For documentation checks, install the tools pinned in .mise.toml with mise:

mise install
just docs lint
just docs build

just docs build builds master, the latest patch release of the newest minor versions (SMV_MINOR_VERSIONS in docs/conf.py), and the latest release of the previous major version if that tag contains docs/conf.py, with sphinx-multiversion. To check the working tree, including uncommitted documentation changes, also run:

uv run sphinx-build -b html docs docs/_build/current

AWS environment

Use a dedicated test account or isolated test resources, not production resources. You are responsible for AWS charges and for removing resources and data left by tests, including after an interrupted run. Configure AWS credentials using the normal boto3 credential chain, for example a local AWS profile. Do not commit credentials or include them in test output shared on a pull request.

The tests need an S3 bucket and prefix for staging and data, an Athena SQL workgroup with an S3 query-result location, and an Athena Spark workgroup with a suitable execution role for Spark tests. The local test identity needs the corresponding Athena, S3, and Glue permissions; additional features require their own service permissions. The Glue metadata fallback tests call glue:GetTable, glue:GetTables, and glue:GetDatabases directly, including against the S3 Tables catalog; the template below grants them. The test infrastructure template describes the resources and permissions used by project CI. Its GitHub OIDC role is not a local credential setup: contributors must configure their own test identity.

Create a gitignored .env file in the repository root, replacing the example values:

AWS_PROFILE=your-test-profile
AWS_DEFAULT_REGION=us-west-2
AWS_ATHENA_S3_STAGING_DIR=s3://your-test-bucket/pyathena-tests/
AWS_ATHENA_WORKGROUP=your-sql-workgroup
AWS_ATHENA_SPARK_WORKGROUP=your-spark-workgroup

Keep the trailing slash on the staging location. If using credentials through another supported mechanism, omit AWS_PROFILE. The test configuration requires all four AWS_DEFAULT_REGION and AWS_ATHENA_* variables above even when the selected tests do not use Spark. Some fixtures use the primary SQL workgroup by default; if it is unavailable or unsuitable, also set:

AWS_ATHENA_DEFAULT_WORKGROUP=your-sql-workgroup

In a git worktree, just worktree-env links the main checkout's gitignored .env. The link alone does not load its variables; use uv run --env-file .env as shown below.

Managed query result storage

Tests of managed query result storage need a workgroup with managed storage enabled:

AWS_ATHENA_MANAGED_WORKGROUP=your-managed-workgroup

Amazon S3 Tables

SQLAlchemy tests of Amazon S3 Tables need a table bucket integrated with the AWS analytics services and its registered catalog:

AWS_ATHENA_S3_TABLES_CATALOG=s3tablescatalog/your-table-bucket

Each test process creates its own namespace in the table bucket, named like its schema, and deletes it with any remaining tables at the end; with pytest-xdist that is each worker, not the controller. The test identity needs s3tables:CreateNamespace, s3tables:DeleteNamespace, s3tables:ListTables, and s3tables:DeleteTable on the table bucket. A session that stops early leaves its namespace behind; scripts/sweep_databases.py removes such namespaces once they are more than one day old. It also removes the database and namespace of a session still running after one day, if they are in the account, region, and table bucket that the sweep covers.

The S3 Tables tests live in tests/pyathena/sqlalchemy/test_base.py and tests/pyathena/test_glue.py and run under just test pyathena, not the SQLAlchemy compliance-suite commands. Managed storage and S3 Tables tests skip when their respective optional configuration is absent. If a change affects one of these features, configure and run its tests; a skip does not validate that change.

Run tests

The test session hooks upload fixture data to S3 and create and remove Athena/Glue databases and tables. Selecting a single test under tests/pyathena/ still invokes the session setup, even when the test itself uses only mocks or pure Python logic. Do not assume that -k or a single test path makes this suite offline.

Run the suites relevant to the change:

uv run --env-file .env just test pyathena
uv run --env-file .env just test sqla
uv run --env-file .env just test sqla-async

Despite the recipe's "unit tests" label, just test pyathena includes AWS integration tests. The recipes use eight pytest workers, so check your account's Athena concurrency quotas. They also rerun a failed test once when its failure message is an Athena internal error or Invalid S3 request, and report the first attempt's traceback. Failures in the session setup hooks are not rerun, and a direct pytest invocation does not rerun. Do not overlap test runs that share AWS resources or quota. For a focused run with a single worker, for example:

uv run --env-file .env pytest -n 1 tests/pyathena/test_cursor.py -v

A targeted run helps during development but does not replace other coverage required by the affected callers or features.

(testing-offline)=

Run self-contained tests offline

The pandas and Polars result-set modules have self-contained tests that can run without AWS access when the session hooks are excluded. Reuse the .env file described in the AWS environment section if it is already configured. For an offline-only setup, create a gitignored .env file in the repository root with these placeholder values:

AWS_DEFAULT_REGION=us-east-1
AWS_ATHENA_S3_STAGING_DIR=s3://pyathena-offline-placeholder/
AWS_ATHENA_WORKGROUP=offline
AWS_ATHENA_SPARK_WORKGROUP=offline
AWS_EC2_METADATA_DISABLED=true

After just lint, load .env and run:

uv run --env-file .env pytest --noconftest -p no:rerunfailures -q \
  tests/pyathena/pandas/test_result_set.py \
  tests/pyathena/polars/test_result_set.py

The four AWS configuration values are required by tests/__init__.py, which pytest still imports with --noconftest. The offline-only .env example disables EC2 metadata to prevent implicit credential lookup through that service. --noconftest excludes the AWS session hooks and fixtures; disabling the rerun plugin also avoids its local socket setup in restricted environments. Use this invocation only for self-contained modules; integration tests need their normal fixtures and a real AWS environment.

S3 null-version moves

The null-version move regression tests create three dedicated temporary buckets: unversioned, versioning-enabled, and versioning-suspended. They run only when explicitly enabled, because provisioning requires additional permissions and a 15-minute wait after first enabling versioning. Run them serially:

AWS_ATHENA_S3_VERSIONING_TESTS=1 uv run --env-file .env pytest -n 1 \
  tests/pyathena/filesystem/test_s3.py \
  tests/pyathena/filesystem/test_s3_async.py -k move_null_version_onto_key -v

These nine cases belong to TestS3FileSystem and TestAioS3FileSystem and use their existing filesystem fixtures. The shared session-scoped bucket fixture in tests/pyathena/filesystem/conftest.py provisions the three buckets once per worker, so a serial run shares one setup and wait across both test modules. The synchronous API, asynchronous API, and asynchronous filesystem's synchronous wrapper each have a separate test method.

The test identity needs s3:CreateBucket, s3:GetBucketVersioning, s3:PutBucketVersioning, s3:PutObject, s3:GetObject, s3:GetObjectVersion, s3:ListBucketVersions, s3:DeleteObjectVersion, and s3:DeleteBucket on the temporary pyathena-mv-null-* buckets and their objects, in addition to the normal test session permissions. The fixture deletes all versions and delete markers, then removes its buckets, including when a test fails. An interrupted process may leave buckets behind; remove only the buckets created by that run.

Ordinary CI does not enable these tests. Its state-specific mock regression tests and existing AWS suite still run; report the opt-in results separately.

SQLAlchemy suites

The SQLAlchemy compliance suites under tests/sqlalchemy/ run with different configurations: use sqla for synchronous dialects and sqla-async for native asyncio dialects. They do not run PyAthena's own dialect regression tests under tests/pyathena/sqlalchemy/ and tests/pyathena/aio/sqlalchemy/. Run the relevant PyAthena tests too, either through just test pyathena or a focused selection during development:

uv run --env-file .env pytest -n 1 tests/pyathena/sqlalchemy/ tests/pyathena/aio/sqlalchemy/ -v

Run tox

To invoke the configured tox environments locally:

uv run --env-file .env just tox

Record results

Record the tested commit, Python and relevant dependency versions, exact commands, and results in the pull request. Include failed and skipped tests and explain any unrun coverage. Separate real AWS results from mock-based tests and static checks. Sanitize logs before sharing them.

Choose what to test

Each test protects a behavior that a caller can observe and that no other test checks with the same kind of evidence. Name that contract before writing the test, and check whether an existing test or parameter already covers it.

Check behavior against Athena and Amazon S3 where the services can observe it. Run only the queries and requests that the contract needs, and put the values a test checks in the columns of one query where possible. Use offline tests with mocks or botocore's Stubber for logic that the services cannot exercise deterministically, such as concurrency, combinations of failures and aborts, and interrupts, and try the relevant patterns in one parametrized test. Inject failures and interrupts through public objects, such as the S3 client, the executor, a future, or Stubber. Patch a private method only to reach a failure point that no public object reaches, and assert the public behavior that follows.

Do not add a test that:

  • asserts what the code literally states, such as a stored constructor argument, an internal default or constant, or a __repr__, or tests Python or a third-party library instead of PyAthena;
  • only exercises a branch or a private helper that has no contract a caller can observe;
  • repeats another test's contract with the same kind of evidence, including a copy of a synchronous test for an asynchronous or aio variant that runs through the same code;
  • asserts the order, count, or exact arguments of internal calls that are not part of the contract;
  • matches exception or log message text that is not documented, or dtypes and reprs that a Python, pandas, Polars, or PyArrow version determines rather than PyAthena;
  • depends on thread or task timing, sleeps, or ordering between workers without synchronizing it;
  • passes without the behavior it names because a mock or fake omits that behavior.

The exception type that a public API raises is a contract; its message text usually is not. A documented public value, such as pyathena.threadsafety, is a contract. When the result type that PyAthena produces depends on a supported library version, state the expected type for each supported version. An offline test and a real-service test of the same contract are different kinds of evidence only when the real-service test is opt-in and skipped by default, or the offline test reaches cases that the service cannot produce on demand. Test asynchronous and aio variants where they have their own code, such as their own cancellation, shutdown, or request scheduling.

Before submitting a test, remove or break the behavior it names, confirm that the test fails, and restore the behavior. Bound every wait in a test so that a regression fails instead of hanging. Before removing a test of a behavior that remains supported, confirm that another test fails without that behavior. When a change intentionally replaces a behavior, replace its tests with tests of the new behavior.

Organize tests

Group cursor and engine integration tests in classes, and use standalone functions for stateless helpers. A unit-test class can group the behavior of one object or common setup, as in TestTypeSignatureParser. Function-oriented utility tests can remain standalone when they use AWS fixtures; tests/pyathena/pandas/test_util.py follows this pattern. SQLAlchemy compliance tests retain the classes, decorators, and plugin setup required by the upstream suite. Preserve attribution for adapted tests as documented in NOTICE.

Parameter definitions should make inputs, options, and expected behavior visible. Build mocks, configured result sets, and one-shot readers during fixture setup or test execution. Pure values and framework type or expression objects can be constructed in parameters. Choose explicit parameter IDs when generated IDs obscure the case, and keep existing IDs, marks, and fixture scopes when reorganizing tests.

Use literal expected values for contract-specific behavior. A library comparison is useful when matching that library is the contract, such as joining pandas chunks versus reading the whole file. Express intentional differences directly rather than recreating the implementation in the expected-value builder. Retain dtype and schema checks, option contexts, and resource cleanup where applicable.

GitHub Actions

The Test workflow runs for pull requests that change files other than docs/ and Markdown. It runs the offline checks (just lint) on each of them, including Drafts and external forks, and runs the AWS suites as follows:

Trigger PyAthena suite SQLAlchemy tests Spark tests Python versions
Draft pull request No No No None
Ready pull request from a branch of this repository When related files change When related files change When related files change Newest supported
Weekly schedule Yes Yes Yes Newest supported
Manual dispatch Yes Yes Yes Requested, or all supported
Release tag (Release workflow) Yes Yes Yes All supported

The SQLAlchemy tests are the compliance suites and the PyAthena suite's tests/pyathena/sqlalchemy/ and tests/pyathena/aio/sqlalchemy/. The Spark tests are the PyAthena suite's tests/pyathena/spark/ and tests/pyathena/aio/spark/. When the PyAthena suite runs but the SQLAlchemy or Spark tests do not, it runs without them. For the PyAthena suite, the related files are pyathena/, tests/, and setup.cfg. For the SQLAlchemy tests, they are pyathena/sqlalchemy/, pyathena/aio/sqlalchemy/, tests/sqlalchemy/, their PyAthena suite test directories, and setup.cfg. For the Spark tests, they are pyathena/spark/, pyathena/aio/spark/, and their PyAthena suite test directories. Changes to the modules directly under pyathena/ and pyathena/aio/, the shared test fixtures such as tests/pyathena/conftest.py and tests/resources/, pyproject.toml, uv.lock, justfile, or the Test workflows run all three. For a pull request from a branch of this repository that still changes files other than docs/ and Markdown, marking the Draft ready for review starts any AWS jobs its changed files select, and converting it back to Draft cancels AWS jobs still running. To run every suite on a branch, dispatch the workflow; it tests every supported Python version unless python-versions lists some of them:

gh workflow run test.yaml --ref <branch>
gh workflow run test.yaml --ref <branch> -f python-versions=3.11,3.14

The Release workflow runs the same suites on every supported Python version for the tagged commit before building, and publishes nothing unless all of them pass. If they fail, nothing is published, and the documentation leaves out the tag because it only lists tags with a GitHub release; delete the tag, fix the failure, and push the tag again.

Project policy excludes external-fork pull requests from AWS integration CI. Maintainers do not approve those jobs as a substitute for contributor testing. Checks without AWS access may still run on a fork pull request.

For maintainers, the reusable test workflow uses OIDC to access the project's AWS environment. Do not treat workflow gating as a substitute for approval controls and AWS IAM trust restrictions. Do not bypass the fork policy by checking out external pull-request code in a privileged workflow or copying it to an internal branch just to run AWS CI.

Contributors can run tests locally with their own AWS credentials without configuring GitHub OIDC. If setting up CI in your own fork, replace the project-specific role ARN and resource settings in .github/workflows/test-suite.yaml with your own values. The GitHub OIDC documentation explains the authentication setup. You can configure tests against suitable existing resources, or provision new resources with the optional CloudFormation template. Review its IAM permissions and resources first. The following create-stack example is for new resources: it creates the staging bucket, table bucket, roles, and workgroups and does not adopt resources already created during manual setup. Choose unused names for all of them. The template also creates the fixed regional Glue catalog s3tablescatalog; do not use this creation example in an account/region where that catalog already exists. Use existing-resource configuration or plan an explicit infrastructure import/adaptation separately.

For a new stack, after replacing every placeholder and checking the names and regional catalog:

aws --region us-west-2 cloudformation create-stack \
  --stack-name pyathena-test-infrastructure \
  --capabilities CAPABILITY_NAMED_IAM \
  --template-body file://./cloudformation/github_actions_oidc.yaml \
  --parameters ParameterKey=GitHubOrg,ParameterValue=YOUR_GITHUB_USER \
    ParameterKey=RepositoryName,ParameterValue=PyAthena \
    ParameterKey=BucketName,ParameterValue=YOUR_UNIQUE_TEST_BUCKET \
    ParameterKey=S3TablesBucketName,ParameterValue=YOUR_UNIQUE_TABLE_BUCKET \
    ParameterKey=RoleName,ParameterValue=pyathena-test-ci \
    ParameterKey=SparkRoleName,ParameterValue=pyathena-test-spark \
    ParameterKey=WorkGroupName,ParameterValue=pyathena-test \
    ParameterKey=SparkWorkGroupName,ParameterValue=pyathena-test-spark \
    ParameterKey=ManagedWorkGroupName,ParameterValue=pyathena-test-managed

If the account already has the GitHub OIDC provider, supply its ARN as the OIDCProviderArn parameter. The stack does not grant your local identity access automatically, and additional account-level service configuration may be needed for the features you test. The staging bucket uses CloudFormation DeletionPolicy: Retain, so deleting the stack does not remove it; account for that in cleanup and before reusing its name in a new stack. See the infrastructure guide for updates to an existing stack.