Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions docs/development/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,80 @@ For connector changes, start with the [public test command](public-tests.md).
It builds its prerequisites and runs without cloud credentials. The
[test matrix](../connector-test-matrix.md) lists coverage and CI limits.

## Credential profiles

The opt-in [profile registry and launcher](../../tools/ci_test_profiles.py)
separate test selection from isolated execution. GCP profiles start pytest
with an allowlisted environment and temporary credential/configuration
directories. General and Snowflake profiles currently support only collection
through this launcher: their remaining Azure and Databricks dependencies have
not been partitioned, so isolated execution fails before setup rather than
silently removing credentials and losing live coverage.

| Profile | Credentials and ownership |
| --- | --- |
| `general` | Non-GCP selection; isolated launcher is collection-only during migration |
| `snowflake` | Non-GCP selection with `--runsnowflake`; isolated launcher is collection-only during migration |
| `gcp` | GCS/BigQuery tests with federated ADC; local storage for REST integration tests |
| `gcp-aws` | BigQuery training integration test with federated ADC and the S3 dataplane |

Run from the checkout root with its test dependencies installed:

```bash
.venv/bin/python -m tools.ci_test_profiles run gcp \
test/io_utils/fsutil/test_gcs.py -q
.venv/bin/python -m tools.ci_test_profiles run gcp-aws
```

During migration, general and Snowflake execution uses pytest directly with
the selection plugin, preserving the existing credential supply:

```bash
.venv/bin/python -m pytest -p tools.pytest_test_profiles \
--test-profile general test/connector/
.venv/bin/python -m pytest -p tools.pytest_test_profiles \
--test-profile snowflake --runsnowflake test/connector/
```

These direct commands select tests but do not scrub the environment or isolate
credential files. They are a transitional coverage-preserving path, not a
claim that general and Snowflake processes already have minimal credentials.

GCP execution requires an explicit `GOOGLE_APPLICATION_CREDENTIALS` file of
type `external_account`; ambient user credentials and stored service-account
keys do not activate these profiles. Set `GCP_TEST_SERVICE_ACCOUNT` and
`GCP_PROJECT_ID` for identity-backed REST tests, plus
`GCP_BIGQUERY_NO_TABLE_SERVICE_ACCOUNT` and
`GCP_BIGQUERY_NO_JOB_SERVICE_ACCOUNT` for BigQuery permission tests.
Missing required credentials fail setup. The
launcher copies ADC into its temporary directory and preserves its external
token-source references, which must remain accessible for the run.

The GCP-only runtime removes AWS secret fallback and uses a local artifact
directory. The GCP+AWS profile retains S3 because the training test actually
uses it. Credentials are prepared before worker startup; function-specific
identity changes are restored during teardown. These profiles constrain
credential inputs, not network access. Use the public Docker suites when
network isolation is required.

For a collection audit, `--manifest` saves the selected node IDs without
running fixtures:

```bash
.venv/bin/python -m tools.ci_test_profiles run gcp \
--manifest /tmp/gcp-nodes.json --collect-only
```

For the unpartitioned baseline, run pytest over the same source revision and
target scope with `-p tools.pytest_test_profiles --collect-only
--test-profile-manifest` and no profile. Its output contains a
`KUMO_TEST_PROFILE_MANIFEST=` JSON record. Compare its `nodeids` with the union
of the `general`, `gcp`, and `gcp-aws` manifests, checking both missing nodes
and overlaps. Audit the Snowflake mode separately with `--runsnowflake`;
existing Snowflake skip rules are independent of GCP ownership. Collection
still imports shared conftests and requires their dependencies. A collection
manifest or local plugin test does not verify live provider access.

## Report evidence

Record the command, environment, result, and any untested dependency. Keep
Expand Down
56 changes: 52 additions & 4 deletions kumo/testing/bigquery.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
ENV_GOOGLE_CREDENTIALS_DEFAULT_PATH,
ENV_GOOGLE_CREDENTIALS_JSON,
)
from tools.ci_test_profiles import TestProfile, profile_for

DEFAULT_BIGQUERY_KEY = 'kumo-test-bigquery-service-account-key'
FAIL_TABLE_PERMISSION_BIGQUERY_KEY = 'kumo-test-permission-fail-bigquery-service-account-key-2'
Expand Down Expand Up @@ -49,6 +50,11 @@ class CredentialKind(Enum):
KEY = 'service_account'


def configured_profile() -> TestProfile | None:
name = os.environ.get('KUMO_TEST_PROFILE')
return profile_for(name) if name else None


def credential_kind(path: str) -> CredentialKind:
if not fsutil.exists(path) or fsutil.isdir(path):
raise RuntimeError('Configured BigQuery credential file does not exist')
Expand Down Expand Up @@ -83,11 +89,20 @@ def setup_bigquery_credentials(
if previous is None:
environment.delenv(name)

path = (os.environ.get('GOOGLE_APPLICATION_CREDENTIALS')
or os.environ.get(ENV_GOOGLE_CREDENTIALS_JSON))
if not path and fsutil.exists(ENV_GOOGLE_CREDENTIALS_DEFAULT_PATH):
profile = configured_profile()
profiled_gcp = profile is not None and 'gcp' in profile.capabilities
path = os.environ.get('GOOGLE_APPLICATION_CREDENTIALS')
if profiled_gcp and not path:
raise RuntimeError('GCP test profiles require explicit '
'GOOGLE_APPLICATION_CREDENTIALS')
if not profiled_gcp:
path = path or os.environ.get(ENV_GOOGLE_CREDENTIALS_JSON)
if (not profiled_gcp and not path
and fsutil.exists(ENV_GOOGLE_CREDENTIALS_DEFAULT_PATH)):
path = ENV_GOOGLE_CREDENTIALS_DEFAULT_PATH
kind = credential_kind(path) if path else None
if profiled_gcp and kind != CredentialKind.FEDERATED:
raise RuntimeError('GCP test profiles require external_account ADC')
target_variable = _IMPERSONATION_ENV_BY_KEY.get(key)
target = None
if kind is not None and kind != CredentialKind.KEY and target_variable:
Expand Down Expand Up @@ -131,9 +146,42 @@ def setup_bigquery_credentials(
fsutil.rm(temporary_dir)


@pytest.fixture(scope='session', autouse=True)
def configure_profile_credentials():
"""Prepare ADC before session resources and worker processes start."""
profile = configured_profile()
if profile is None or 'gcp' not in profile.capabilities:
yield
return
with setup_bigquery_credentials():
yield


@pytest.fixture(autouse=True)
def configure_bigquery_credentials(
request: pytest.FixtureRequest,
configure_profile_credentials,
):
marker = request.node.get_closest_marker('bigquery_credentials')
if configured_profile() is None or marker is None:
yield
return
key, = marker.args
if key is not None and not isinstance(key, str):
raise TypeError('bigquery_credentials requires a string identity key')
with setup_bigquery_credentials(key):
yield


def withBigQueryCredentials(func: Optional[Callable] = None,
service_account_key: Optional[str] = None):
def decorator(inner_func: Callable):
marked = pytest.mark.requires_credentials('gcp')(
pytest.mark.bigquery_credentials(service_account_key)(inner_func))
if configured_profile() is not None:
return marked
# Temporary unprofiled path: existing workflows retain phase-one
# behavior until all invocations use the profile launcher.
if asyncio.iscoroutinefunction(inner_func):

async def async_wrapper(*args, **kwargs):
Expand All @@ -148,6 +196,6 @@ def sync_wrapper(*args, **kwargs):
return inner_func(*args, **kwargs)

wrapper = sync_wrapper
return functools.wraps(inner_func)(wrapper)
return functools.wraps(marked)(wrapper)

return decorator if func is None else decorator(func)
Loading
Loading