ETL pipeline orchestration using Apache Airflow and DBT. Handles data extraction, transformation, and loading with automated workflows and data quality checks.
| Environment variables | Value | Description |
|---|---|---|
| GCP_PROJECT_ID | ||
| GCP_SERVICE_ACCOUNT_KEY_JSON_FILEPATH | ||
| GCP_SERVICE_ACCOUNT_KEY | ||
| POSTGRES_DB | ||
| POSTGRES_PASSWORD | ||
| POSTGRES_USER | ||
| DOCKER_GCP_SERVICE_ACCOUNT_KEY_JSON_FILEPATH | ||
| AIRFLOW_ADMIN_USERNAME | ||
| AIRFLOW_ADMIN_PASSWORD | ||
| AIRFLOW_ADMIN_EMAIL | ||
| AIRFLOW_ADMIN_FIRST_NAME | ||
| AIRFLOW_ADMIN_LAST_NAME | ||
| GCP_BUCKET | ||
| GOOGLE_MAPS_PLATFORM_API_KEY | ||
| GOOGLE_MAPS_PLATFORM_API_LIMIT | ||
| TRIPADISOR_API_KEY | ||
| TRIPADISOR_API_LIMIT | ||
| GCP_ZONE | ||
| GCP_INSTANCE_NAME |
cd suretbon && uv run dbt deps && uv run dbt testcd suretbon && uv run dbt run --selector osm-france-food-service_export_alimconfiancecd suretbon && uv run dbt run --selector restaurantscd suretbon && uv run dbt cleanrm -rf suretbon/logsrm -f suretbon/.user.yml