diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..1a76911 --- /dev/null +++ b/.env.example @@ -0,0 +1,5 @@ +# Copy to .env (git-ignored) and fill in. Never commit real values. +DBRICKS_HOST="adb-xxxx.azuredatabricks.net" +DBRICKS_HTTP_PATH="/sql/1.0/warehouses/xxxxxxxx" +DBRICKS_TOKEN="dapi..." +DBT_SCHEMA="dev_yourname" diff --git a/.github/workflows/grade-assignment.yml b/.github/workflows/grade-assignment.yml deleted file mode 100644 index d06dc2d..0000000 --- a/.github/workflows/grade-assignment.yml +++ /dev/null @@ -1,13 +0,0 @@ -name: Grade Assignment - -on: - pull_request_target: - branches: - - main - -jobs: - grade: - permissions: - issues: write - pull-requests: write - uses: HackYourFuture/github-actions/.github/workflows/auto-grade.yml@main diff --git a/.hyf/README.md b/.hyf/README.md deleted file mode 100644 index 38f1a4d..0000000 --- a/.hyf/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# Auto grade tool - -## How it works -1. The auto grade tool runs the `test.sh` script located in this directory. -2. `test.sh` should write to a file named `score.json` with following JSON format: - ```json - { - "score": , - "passingScore": , - "pass": "" - } - ``` - All scores are out of 100. It is up to the assignment to determine how to calculate the score. -3. The auto grade runs via a github action on PR creation and updates the PR with the score. - diff --git a/.hyf/score.example.json b/.hyf/score.example.json deleted file mode 100644 index 8d931f5..0000000 --- a/.hyf/score.example.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "score": 75, - "pass": true, - "passingScore": 50 -} \ No newline at end of file diff --git a/.hyf/test.sh b/.hyf/test.sh deleted file mode 100644 index ee037fc..0000000 --- a/.hyf/test.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -# Run your test scripts here. -# Auto grade tool will execute this file within the .hyf working directory. -# The result should be stored in score.json file with the format shown below. -cat << EOF > score.json -{ - "score": 0, - "pass": true, - "passingScore": 0 -} -EOF diff --git a/README.md b/README.md index 96ce7bc..b124e09 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,47 @@ -# [Track] week X assignment -HackYourFuture week X assignment -The Week X assignment for the HackYourFuture can be found at the following link: [TODO: Assignment url in the learning platform] +# Data Track — Week 13 Assignment: Databricks Lab +Demonstrate the core skill of Week 13: running the analytics-engineering workflow you already know, at platform scale on **Databricks**. There is **no autograder** here: this assignment is reviewed by your teacher against the rubric below. You submit by opening a pull request with your work in the `task-1/` and `task-2/` folders. -## Implementation Instructions +> ⚠️ **Never commit your Databricks token or connection string.** Keep them in environment variables, exactly as you did with the Postgres password in Week 12. A committed secret is an automatic fail and a real-world security incident. -Provide clear instructions on how trainees should implement the tasks. +## Task 1 — PySpark exploration (required) -### Task 1 -Instructions for Task 1 +In a Databricks notebook, read `hyf.nyc_yellow.raw_trips` into a PySpark DataFrame and answer two questions using only transformations and a single action (`show()`, not `collect()` on the raw table): -### Task 2 -Instructions for Task 2 +- Which pickup borough has the most trips? (Join `hyf.nyc_yellow.raw_zones` on `pickup_location_id = location_id` to turn the location id into a `borough`.) +- What is the average `total_amount` per `payment_type`? -... +Then write two or three sentences on when you would reach for PySpark versus dbt SQL for a transformation. +**Deliverable:** export your notebook (`.ipynb` or `.py`) into `task-1/`. See `task-1/README.md`. + +## Task 2 — dbt on Databricks (required) + +Port your Week 10 dbt project to Databricks and run `fct_trips` as an incremental model. + +1. Install `dbt-databricks` and configure a Databricks target in `profiles.yml` (token in an environment variable, never committed). +2. Confirm `dbt debug` passes against Databricks. +3. Change `fct_trips` to `materialized='incremental'` with `incremental_strategy='merge'` and a correct `unique_key`. +4. Run `dbt build --select fct_trips` twice and capture both run times. + +**Deliverable:** your dbt project in `task-2/` (secrets and `target/` excluded) and the two build times plus explanation in `task-2/WRITEUP.md`. See `task-2/README.md`. + +## Task 3 — optional bonuses + +These build on the optional Going Further topics and do not affect whether the assignment passes. + +- **Governance:** run `SHOW GRANTS` on `fct_trips`, capture its lineage graph, and write the `ALTER TABLE ... SET TAGS` statement you would use to tag a hypothetical PII column. +- **Streaming:** run the `rate`-source Structured Streaming demo, write the results to a Delta table, and describe what you observed. + +## How to submit + +1. **Fork** this repository. +2. Add your work to `task-1/` and `task-2/` (each folder has a README telling you what goes there). +3. Double-check that no token or connection string is committed anywhere (`.env` is git-ignored; a `.env.example` is provided as a template). +4. Open a **pull request** back to this repository. Your teacher reviews it against the rubric below. + +## Rubric — what good looks like + +- **PySpark:** your answers use a small aggregated result displayed with `show()`, not a `collect()` of the raw table; a sensible PySpark-versus-dbt answer. +- **dbt:** the project runs on Databricks with only the connection profile changed from Week 10; the `fct_trips` incremental config is correct (`merge`, a real `unique_key`, `>` not `>=`); the second `dbt build` is visibly faster than the first, and you can explain why. +- **Secrets:** no Databricks token or connection string committed anywhere. diff --git a/task-1/README.md b/task-1/README.md new file mode 100644 index 0000000..2779b32 --- /dev/null +++ b/task-1/README.md @@ -0,0 +1,13 @@ +# Task 1 — PySpark exploration + +Put your exported PySpark notebook here (`.ipynb` or `.py`). + +It should: + +- Read `hyf.nyc_yellow.raw_trips` into a DataFrame. +- Answer the two questions using only PySpark transformations and a single action (`show()`): + - Which pickup borough has the most trips? (Join `hyf.nyc_yellow.raw_zones` on `pickup_location_id = location_id`.) + - What is the average `total_amount` per `payment_type`? +- Include your two-to-three sentences on PySpark versus dbt SQL (a markdown cell or a comment is fine). + +Do **not** use `collect()` on the raw table. diff --git a/task-1/task 1 files b/task-1/task 1 files deleted file mode 100644 index e69de29..0000000 diff --git a/task-2/README.md b/task-2/README.md new file mode 100644 index 0000000..4efcce7 --- /dev/null +++ b/task-2/README.md @@ -0,0 +1,10 @@ +# Task 2 — dbt on Databricks + +Put your **ported dbt project** here (the whole project), minus: + +- secrets (the token lives in an environment variable, never in a committed file), +- the `target/` build directory and any `logs/`. + +Record your results in `WRITEUP.md` (a template is provided). + +A `.gitignore` for `target/`, `logs/`, and `.env` is recommended inside your dbt project so nothing sensitive or generated is committed. diff --git a/task-2/WRITEUP.md b/task-2/WRITEUP.md new file mode 100644 index 0000000..3db133f --- /dev/null +++ b/task-2/WRITEUP.md @@ -0,0 +1,12 @@ +# Task 2 write-up + +## Build times + +- First `dbt build --select fct_trips` (full build): `___` +- Second `dbt build --select fct_trips` (incremental): `___` + +## Why was the second run faster? + +Two or three sentences, referencing `is_incremental()` and `{{ this }}`: + +`___` diff --git a/task-2/task 2 files b/task-2/task 2 files deleted file mode 100644 index e69de29..0000000