diff --git a/lambda_function/Dockerfile b/lambda_function/Dockerfile index 81342a5..3329398 100755 --- a/lambda_function/Dockerfile +++ b/lambda_function/Dockerfile @@ -36,6 +36,13 @@ COPY src/. ${FUNCTION_DIR} RUN chmod -R 755 ${FUNCTION_DIR} RUN chown -R 1000:1000 ${FUNCTION_DIR} +# Package config locations. Source-of-truth YAML lives in src/config// +# (baked into the image at ${FUNCTION_DIR}config//) and is mirrored to +# the writable /tmp/config// at runtime by entry_script.sh. Each package +# env var below points at the /tmp mirror because some packages (e.g. +# ccsdspy) require their CONFIGDIR to be writable. See src/config/README.md. +ENV ccsdspy_CONFIGDIR=/tmp/config/ccsdspy + # Copy entry script into function director (Script is used distinguish dev/production mode) COPY entry_script.sh ${FUNCTION_DIR} diff --git a/lambda_function/entry_script.sh b/lambda_function/entry_script.sh index 816f0d7..4677a08 100755 --- a/lambda_function/entry_script.sh +++ b/lambda_function/entry_script.sh @@ -1,5 +1,17 @@ #!/bin/sh +# Seed writable runtime config tree from baked package configs. +# Some packages (e.g. ccsdspy) require their *_CONFIGDIR env var to point at a +# writable directory, but the Lambda image filesystem is read-only at runtime +# (only /tmp is writable). We mirror the baked /lambda_function/config/ tree +# into /tmp/config/ here, before the Python runtime starts. AWS Lambda mounts +# a fresh /tmp per execution environment, so this runs on every cold start; +# warm starts re-copy harmlessly (idempotent). See src/config/README.md. +if [ -d "/lambda_function/config" ]; then + mkdir -p /tmp/config + cp -R /lambda_function/config/. /tmp/config/ +fi + if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then exec /usr/local/bin/aws-lambda-rie python3 -m awslambdaric $@ else diff --git a/lambda_function/src/config/README.md b/lambda_function/src/config/README.md new file mode 100644 index 0000000..47032c0 --- /dev/null +++ b/lambda_function/src/config/README.md @@ -0,0 +1,161 @@ +# Baked package configuration + +This directory holds **source-of-truth, build-time configuration files** for +upstream Python packages whose import-time behavior we need to control before +any handler code runs. Each immediate subdirectory corresponds to one package. + +``` +lambda_function/src/config/ +├── README.md <- this file +└── / <- one directory per package needing baked config + └── config.yml <- (or whatever filename the package expects) +``` + +## Three-stage path: repo → image → `/tmp` + +Config files live in this directory in the repo (versioned, reviewed). At +build time, the existing `COPY src/. ${FUNCTION_DIR}` step in +`lambda_function/Dockerfile` ships the tree to `/lambda_function/config/` +inside the image, and the recursive `chmod -R 755` / `chown -R 1000:1000` +steps apply correct permissions. At runtime, `lambda_function/entry_script.sh` +mirrors `/lambda_function/config/` into the writable `/tmp/config/` *before* +the Python runtime starts, and the Dockerfile `ENV` lines point each package +at the `/tmp/config//` mirror. + +``` +repo: lambda_function/src/config//... + └─ COPY src/. ${FUNCTION_DIR} (build time) +image: /lambda_function/config//... (read-only at runtime) + └─ cp -R /lambda_function/config/. /tmp/config/ (cold start, entry_script.sh) +runtime: /tmp/config//... (writable, what the ENV var points at) +``` + +## Why bake config into the image? + +Some packages (e.g. `ccsdspy`) run a config loader and logger init at +**package import time**. If the default config asks for a writable path on a +read-only filesystem, or simply requires `_CONFIGDIR` itself to be +writable, the very first `import ` blows up before our handler can +do anything about it. Pointing the package at a pre-seeded, writable +`/tmp/config//` lets us pre-empt the bad default without patching the +package or shimming the handler. + +## AWS Lambda runtime filesystem constraint + +> **The entire image filesystem is read-only at Lambda runtime. Only `/tmp` +> is writable.** + +Consequences: + +1. **`/lambda_function/config//` cannot be the env-var target** for any + package that requires its config dir to be writable (e.g. `ccsdspy`'s + `_get_user_configdir()` calls `mkdir` + `os.access(W_OK)` *before* it + reads the YAML). That is why `entry_script.sh` mirrors the tree into + `/tmp/config/` on every invocation, and why the `ENV` lines point at the + `/tmp` mirror. +2. **`/tmp` is wiped between execution environments**, so the seed step has + to re-run on every cold start. The `cp -R` is idempotent for warm starts. +3. **Log files, caches, and other runtime state** for a baked-config package + may live alongside its config under `/tmp/config//` — the tree is + writable. They are ephemeral (gone on the next cold start), so for + anything durable prefer Lambda's stdout/stderr → CloudWatch pipeline. + +## Dockerfile wiring + +For each package with a directory here, add **one** `ENV` line to +`lambda_function/Dockerfile` in the post-COPY env block (after +`RUN chown -R 1000:1000 ${FUNCTION_DIR}` and before +`# Copy entry script into function director`): + +```dockerfile +# Package config locations. Source-of-truth YAML lives in src/config// +# (baked into the image at ${FUNCTION_DIR}config//) and is mirrored to +# the writable /tmp/config// at runtime by entry_script.sh. +ENV ccsdspy_CONFIGDIR=/tmp/config/ccsdspy +# ENV _CONFIGDIR=/tmp/config/ <- pattern for future packages +``` + +Notes: + +- The env target is `/tmp/config/`, **not** `${FUNCTION_DIR}config/`, + because some packages require the dir to be writable. The seed step in + `entry_script.sh` is what makes `/tmp/config/` exist. +- Use `ENV`, **not** `ARG`. `ARG` values are build-time only and are not + visible to the Python runtime via `os.environ`. +- The exact variable name (`ccsdspy_CONFIGDIR`, `SOMETHING_CONFIG_DIR`, etc.) + is dictated by the upstream package — match whatever it reads. +- Place the `ENV` lines together as a single block so they are easy to grep. + +## Worked example: `ccsdspy` + +**Problem.** Recent `ccsdspy` releases (a) unconditionally open a +`logging.FileHandler` against a relative `log_file_path` at package import +time, and (b) require `ccsdspy_CONFIGDIR` itself to be writable +(`_get_user_configdir()` does `mkdir(...exist_ok=True)` + `os.access(W_OK)` +before reading `config.yml`). In the Lambda image both fail: CWD +`/lambda_function/` is read-only at runtime, and so is any baked subdir. +`import ccsdspy` — pulled in transitively by `padre_meddea.io.file_tools` — +raises one of: + +``` +OSError: [Errno 30] Read-only file system: '/lambda_function/ccsdspy.log' +RuntimeError: Could not write to ccsdspy_CONFIGDIR="/lambda_function/config/ccsdspy" +``` + +**Fix.** Bake a `config.yml` that disables file logging, seed it into a +writable `/tmp/config/ccsdspy/` at cold start, and point `ccsdspy_CONFIGDIR` +at the writable mirror: + +`lambda_function/src/config/ccsdspy/config.yml` (the relevant key): + +```yaml +logger: + log_to_file: false +``` + +`lambda_function/entry_script.sh` (added near the top): + +```sh +if [ -d "/lambda_function/config" ]; then + mkdir -p /tmp/config + cp -R /lambda_function/config/. /tmp/config/ +fi +``` + +`lambda_function/Dockerfile` (in the post-COPY env block): + +```dockerfile +ENV ccsdspy_CONFIGDIR=/tmp/config/ccsdspy +``` + +`ccsdspy._get_user_configdir()` now sees a writable `/tmp/config/ccsdspy/` +seeded with our YAML, `load_config()` reads it instead of the packaged +default, and `_init_log` never reaches the `FileHandler` branch. The +`StreamHandler` still attaches, so log records continue to flow to CloudWatch +via stdout. + +If you ever need `ccsdspy` to write a real log file, change the YAML to: + +```yaml +logger: + log_to_file: true + log_file_path: /tmp/config/ccsdspy/ccsdspy.log +``` + +No extra `mkdir` is needed — the seed step already created +`/tmp/config/ccsdspy/`. The file is ephemeral (wiped per execution +environment), so for durable logs stick with the stdout/CloudWatch route. + +## Adding a new package + +1. `mkdir lambda_function/src/config//` and drop the config file(s) + the package expects. The existing `COPY src/. ${FUNCTION_DIR}` step ships + it; no Dockerfile COPY/chmod edits needed. +2. If the baked config refers to any writable runtime path (log file, cache), + point it at `/tmp/config//...`. The `entry_script.sh` seed step + already guarantees that directory exists and is writable. +3. Add one `ENV _CONFIGDIR=/tmp/config/` line to the post-COPY env + block in `lambda_function/Dockerfile` (match whatever env var name the + package actually reads). +4. No `entry_script.sh` change is needed — the generic seed step already + mirrors the entire `/lambda_function/config/` tree. diff --git a/lambda_function/src/config/ccsdspy/config.yml b/lambda_function/src/config/ccsdspy/config.yml new file mode 100644 index 0000000..6cadbe1 --- /dev/null +++ b/lambda_function/src/config/ccsdspy/config.yml @@ -0,0 +1,33 @@ +# Configuration +# This is the default configuration file + +general: + # Time Format to be used for displaying time in output (e.g. graphs) + # The default time format is based on ISO8601 (replacing the T with space) + # note that the extra '%'s are escape characters + time_format: "%Y-%m-%d %H:%M:%S" + +logger: + # Threshold for the logging messages. Logging messages that are less severe + # than this level will be ignored. The levels are 'DEBUG', 'INFO', 'WARNING', 'ERROR' + log_level: DEBUG + + log_format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + + # Whether to always log messages to a log file + log_to_file: true + + # Whether the log file should be in JSON format + log_file_json: false + + # The file to log messages to. + # Must point at a writable location at Lambda runtime. /tmp/config/ccsdspy/ + # is seeded by entry_script.sh and is the directory ccsdspy_CONFIGDIR points + # at, so it is guaranteed to exist and be writable. See src/config/README.md. + log_file_path: /tmp/config/ccsdspy/ccsdspy.log + + # Threshold for logging messages to log_file_path + log_file_level: INFO + + # Format for log file entries + log_file_format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" \ No newline at end of file