From 7d00c0aed133c7a8729a1c9e796f7121a23b5c3a Mon Sep 17 00:00:00 2001 From: Andrew Robbertz <24920994+Alrobbertz@users.noreply.github.com> Date: Tue, 30 Jun 2026 10:07:28 -0400 Subject: [PATCH 1/2] Create CCSDSPy Config --- lambda_function/Dockerfile | 5 + lambda_function/src/config/README.md | 130 ++++++++++++++++++ lambda_function/src/config/ccsdspy/config.yml | 30 ++++ 3 files changed, 165 insertions(+) create mode 100644 lambda_function/src/config/README.md create mode 100644 lambda_function/src/config/ccsdspy/config.yml diff --git a/lambda_function/Dockerfile b/lambda_function/Dockerfile index 407d0f6..ff14dfc 100755 --- a/lambda_function/Dockerfile +++ b/lambda_function/Dockerfile @@ -34,6 +34,11 @@ COPY src/. ${FUNCTION_DIR} RUN chmod -R 755 ${FUNCTION_DIR} RUN chown -R 1000:1000 ${FUNCTION_DIR} +# Package config locations (baked under ${FUNCTION_DIR}config//). +# See src/config/README.md for the convention and the /tmp// rule for +# any package that needs writable scratch space at runtime. +ENV ccsdspy_CONFIGDIR=${FUNCTION_DIR}config/ccsdspy + # Copy entry script into function director (Script is used distinguish dev/production mode) COPY entry_script.sh ${FUNCTION_DIR} diff --git a/lambda_function/src/config/README.md b/lambda_function/src/config/README.md new file mode 100644 index 0000000..74a5f5b --- /dev/null +++ b/lambda_function/src/config/README.md @@ -0,0 +1,130 @@ +# Baked package configuration + +This directory holds **read-only, build-time configuration files** that ship +inside the Lambda image. Each immediate subdirectory corresponds to one +upstream Python package whose import-time behavior we need to control before +any handler code runs. + +``` +lambda_function/src/config/ +├── README.md <- this file +└── / <- one directory per package needing baked config + └── config.yml <- (or whatever filename the package expects) +``` + +At build time, the existing `COPY src/. ${FUNCTION_DIR}` step in +`lambda_function/Dockerfile` ships this tree to `/lambda_function/config/` +inside the image, and the recursive `chmod -R 755` / `chown -R 1000:1000` +steps already apply correct permissions. + +## Why bake config into the image? + +Some packages (e.g. `ccsdspy`) run a config loader and logger init at +**package import time**. If their default config asks for a writable path +(log files, cache dirs) on a read-only filesystem, the very first +`import ` blows up before our handler can do anything about it. + +Pointing the package at a baked-in `config.yml` via an environment variable +(see "Dockerfile wiring" below) lets us pre-empt the bad default without +patching the package or shimming the handler. + +## AWS Lambda runtime filesystem constraint + +> **The entire image filesystem is read-only at Lambda runtime. Only `/tmp` +> is writable.** + +That means **nothing under `/lambda_function/config//` may be written +to at runtime**. If a package needs to write log files, caches, or any +other state, the config you bake here **must** point those writes at a +path under `/tmp//`. Two corollaries: + +1. The caller (handler module, entry script, or — preferred — the package + itself, via an upstream fix) is responsible for `mkdir -p /tmp/` + on cold start. `/tmp` is a fresh empty mount on each execution + environment, so the directory will not survive across cold starts. +2. Lambda's `/tmp` is wiped between execution environments, so any log + files written there are ephemeral. If you want durable, queryable logs, + prefer disabling file logging entirely and rely on Lambda's + stdout/stderr → CloudWatch pipeline (which is what we do for `ccsdspy` + below). + +## Dockerfile wiring + +For each package with a directory here, add **one** `ENV` line to +`lambda_function/Dockerfile` in the post-COPY env block (after +`RUN chown -R 1000:1000 ${FUNCTION_DIR}` and before +`# Copy entry script into function director`): + +```dockerfile +# Package config locations (baked under /lambda_function/config//) +ENV ccsdspy_CONFIGDIR=/lambda_function/config/ccsdspy +# ENV _CONFIGDIR=/lambda_function/config/ <- pattern for future packages +``` + +Notes: + +- Use `ENV`, **not** `ARG`. `ARG` values are build-time only and are not + visible to the Python runtime via `os.environ`, which is what these + packages consult at import time. +- The exact variable name (`ccsdspy_CONFIGDIR`, `SOMETHING_CONFIG_DIR`, + etc.) is dictated by the upstream package — match whatever it reads. +- Place the `ENV` lines together as a single block so they are easy to + grep and review. + +## Worked example: `ccsdspy` + +**Problem.** Recent `ccsdspy` releases unconditionally open a +`logging.FileHandler` against a relative `log_file_path` at package +import time. In the Lambda image the CWD is `/lambda_function/` (which +is read-only at runtime), so `import ccsdspy` — pulled in transitively +by `padre_meddea.io.file_tools` — raises: + +``` +OSError: [Errno 30] Read-only file system: '/lambda_function/ccsdspy.log' +``` + +**Fix.** Bake a minimal `config.yml` that disables file logging, and +point `ccsdspy_CONFIGDIR` at it: + +`lambda_function/src/config/ccsdspy/config.yml`: + +```yaml +logger: + log_to_file: false +``` + +`lambda_function/Dockerfile` (in the post-COPY env block): + +```dockerfile +ENV ccsdspy_CONFIGDIR=/lambda_function/config/ccsdspy +``` + +`ccsdspy.config.load_config()` finds our YAML first and skips the +packaged default, so `_init_log` never reaches the `FileHandler` branch. +The `StreamHandler` still attaches, and log records continue to flow to +CloudWatch via stdout. + +If you ever need `ccsdspy` to write a real log file, change the YAML to: + +```yaml +logger: + log_to_file: true + log_file_path: /tmp/ccsdspy/ccsdspy.log +``` + +…and ensure `/tmp/ccsdspy` is created on cold start (in +`entry_script.sh` or at the very top of `lambda.py`, before any +`import ccsdspy`). Do **not** point `log_file_path` at anything under +`/lambda_function/`. + +## Adding a new package + +1. `mkdir lambda_function/src/config//` and drop the config file(s) + the package expects. +2. If the package needs writable scratch space, make sure the baked + config points it at `/tmp//...`, never inside `config/`. +3. Add one `ENV _CONFIGDIR=/lambda_function/config/` line to + the post-COPY env block in `lambda_function/Dockerfile`. +4. If `/tmp//` needs to exist before the package is imported, add a + `mkdir -p /tmp/` to `entry_script.sh` (preferred over runtime + Python so it happens before the Python interpreter starts). diff --git a/lambda_function/src/config/ccsdspy/config.yml b/lambda_function/src/config/ccsdspy/config.yml new file mode 100644 index 0000000..8ad74e0 --- /dev/null +++ b/lambda_function/src/config/ccsdspy/config.yml @@ -0,0 +1,30 @@ +# Configuration +# This is the default configuration file + +general: + # Time Format to be used for displaying time in output (e.g. graphs) + # The default time format is based on ISO8601 (replacing the T with space) + # note that the extra '%'s are escape characters + time_format: "%Y-%m-%d %H:%M:%S" + +logger: + # Threshold for the logging messages. Logging messages that are less severe + # than this level will be ignored. The levels are 'DEBUG', 'INFO', 'WARNING', 'ERROR' + log_level: DEBUG + + log_format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + + # Whether to always log messages to a log file + log_to_file: false + + # Whether the log file should be in JSON format + log_file_json: false + + # The file to log messages to + log_file_path: ccsdspy.log + + # Threshold for logging messages to log_file_path + log_file_level: INFO + + # Format for log file entries + log_file_format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" \ No newline at end of file From 6c152fa559e862720849c4ca20eb4744b13aac26 Mon Sep 17 00:00:00 2001 From: Andrew Robbertz <24920994+Alrobbertz@users.noreply.github.com> Date: Tue, 30 Jun 2026 11:05:27 -0400 Subject: [PATCH 2/2] Move CCSDSPy Config to /tmp --- lambda_function/Dockerfile | 10 +- lambda_function/entry_script.sh | 12 ++ lambda_function/src/config/README.md | 157 +++++++++++------- lambda_function/src/config/ccsdspy/config.yml | 9 +- 4 files changed, 118 insertions(+), 70 deletions(-) diff --git a/lambda_function/Dockerfile b/lambda_function/Dockerfile index ff14dfc..986776a 100755 --- a/lambda_function/Dockerfile +++ b/lambda_function/Dockerfile @@ -34,10 +34,12 @@ COPY src/. ${FUNCTION_DIR} RUN chmod -R 755 ${FUNCTION_DIR} RUN chown -R 1000:1000 ${FUNCTION_DIR} -# Package config locations (baked under ${FUNCTION_DIR}config//). -# See src/config/README.md for the convention and the /tmp// rule for -# any package that needs writable scratch space at runtime. -ENV ccsdspy_CONFIGDIR=${FUNCTION_DIR}config/ccsdspy +# Package config locations. Source-of-truth YAML lives in src/config// +# (baked into the image at ${FUNCTION_DIR}config//) and is mirrored to +# the writable /tmp/config// at runtime by entry_script.sh. Each package +# env var below points at the /tmp mirror because some packages (e.g. +# ccsdspy) require their CONFIGDIR to be writable. See src/config/README.md. +ENV ccsdspy_CONFIGDIR=/tmp/config/ccsdspy # Copy entry script into function director (Script is used distinguish dev/production mode) COPY entry_script.sh ${FUNCTION_DIR} diff --git a/lambda_function/entry_script.sh b/lambda_function/entry_script.sh index 816f0d7..4677a08 100755 --- a/lambda_function/entry_script.sh +++ b/lambda_function/entry_script.sh @@ -1,5 +1,17 @@ #!/bin/sh +# Seed writable runtime config tree from baked package configs. +# Some packages (e.g. ccsdspy) require their *_CONFIGDIR env var to point at a +# writable directory, but the Lambda image filesystem is read-only at runtime +# (only /tmp is writable). We mirror the baked /lambda_function/config/ tree +# into /tmp/config/ here, before the Python runtime starts. AWS Lambda mounts +# a fresh /tmp per execution environment, so this runs on every cold start; +# warm starts re-copy harmlessly (idempotent). See src/config/README.md. +if [ -d "/lambda_function/config" ]; then + mkdir -p /tmp/config + cp -R /lambda_function/config/. /tmp/config/ +fi + if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then exec /usr/local/bin/aws-lambda-rie python3 -m awslambdaric $@ else diff --git a/lambda_function/src/config/README.md b/lambda_function/src/config/README.md index 74a5f5b..47032c0 100644 --- a/lambda_function/src/config/README.md +++ b/lambda_function/src/config/README.md @@ -1,9 +1,8 @@ # Baked package configuration -This directory holds **read-only, build-time configuration files** that ship -inside the Lambda image. Each immediate subdirectory corresponds to one -upstream Python package whose import-time behavior we need to control before -any handler code runs. +This directory holds **source-of-truth, build-time configuration files** for +upstream Python packages whose import-time behavior we need to control before +any handler code runs. Each immediate subdirectory corresponds to one package. ``` lambda_function/src/config/ @@ -12,41 +11,54 @@ lambda_function/src/config/ └── config.yml <- (or whatever filename the package expects) ``` -At build time, the existing `COPY src/. ${FUNCTION_DIR}` step in -`lambda_function/Dockerfile` ships this tree to `/lambda_function/config/` +## Three-stage path: repo → image → `/tmp` + +Config files live in this directory in the repo (versioned, reviewed). At +build time, the existing `COPY src/. ${FUNCTION_DIR}` step in +`lambda_function/Dockerfile` ships the tree to `/lambda_function/config/` inside the image, and the recursive `chmod -R 755` / `chown -R 1000:1000` -steps already apply correct permissions. +steps apply correct permissions. At runtime, `lambda_function/entry_script.sh` +mirrors `/lambda_function/config/` into the writable `/tmp/config/` *before* +the Python runtime starts, and the Dockerfile `ENV` lines point each package +at the `/tmp/config//` mirror. + +``` +repo: lambda_function/src/config//... + └─ COPY src/. ${FUNCTION_DIR} (build time) +image: /lambda_function/config//... (read-only at runtime) + └─ cp -R /lambda_function/config/. /tmp/config/ (cold start, entry_script.sh) +runtime: /tmp/config//... (writable, what the ENV var points at) +``` ## Why bake config into the image? Some packages (e.g. `ccsdspy`) run a config loader and logger init at -**package import time**. If their default config asks for a writable path -(log files, cache dirs) on a read-only filesystem, the very first -`import ` blows up before our handler can do anything about it. - -Pointing the package at a baked-in `config.yml` via an environment variable -(see "Dockerfile wiring" below) lets us pre-empt the bad default without -patching the package or shimming the handler. +**package import time**. If the default config asks for a writable path on a +read-only filesystem, or simply requires `_CONFIGDIR` itself to be +writable, the very first `import ` blows up before our handler can +do anything about it. Pointing the package at a pre-seeded, writable +`/tmp/config//` lets us pre-empt the bad default without patching the +package or shimming the handler. ## AWS Lambda runtime filesystem constraint > **The entire image filesystem is read-only at Lambda runtime. Only `/tmp` > is writable.** -That means **nothing under `/lambda_function/config//` may be written -to at runtime**. If a package needs to write log files, caches, or any -other state, the config you bake here **must** point those writes at a -path under `/tmp//`. Two corollaries: - -1. The caller (handler module, entry script, or — preferred — the package - itself, via an upstream fix) is responsible for `mkdir -p /tmp/` - on cold start. `/tmp` is a fresh empty mount on each execution - environment, so the directory will not survive across cold starts. -2. Lambda's `/tmp` is wiped between execution environments, so any log - files written there are ephemeral. If you want durable, queryable logs, - prefer disabling file logging entirely and rely on Lambda's - stdout/stderr → CloudWatch pipeline (which is what we do for `ccsdspy` - below). +Consequences: + +1. **`/lambda_function/config//` cannot be the env-var target** for any + package that requires its config dir to be writable (e.g. `ccsdspy`'s + `_get_user_configdir()` calls `mkdir` + `os.access(W_OK)` *before* it + reads the YAML). That is why `entry_script.sh` mirrors the tree into + `/tmp/config/` on every invocation, and why the `ENV` lines point at the + `/tmp` mirror. +2. **`/tmp` is wiped between execution environments**, so the seed step has + to re-run on every cold start. The `cp -R` is idempotent for warm starts. +3. **Log files, caches, and other runtime state** for a baked-config package + may live alongside its config under `/tmp/config//` — the tree is + writable. They are ephemeral (gone on the next cold start), so for + anything durable prefer Lambda's stdout/stderr → CloudWatch pipeline. ## Dockerfile wiring @@ -56,75 +68,94 @@ For each package with a directory here, add **one** `ENV` line to `# Copy entry script into function director`): ```dockerfile -# Package config locations (baked under /lambda_function/config//) -ENV ccsdspy_CONFIGDIR=/lambda_function/config/ccsdspy -# ENV _CONFIGDIR=/lambda_function/config/ <- pattern for future packages +# Package config locations. Source-of-truth YAML lives in src/config// +# (baked into the image at ${FUNCTION_DIR}config//) and is mirrored to +# the writable /tmp/config// at runtime by entry_script.sh. +ENV ccsdspy_CONFIGDIR=/tmp/config/ccsdspy +# ENV _CONFIGDIR=/tmp/config/ <- pattern for future packages ``` Notes: +- The env target is `/tmp/config/`, **not** `${FUNCTION_DIR}config/`, + because some packages require the dir to be writable. The seed step in + `entry_script.sh` is what makes `/tmp/config/` exist. - Use `ENV`, **not** `ARG`. `ARG` values are build-time only and are not - visible to the Python runtime via `os.environ`, which is what these - packages consult at import time. -- The exact variable name (`ccsdspy_CONFIGDIR`, `SOMETHING_CONFIG_DIR`, - etc.) is dictated by the upstream package — match whatever it reads. -- Place the `ENV` lines together as a single block so they are easy to - grep and review. + visible to the Python runtime via `os.environ`. +- The exact variable name (`ccsdspy_CONFIGDIR`, `SOMETHING_CONFIG_DIR`, etc.) + is dictated by the upstream package — match whatever it reads. +- Place the `ENV` lines together as a single block so they are easy to grep. ## Worked example: `ccsdspy` -**Problem.** Recent `ccsdspy` releases unconditionally open a -`logging.FileHandler` against a relative `log_file_path` at package -import time. In the Lambda image the CWD is `/lambda_function/` (which -is read-only at runtime), so `import ccsdspy` — pulled in transitively -by `padre_meddea.io.file_tools` — raises: +**Problem.** Recent `ccsdspy` releases (a) unconditionally open a +`logging.FileHandler` against a relative `log_file_path` at package import +time, and (b) require `ccsdspy_CONFIGDIR` itself to be writable +(`_get_user_configdir()` does `mkdir(...exist_ok=True)` + `os.access(W_OK)` +before reading `config.yml`). In the Lambda image both fail: CWD +`/lambda_function/` is read-only at runtime, and so is any baked subdir. +`import ccsdspy` — pulled in transitively by `padre_meddea.io.file_tools` — +raises one of: ``` OSError: [Errno 30] Read-only file system: '/lambda_function/ccsdspy.log' +RuntimeError: Could not write to ccsdspy_CONFIGDIR="/lambda_function/config/ccsdspy" ``` -**Fix.** Bake a minimal `config.yml` that disables file logging, and -point `ccsdspy_CONFIGDIR` at it: +**Fix.** Bake a `config.yml` that disables file logging, seed it into a +writable `/tmp/config/ccsdspy/` at cold start, and point `ccsdspy_CONFIGDIR` +at the writable mirror: -`lambda_function/src/config/ccsdspy/config.yml`: +`lambda_function/src/config/ccsdspy/config.yml` (the relevant key): ```yaml logger: log_to_file: false ``` +`lambda_function/entry_script.sh` (added near the top): + +```sh +if [ -d "/lambda_function/config" ]; then + mkdir -p /tmp/config + cp -R /lambda_function/config/. /tmp/config/ +fi +``` + `lambda_function/Dockerfile` (in the post-COPY env block): ```dockerfile -ENV ccsdspy_CONFIGDIR=/lambda_function/config/ccsdspy +ENV ccsdspy_CONFIGDIR=/tmp/config/ccsdspy ``` -`ccsdspy.config.load_config()` finds our YAML first and skips the -packaged default, so `_init_log` never reaches the `FileHandler` branch. -The `StreamHandler` still attaches, and log records continue to flow to -CloudWatch via stdout. +`ccsdspy._get_user_configdir()` now sees a writable `/tmp/config/ccsdspy/` +seeded with our YAML, `load_config()` reads it instead of the packaged +default, and `_init_log` never reaches the `FileHandler` branch. The +`StreamHandler` still attaches, so log records continue to flow to CloudWatch +via stdout. If you ever need `ccsdspy` to write a real log file, change the YAML to: ```yaml logger: log_to_file: true - log_file_path: /tmp/ccsdspy/ccsdspy.log + log_file_path: /tmp/config/ccsdspy/ccsdspy.log ``` -…and ensure `/tmp/ccsdspy` is created on cold start (in -`entry_script.sh` or at the very top of `lambda.py`, before any -`import ccsdspy`). Do **not** point `log_file_path` at anything under -`/lambda_function/`. +No extra `mkdir` is needed — the seed step already created +`/tmp/config/ccsdspy/`. The file is ephemeral (wiped per execution +environment), so for durable logs stick with the stdout/CloudWatch route. ## Adding a new package 1. `mkdir lambda_function/src/config//` and drop the config file(s) - the package expects. -2. If the package needs writable scratch space, make sure the baked - config points it at `/tmp//...`, never inside `config/`. -3. Add one `ENV _CONFIGDIR=/lambda_function/config/` line to - the post-COPY env block in `lambda_function/Dockerfile`. -4. If `/tmp//` needs to exist before the package is imported, add a - `mkdir -p /tmp/` to `entry_script.sh` (preferred over runtime - Python so it happens before the Python interpreter starts). + the package expects. The existing `COPY src/. ${FUNCTION_DIR}` step ships + it; no Dockerfile COPY/chmod edits needed. +2. If the baked config refers to any writable runtime path (log file, cache), + point it at `/tmp/config//...`. The `entry_script.sh` seed step + already guarantees that directory exists and is writable. +3. Add one `ENV _CONFIGDIR=/tmp/config/` line to the post-COPY env + block in `lambda_function/Dockerfile` (match whatever env var name the + package actually reads). +4. No `entry_script.sh` change is needed — the generic seed step already + mirrors the entire `/lambda_function/config/` tree. diff --git a/lambda_function/src/config/ccsdspy/config.yml b/lambda_function/src/config/ccsdspy/config.yml index 8ad74e0..6cadbe1 100644 --- a/lambda_function/src/config/ccsdspy/config.yml +++ b/lambda_function/src/config/ccsdspy/config.yml @@ -15,13 +15,16 @@ logger: log_format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" # Whether to always log messages to a log file - log_to_file: false + log_to_file: true # Whether the log file should be in JSON format log_file_json: false - # The file to log messages to - log_file_path: ccsdspy.log + # The file to log messages to. + # Must point at a writable location at Lambda runtime. /tmp/config/ccsdspy/ + # is seeded by entry_script.sh and is the directory ccsdspy_CONFIGDIR points + # at, so it is guaranteed to exist and be writable. See src/config/README.md. + log_file_path: /tmp/config/ccsdspy/ccsdspy.log # Threshold for logging messages to log_file_path log_file_level: INFO