update

Author: nocomplexity

guidelines/securityprinciples.md

@@ -7,7 +7,7 @@ short_title: Security Principles Checklist
 Applying Security by Design principles is essential when building secure Python applications.
 :::
 
-
+(security-principles)=
 ## Security Principles
 
 When developing Python applications, you should apply at **minimum** the following Security by Design principles:

recipes/recipes_intro.md

@@ -0,0 +1,23 @@
+---
+title: Python Security Recipes
+short_title: Security Recipes
+---
+
+
+When developing secure applications in Python, clear and practical examples are indispensable. This section provides actionable, straightforward solutions to common Python security challenges.
+
+Each recipe follows a consistent, easy-to-digest blueprint:
+
+* **Challenge:** The development goal or requirement.
+* **Threat:** The risk or attack vector associated with incorrect implementation.
+* **Vulnerable Example:** Real-world code demonstrating the flaw.
+* **Secure Mitigation:** Corrected, production-ready code implementing defensive practices.
+* **Discussion:** An architectural breakdown of why the mitigation works and further security nuances.
+
++++ {"no-pdf": true}
+
+:::{toc}
+:context: children
+:::
+
++++

recipes/use_secrets.md

@@ -0,0 +1,261 @@
+---
+title: How To Use Secrets in Python Code
+short_title: How To Use Secrets
+---
+
+## The Challenge
+
+Every non-trivial Python application relies on secrets: API keys for third-party services, database passwords, JWT signing keys, cloud provider credentials, and encryption passphrases. The challenge is both operational and cryptographic: you must deliver these values to your application at runtime while preventing them from leaking into logs, version control, error reports, or attacker-controlled memory dumps.
+
+In practice, this means solving three distinct problems:
+
+1. **Storage**: Where do you keep secrets when the application is not running?
+2. **Transport**: How do you securely inject secrets into the running process?
+3. **Lifecycle**: How do you rotate, revoke, and audit secret usage without downtime?
+
+Many teams treat these as afterthoughts—hard-coding credentials during development and "fixing" them later. But always use a [Security By Design](https://nocomplexity.com/securitybydesign/) approach. Fixing security later is not possible. Start with security from the start!
+
+---
+
+## The Threat
+
+To understand why secret management matters, think like an attacker. Secrets can be stolen or get lost in many different ways, each with different mitigation strategies:
+
+**Exfiltration from Version Control**
+Hard-coded secrets in source code are the most common vulnerability. Attackers scan public repositories for patterns matching API keys, AWS access keys, and private keys. Even private repositories are not safe—misconfigured CI pipelines, compromised developer accounts, and insider threats all expose stored credentials.
+
+**Leakage via Logging and Debug Output**
+Python's logging framework, exception tracebacks, and debuggers frequently output variable contents. If your code logs a request payload, a configuration dump, or an error context that includes a secret, that secret ends up in log files, monitoring systems, and potentially aggregated into centralized logging platforms with weaker access controls.
+
+**Environment Variable Exposure**
+Environment variables are the most common delivery mechanism, but they are also visible to:
+- Any process running under the same user (`/proc/PID/environ` on Linux)
+- Child processes that inherit the environment
+- Debuggers and crash dumps
+- Container orchestration UIs that display environment variables by default
+
+
+**In-Memory Exposure**
+Python's object model makes it difficult to securely scrub secrets from memory. Strings are immutable—when you overwrite a secret string, the old value remains in memory until the garbage collector reclaims it. Additionally, Python's memory allocator may not return freed memory to the operating system, leaving secret fragments in process memory that could be captured via core dumps, swap files, or cold-boot attacks.
+
+**Side-Channel Exposure**
+Secrets can leak through timing attacks (comparing secret values byte-by-byte), through error messages that reveal whether a secret matched partially, or through monitoring that records the frequency of secret usage.
+
+**Supply Chain Risk**
+Third-party dependencies can inadvertently expose secrets through telemetry, crash reporting, or debug logs. More maliciously, a compromised dependency could exfiltrate environment variables at runtime.
+
+**The Principle**: If a secret touches disk, stdout, stderr, or a log file, consider it compromised. If you cannot guarantee that a secret is ephemeral and scoped, assume it will eventually be disclosed.
+
+---
+
+## Vulnerable Code Example
+
+This simple script demonstrates a background job querying a database. It contains severe anti-patterns: hard-coding secrets directly into the source code, performing non-constant-time string comparisons, and leaking system contexts into error outputs.
+
+```python
+import sqlite3
+
+# VULNERABILITY 1: Hard-coded production credentials in source control
+DB_PASSWORD = "super_secret_production_password_123"
+ADMIN_KEY = "admin123"
+
+def authenticate_and_query(user_provided_key):
+    # VULNERABILITY 3: Non-constant-time string comparison (==)
+    # Python stops comparing at the first mismatched byte, introducing timing side-channels.
+    if user_provided_key == ADMIN_KEY:
+        try:
+            # VULNERABILITY 2: Hard-coded credentials passed directly into the connection
+            conn = sqlite3.connect(f"file:prod_db?password={DB_PASSWORD}", uri=True)
+            cursor = conn.cursor()
+            cursor.execute("SELECT * FROM sensitive_records")
+            records = cursor.fetchall()
+            conn.close()
+            return records
+        except Exception as e:
+            # VULNERABILITY 4: Leaking raw exception details and variable state to stdout
+            print(f"Failed to connect using password {DB_PASSWORD}. Error: {e}")
+            return None
+    else:
+        print("Access denied.")
+        return None
+
+```
+
+---
+
+## Secure Mitigation
+
+This secure implementation mitigates these risks by pulling configuration dynamically via a single, session-persistent Secret Manager client, using constant-time comparisons, and executing low-level memory zeroing before clean exit routines.
+
+```python
+import ctypes
+import hmac
+import logging
+import os
+import sqlite3
+import hvac
+
+# Configure safe structured logging
+logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
+logger = logging.getLogger(__name__)
+
+
+class LocalVaultManager:
+    """
+    Encapsulates persistent local HashiCorp Vault interactions.
+    Reuses sessions rather than authenticating per-request.
+    """
+    def __init__(self):
+        self._client = None
+
+    def _get_client(self) -> hvac.Client:
+        if self._client and self._client.is_authenticated():
+            return self._client
+
+        url = os.environ.get("VAULT_ADDR")
+        role_id = os.environ.get("VAULT_ROLE_ID")
+        secret_id = os.environ.get("VAULT_SECRET_ID")
+
+        if not all([url, role_id, secret_id]):
+            raise RuntimeError("CRITICAL: Vault orchestration credentials missing from process memory.")
+
+        client = hvac.Client(url=url)
+        client.auth.approle.login(role_id=role_id, secret_id=secret_id)
+        self._client = client
+        return self._client
+
+    def fetch_secrets(self, path: str) -> dict:
+        try:
+            client = self._get_client()
+            response = client.secrets.kv.v2.read_secret_version(path=path)
+            return response["data"]["data"]
+        except Exception as e:
+            logger.error(f"Failed to fetch secrets securely: {type(e).__name__}")
+            return {}
+
+
+def zero_string_buffer(s: str):
+    """
+    Overwrites the underlying memory allocation buffer of a Python string 
+    with null bytes to remove traces before garbage collection triggers.
+    """
+    if not isinstance(s, str) or not s:
+        return
+    # Trace specific memory offset for standard CPython string configurations
+    offset = id(s) + ctypes.sizeof(ctypes.c_void_p) * 4
+    ctypes.memset(offset, 0, len(s))
+
+
+# Global vault manager allocation handles authentication persistence
+vault = LocalVaultManager()
+
+
+def authenticate_and_query(user_provided_key: str):
+    # Retrieve scoped secrets dictionary in a single network transaction
+    secrets_package = vault.fetch_secrets("secret/data/prod/app")
+    
+    expected_key = secrets_package.get("admin_key")
+    db_password = secrets_package.get("password")
+
+    if not expected_key or not db_password:
+        logger.error("Configuration payload generation failed.")
+        return None
+
+    # MITIGATION 3: Constant-time comparison eliminates timing side-channels
+    if not hmac.compare_digest(user_provided_key, expected_key):
+        logger.warning("Unauthorized access attempt.")
+        zero_string_buffer(expected_key)
+        zero_string_buffer(db_password)
+        return None
+
+    try:
+        # MITIGATION 2 & 5: Ephemeral connection context scoped tightly
+        with sqlite3.connect(f"file:prod_db?password={db_password}", uri=True) as conn:
+            cursor = conn.cursor()
+            cursor.execute("SELECT * FROM sensitive_records")
+            return cursor.fetchall()
+            
+    except sqlite3.Error as db_err:
+        # MITIGATION 4: Generic logs mask underlying parameters and call-stack variables
+        logger.error(f"Database operation failed: {type(db_err).__name__}")
+        return None
+    finally:
+        # MITIGATION 5: Hard zeroing of memory locations before discarding namespaces
+        zero_string_buffer(expected_key)
+        zero_string_buffer(db_password)
+        del expected_key
+        del db_password
+
+```
+
+---
+
+## Discussion
+
+The secure mitigation presented above demonstrates how fundamental security principles translate into practical Python code. Managing and using secrets in Python applications requires applying these principles rigorously—they are not theoretical abstractions but actionable guidelines that directly inform every design decision. Below, we examine how each principle maps to our implementation, alongside the trade-offs and residual risks that remain.
+
+### Operational Considerations (Design for Secure Updates)
+
+> **Systems must safely apply patches. Update ability is a security feature.**
+
+**Secret Rotation**: The secure version pulls secrets programmatically at runtime from local storage managers. Because keys are pulled inside local variable contexts, standard key updates inside Vault do not require code changes or system rollouts.
+
+**Incident Response**: When a secret is compromised, you need to:
+
+1. Immediately revoke the token leases inside the Vault controller registry.
+2. Rotate underlying application and target master credentials.
+3. Check access metrics over Vault audit pipelines.
+
+---
+
+### Principles in Practice when managing and using secrets
+
+| Principle | Summary | How We Implemented It |
+| --- | --- | --- |
+| **Minimise attack surface area** | Remove unnecessary features | Handled configuration data using minimal scripts, removing exposed parameters. |
+| **Establish secure defaults** | Deny by default | Verification checks drop immediately back to a closed state (`return None`). |
+| **Least privilege** | Minimum permissions | Vault tokens bound uniquely to the specific application path read scope. |
+| **Separation of duties** | Split critical functions | Application reads credential metadata, but lacks rights to manage infrastructure targets. |
+| **Defence in depth** | Layer independent controls | Single-path fetch protocols combined with constant-time comparison layers. |
+| **Fail securely** | Never fail open | Captured processing failures cleanly, preventing raw execution traces from outputting. |
+| **Complete mediation** | Every access checked | Constant-time comparisons check input authenticity on every operational iteration. |
+| **Economy of mechanism** | Keep it simple | Session caching avoids complex internal management systems or background handlers. |
+| **Open design** | No security by obscurity | Design safely shifts assumptions; security lives in the token, not hidden code branches. |
+| **Zero Trust** | Verify everything | Application strictly fetches current runtime values rather than caching ambient fields. |
+| **Compartmentalisation** | Isolate components | Function instances manage connection scope properties safely apart from peripheral execution loops. |
+| **Protect data everywhere** | Encrypt everywhere | Realized through zeroing structures and native local transport loop parameters. |
+| **Design for secure updates** | Safe patching | Runtime polling infrastructure accepts dynamic rotations seamlessly. |
+
+---
+
+:::{caution}
+Secrets are not source code. They are operational data that should be treated with the same care as encryption keys. Your application must be designed so that secrets can be changed without redeploying, and the system must gracefully handle secret rotation without downtime.
+:::
+
+## Dangerous Solutions (Anti Patterns)
+
+:::{danger} Dangerous Anti-Pattern: `.env` Files and `python-dotenv`
+A pervasive anti-pattern in Python tutorials is the recommendation to use `python-dotenv` for managing secrets. This advice is dangerously misguided.
+:::
+
+### 1. Persistent Disk Exposure
+
+`.env` files are plain-text files stored on disk. They are vulnerable to:
+
+* Directory traversal attacks exposing the file
+* Accidental commits to version control
+* Backup and snapshot exposure
+
+### 2. Inadequate Security Controls
+
+Security audits of `python-dotenv` reveal it fails on crucial [security validations](#security-principles):
+
+* **Lack of Access Auditing:** Plain-text files can be copied or read by any rogue dependency without logging access events.
+* **Cleartext Lifetime:** Storing passwords in cleartext on standard storage blocks violates core regulatory standards (e.g., SOC2, PCI-DSS).
+* **No Dynamic Rotation Support:** File states cannot accept rapid programmatic cryptographic changes without introducing operational instability.
+
+### 3. The ".env Environment" Fallacy
+
+`python-dotenv` loads secrets from disk into `os.environ`, creating a false sense of security. True environment variables are set by the operating system or orchestration platform and never touch disk.
+
+So **do not** use `python-dotenv` or `.env` files for managing secrets in Python applications. They are a dangerous anti-pattern that introduces unnecessary risk without providing any security benefit. Use environment variables, secret managers, or orchestration-native solutions instead.

toc.yml

@@ -68,13 +68,14 @@ project:
         - file: guidelines/validatetips.md
         - file: guidelines/securityprinciples.md
 
-    - title: Python Security Recipes
-      children:
+    - file: recipes/recipes_intro.md 
+      children:  # drop down    
         - file: recipes/openfile.md
         - file: recipes/traversalattacks.md
         - file: recipes/path_module.md
         - file: recipes/assert_use.md
         - file: recipes/data_egress.md
+        - file: recipes/use_secrets.md
 
     - title: Appendices #section header, no page
       children: