General Python Project - Coding Guidelines

1. Introduction

This document provides a set of general coding standards and best practices for Python projects. The goal is to maintain code quality, consistency, and maintainability across all services and applications.

Project-Specific Documentation

This document is for general Python standards. All project-specific information, such as:

Detailed architecture (e.g., specific microservices, data flow)

Domain logic and business rules

Environment setup (e.g., specific .env keys)

Deployment procedures

...must be documented in a separate file, such as docs/README.md or docs/architecture.md.

3. General Quick Guidelines

Project Structure: Follow the standard src/ layout. (See Project Structure).
Dependencies: Define abstract dependencies in pyproject.toml. Use pip-compile to generate a requirements.txt lockfile. (See Dependency Management).
Virtual Environments: All development must be done within a Python virtual environment (venv).
Configuration: Use pydantic-settings to load configuration from environment variables. Do not read os.environ directly in application code.
Logging: Use Python's built-in logging module. Get loggers with logging.getLogger(__name__).
Data Models: Use pydantic models for all data structures, API payloads, and message formats. This provides runtime validation.
Testing: Use pytest for all tests. Use pytest-mock for mocking and pytest.mark.parametrize for table-driven tests.
Formatting: Use ruff format for code formatting and ruff for linting and import sorting. This is non-negotiable and enforced by CI.
Pre-commit Hooks: Install pre-commit hooks with pre-commit install to automatically format and lint code before each commit.
Type Hinting: All new code must include type hints for function arguments and return values.
Type Checking: All code must pass static type checking with mypy. This is enforced by CI.

4. Project Structure

A standard src-based layout is preferred as it prevents common import issues. The tests directory should mirror the src directory's structure.

project-name/ ├── .github/ # CI/CD workflows (GitHub Actions) ├── .gitignore ├── .pre-commit-config.yaml # pre-commit hooks (see CI/CD) ├── README.md # Project overview, setup, and usage ├── LICENSE ├── pyproject.toml # Project metadata, dependencies, and tool config ├── requirements.in # Abstract dependencies for pip-compile ├── requirements.txt # Pinned lockfile generated by pip-compile ├── docs/ # Project-specific documentation │ └── architecture.md ├── src/ │ └── project_name/ │ ├── init.py │ ├── main.py # Main application entrypoint │ ├── core/ # Core business logic, models, services │ ├── adapters/ # External clients (DB, APIs, etc.) │ ├── api/ # Web-facing layer (FastAPI/Flask) │ ├── config.py # Pydantic settings module │ └── exceptions.py # Custom application exceptions └── tests/ # Tests mirror the src/ layout ├── init.py ├── conftest.py # Pytest fixtures ├── test_main.py └── core/ └── test_core_logic.py

5. Code Organization

Imports: All imports should be absolute (e.g., from project_name.core import models) for clarity. Relative imports (e.g., from . import utils) are only allowed for modules within the same parent directory. ruff will enforce sorting.
Separation of Concerns: We must maintain a clear separation between layers.
- core: Contains the pure, domain-agnostic business logic and data models. Should not import from api or adapters.
- adapters: Contains the "outside world" code: database connections, external API clients, message queue producers.
- api / main: The entrypoint (e.g., FastAPI app) that wires everything together. It handles web requests, calls core logic, and uses adapters to fetch/store data.

6. Dependency Management

Source of Truth: The [project.dependencies] section of pyproject.toml is the high-level source of truth for abstract dependencies (e.g., pydantic>=2.0). For pip-compile, these can be listed in requirements.in.
Reproducible Builds: We use pip-tools to create pinned lockfiles.
- pip-compile: Generates requirements.txt from requirements.in (or pyproject.toml).
- requirements.txt: This generated file is committed to the repository. It contains all direct and transitive dependencies pinned to exact versions, ensuring CI and production builds are reproducible.
Installation:
1. Create a virtual environment: python -m venv .venv
2. Source it: source .venv/bin/activate
3. Install dependencies: pip install -r requirements.txt
4. Install development dependencies: pip install -r requirements-dev.txt (if one exists)
Updating Dependencies:
1. Edit requirements.in (or pyproject.toml).
2. Run pip-compile to regenerate requirements.txt.
3. Run pip install -r requirements.txt.
4. Commit the updated requirements.txt file.

7. Configuration Management

Source: All configuration must be loaded from environment variables.
Tool: Use pydantic-settings to define a Settings class (e.g., in config.py) that loads and validates environment variables.
No Hardcoding: No secrets (API keys, passwords, tokens) are ever to be hardcoded in the source, committed to Git, or stored in default .env files.
Production vs. Development:
- Development: It is acceptable to use a .env file (listed in .gitignore) to load variables for local testing.
- Production: All environment variables must be injected by the CI/CD system or the runtime environment (e.g., Kubernetes Secrets, Docker environment variables).

8. Error Handling & Exceptions

Custom Exceptions: A project-specific exceptions.py module must be created.
Base Exception: This module must define a base exception for the application (e.g., ProjectNameException(Exception)).
Specific Exceptions: All custom, domain-specific errors (e.g., ValidationError, UserNotFoundError) must inherit from this base exception.
raise Rule: Never raise Exception("...") directly. Always raise a specific, custom exception.
except Rule: Never catch the base Exception class, except at the highest level (e.img., an API middleware) to log the error and return a generic 500 response. Always catch the most specific exception possible.

9. Logging Standards

Structured Logging: All services intended for production must log in a structured JSON format. This is non-negotiable for observability in tools like Datadog, Splunk, or OpenTelemetry.
Log Context: Do not log simple strings. Log key events and provide context.
- Good: logging.info("User processed order", extra={"user_id": 123, "order_id": "xyz"})
- Bad: logging.info("Processing order...")
Log Levels: We will adhere to standard log levels.
- ERROR: A critical, unrecoverable error that requires human intervention.
- WARNING: A recoverable error, unexpected situation, or potential problem (e.g., API retry, missing optional config, deprecation).
- INFO: Key application lifecycle events (e.g., "Service started," "Processed message," "User created").
- DEBUG: Verbose information for debugging only (e.g., "Payload received," "Database query result"). This level must be disabled in production.

10. Testing Guidelines

Framework: All tests must be written using pytest.
Structure: The tests/ directory must mirror the src/project_name/ directory structure.
Fixtures: Common setup and teardown logic (e.g., database connections, API test clients) must be implemented as pytest.fixture functions, preferably in conftest.py files.
Mocking:
- Use pytest-mock (the mocker fixture) for all mocking.
- Rule: Only mock at the boundaries of your system. This includes:
  - External API calls (e.g., requests.post)
  - Database driver calls
  - System functions (e.g., datetime.now(), time.sleep())
- Anti-Pattern: Do not mock internal functions or classes within our own application. If you feel the need to, it is a sign that the code should be refactored for better testability (e.g., using dependency injection).
Coverage: We will enforce a minimum test coverage percentage (e.g., 80%) in the CI pipeline using pytest-cov. Pull requests that drop coverage below this threshold will be blocked.

11. Documentation

Docstrings: All new def (functions, methods) and class definitions must have docstrings.
Format: We will standardize on the Google docstring format. It is human-readable and easily parsed by tools like Sphinx. ruff can be configured to lint for this.
README.md Template: Every project's README.md must contain, at a minimum:
1. A short description of what the project does.
2. Development Setup: How to create the virtual environment and install dependencies.
3. Running the Application: The exact command(s) to run the service.
4. Running Tests: The exact command(s) to run the test suite (e.g., pytest).

12. CI/CD

The Continuous Integration (CI) pipeline (e.g., GitHub Actions) is the guardian of our code quality. Every pull request must pass all of the following checks before it can be merged.

Local Validation

Always use run.bat (Windows) or run.sh (Linux/macOS) to validate code changes locally. This script automatically:

Auto-fixes code formatting with ruff format
Auto-fixes lint issues with ruff check --fix
Runs all tests with pytest
Runs type checking with mypy

# Windows (use cmd /c to avoid IDE console issues)
cmd /c run.bat

# Linux/macOS
./run.sh

# Skip checks and run app directly
cmd /c run.bat --no-check

Do not run individual linting/testing commands directly. The run.bat/run.sh scripts ensure consistent validation across all environments.

CI Pipeline Checks

Lint / Format:
- ruff check . (linting)
- ruff format --check . (replaces black)
Type Check:
- mypy src/
Security Audit:
- bandit -r src/ (static analysis for common vulnerabilities)
- pip-audit (audits installed dependencies for known vulnerabilities)
Test & Coverage:
- pytest --cov=src/project_name --cov-fail-under=80

pre-commit: It is strongly recommended that all developers install and use pre-commit. This runs the checks (like ruff, black, mypy) locally before a commit is made, saving time and CI-related frustration. A standard .pre-commit-config.yaml will be provided in the Appendix.

13. Security

Secrets: Reiterate that no secrets (API keys, passwords) are ever stored in code, .env files, or pyproject.toml. They must be loaded from environment variables injected by the CI/CD or runtime environment.
Static Analysis: bandit must be run as part of the CI pipeline to catch common security issues.
Dependency Scanning: pip-audit must be run as part of the CI pipeline. In addition, all repositories must enable GitHub's Dependabot alerts to be notified of new vulnerabilities in our dependencies.

14. Appendix: Tool Configuration

`pyproject.toml` Configuration

All tool configuration should live in pyproject.toml where possible.

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "project_name"
version = "0.1.0"
dependencies = [
    "pydantic>=2.0",
    "pydantic-settings>=2.0",
    "fastapi>=0.100.0",
    # ... other dependencies (or use requirements.in)
]

[project.optional-dependencies]
dev = [
    "pytest",
    "pytest-cov",
    "pytest-mock",
    "mypy",
    "ruff",
    "black", # Kept for compatibility, but ruff format is preferred
    "bandit",
    "pip-audit",
    "pip-tools",
]

# --- Tool Configurations ---

[tool.ruff]
line-length = 88
# Use ruff as a drop-in for isort
[tool.ruff.isort]
force-single-line = true
known-first-party = ["project_name"]

[tool.ruff.lint]
# See all rules here: [https://docs.astral.sh/ruff/rules/](https://docs.astral.sh/ruff/rules/)
select = [
    "E",  # pycodestyle errors
    "W",  # pycodestyle warnings
    "F",  # pyflakes
    "I",  # isort
    "C",  # flake8-comprehensions
    "B",  # flake8-bugbear
]
ignore = []

[tool.mypy]
python_version = "3.11"
warn_return_any = true
warn_unused_ignores = true
strict_optional = true
check_untyped_defs = true
ignore_missing_imports = true # Can be set to false for full strictness

[tool.pytest.ini_options]
minversion = "7.0"
addopts = "-ra -q --cov=src/project_name --cov-report=term-missing"
testpaths = [
    "tests",
]

[tool.coverage.run]
omit = [
    "src/project_name/main.py", # Often hard to test startup
    "src/project_name/config.py", # Just declarations
]

.pre-commit-config.yaml

A standard pre-commit configuration to enforce checks locally.
YAML

repos:
-   repo: [https://github.com/pre-commit/pre-commit-hooks](https://github.com/pre-commit/pre-commit-hooks)
    rev: v4.5.0
    hooks:
    -   id: check-yaml
    -   id: end-of-file-fixer
    -   id: trailing-whitespace
    -   id: check-toml

-   repo: [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit)
    rev: v0.8.4
    hooks:
    # Run linter first with auto-fix
    -   id: ruff
        args: [--fix]
    # Run formatter after linting
    -   id: ruff-format

-   repo: [https://github.com/pre-commit/mirrors-mypy](https://github.com/pre-commit/mirrors-mypy)
    rev: v1.7.0
    hooks:
    -   id: mypy
        additional_dependencies: ["pydantic"] # Add any stubs-only packages here
        args: [--config-file, pyproject.toml]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

General Python Project - Coding Guidelines

1. Introduction

2. Table of Contents

3. General Quick Guidelines

4. Project Structure

5. Code Organization

6. Dependency Management

7. Configuration Management

8. Error Handling & Exceptions

9. Logging Standards

10. Testing Guidelines

11. Documentation

12. CI/CD

Local Validation

CI Pipeline Checks

13. Security

14. Appendix: Tool Configuration

`pyproject.toml` Configuration

Uh oh!

FilesExpand file tree

CodingGuidelines.md

Latest commit

History

CodingGuidelines.md

File metadata and controls

General Python Project - Coding Guidelines

1. Introduction

2. Table of Contents

3. General Quick Guidelines

4. Project Structure

5. Code Organization

6. Dependency Management

7. Configuration Management

8. Error Handling & Exceptions

9. Logging Standards

10. Testing Guidelines

11. Documentation

12. CI/CD

Local Validation

CI Pipeline Checks

13. Security

14. Appendix: Tool Configuration

pyproject.toml Configuration

`pyproject.toml` Configuration