Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .github/workflows/lint-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
types: |
fix # A bug fix for the user, not a fix to a build script
feat # A new feature for the user, not a new feature for builds
docs # Changes to the documentation
style # Formatting, missing semi colons, etc; no production code change
refactor # Refactoring production code, eg. renaming a variable
perf # Code changes that improve performance
test # Adding missing tests, refactoring tests; no production code change
build # Changes that affect the build system or external dependencies
ci # Changes to our CI configuration files and scripts
fix
feat
docs
style
refactor
perf
test
build
ci
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
python-version: ["3.11", "3.12", "3.13", "3.14"]

steps:
- name: Checkout code
Expand Down
39 changes: 28 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

# Processes: Smart Task Orchestration

[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
[![Python Version](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
![Fast & Lightweight](https://img.shields.io/badge/Library-Pure%20Python-green.svg)
[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue.svg)](https://oliverm91.github.io/processes/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
Expand All @@ -16,7 +16,7 @@

---

**Run a list of Python callables that depend on each other — in parallel when possible, with per-task log files and optional HTML email notification on failure. Zero dependencies. Pure Python 3.10+.**
**Run a list of Python callables that depend on each other — in parallel when possible, with per-task log files and optional HTML email notification on failure. Zero dependencies. Pure Python 3.11+.**

---

Expand All @@ -27,7 +27,7 @@
- 🛡️ **One failure doesn't stop the rest** — a failed task skips only the jobs that depend on it, and **every other part of the workflow keeps running**.
- 📝 **One log file per task** — share a single log across the whole run, or keep them separate for easier debugging.
- 📧 **Email alerts when something breaks** — pass an `SMTPConfig` to a task and get a styled HTML email (with traceback, task context, and the list of jobs that were skipped) the instant it raises.
- 🧰 **Modern, strictly-typed Python 3.10+** — `from __future__ import annotations`, full `mypy --strict` clean, `dict[str, TaskResult]`, `set[str]`, `|` unions.
- 🧰 **Modern, strictly-typed Python 3.11+** — `from __future__ import annotations`, full `mypy --strict` clean, `dict[str, TaskResult]`, `set[str]`, `|` unions.

---

Expand Down Expand Up @@ -87,7 +87,7 @@ A realistic mini-pipeline: fetch two sources **in parallel**, transform them, ag
import logging
from pathlib import Path

from processes import HTMLEmailStyle, Process, SMTPConfig, Task, TaskDependency
from processes import EmailChannel, HTMLEmailStyle, Process, SMTPConfig, Task, TaskDependency

LOG_DIR = Path("logs")
LOG_DIR.mkdir(exist_ok=True)
Expand Down Expand Up @@ -188,7 +188,7 @@ tasks = [
LOG_DIR / "notify_slack.log",
notify_slack,
dependencies=[TaskDependency("build_report", use_result_as_additional_args=True)],
smtp_config=smtp,
channels=[EmailChannel(smtp)],
),
Task(
"archive_report",
Expand Down Expand Up @@ -230,19 +230,17 @@ Task(
args: tuple = (),
kwargs: dict | None = None,
dependencies: list[TaskDependency] | None = None,
smtp_config: SMTPConfig | None = None,
email_style: HTMLEmailStyle | None = None,
channels: list[NotificationChannel] | None = None,
timeout: float | None = None,
retries: int | None = 0,
retry_on: tuple[type[Exception], ...] | None = None,
)
```

- `name` — unique within the `Process`; no spaces.
- `log_path` — the file this task logs to (INFO level, format `%(asctime)s - %(name)s - %(levelname)s - %(message)s`).
- `log_path` — the file this task logs to (INFO level, format `%(asctime)s - %(name)s - %(levelname)s - %(message)s`); wired internally into a file `NotificationChannel`.
- `func` — the callable; receives `func(*args, **kwargs)` after result-injection.
- `smtp_config` — when set, fires an HTML email on `logging.ERROR`; body includes `task_name`, `function`, `args`, `kwargs`, and `downstream_impact`.
- `email_style` — optional presentation override; defaults to `HTMLEmailStyle()` (modern, neutral, English) when `smtp_config` is set.
- `channels` — additional `NotificationChannel`s attached to the task's logger. Use `EmailChannel(smtp_config, style=None)` to fire an HTML email on `logging.ERROR`; body includes `task_name`, `function`, `args`, `kwargs`, and `downstream_impact`. `style` defaults to `HTMLEmailStyle()` (modern, neutral, English).
- `timeout` — seconds allowed per attempt; `None` means no limit. When the timeout fires the underlying thread is detached (Python threading limitation).
- `retries` — additional attempts after the first failure; `0` or `None` means a single attempt. Defaults to `0`.
- `retry_on` — tuple of exception types that trigger a retry. When `retries >= 1` and `retry_on` is `None`, defaults to `(ConnectionError, TimeoutError)` at call time.
Expand Down Expand Up @@ -309,6 +307,25 @@ HTMLEmailStyle(
)
```

### `NotificationChannel`

```python
NotificationChannel # ABC: subclass and implement build_handler(task_name) -> logging.Handler
```

Every `Task` always attaches an internal file channel built from `log_path`. Extra channels passed via `channels` are attached on top of it.

### `EmailChannel`

```python
EmailChannel(
smtp_config: SMTPConfig,
style: HTMLEmailStyle | None = None, # defaults to HTMLEmailStyle()
)
```

Fires a styled HTML email on `logging.ERROR` and above.

All fields are optional — omit `HTMLEmailStyle` entirely to use the defaults.

#### Traced Variables
Expand Down Expand Up @@ -386,7 +403,7 @@ Or straight from the repository (pure Python, no build step):
pip install git+https://github.com/oliverm91/processes.git
```

Requires **Python 3.10+**.
Requires **Python 3.11+**.

---

Expand Down
8 changes: 4 additions & 4 deletions docs/examples/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,9 +197,9 @@ If a task fails it can notify via email:
- The traceback of the error
- The tasks that could not be executed in the process due to this failure.

To set this up, pass an `SMTPConfig` to the Task constructor:
To set this up, pass an `EmailChannel` to the Task constructor via `channels`:
```python
from processes import SMTPConfig, HTMLEmailStyle, Task
from processes import SMTPConfig, HTMLEmailStyle, EmailChannel, Task

smtp = SMTPConfig(
mailhost=('smtp_server', 587),
Expand All @@ -216,7 +216,7 @@ style = HTMLEmailStyle(
language='en', # en | es | pt | fr | de | it
)

t = Task("task_name", "logfile", func_to_run, smtp_config=smtp, email_style=style)
t = Task("task_name", "logfile", func_to_run, channels=[EmailChannel(smtp, style)])
```

## ⏱️ Retries & Timeouts
Expand Down Expand Up @@ -254,4 +254,4 @@ t_fetch = Task(
If every attempt fails, the task is marked failed with the **last**
exception raised — `retries` only controls how many times `func` is
retried, not whether the failure is eventually reported. Combine with
`smtp_config` to be paged only once all attempts are exhausted.
an `EmailChannel` to be paged only once all attempts are exhausted.
18 changes: 9 additions & 9 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

# 🚀 Processes: Robust Routines Management

[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
[![Python Version](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
![Fast & Lightweight](https://img.shields.io/badge/Library-Pure%20Python-green.svg)


Expand Down Expand Up @@ -71,7 +71,7 @@ Define your tasks and their dependencies. **Processes** will handle the executio
```python
from datetime import date

from processes import Process, Task, TaskDependency, SMTPConfig, HTMLEmailStyle
from processes import Process, Task, TaskDependency, SMTPConfig, HTMLEmailStyle, EmailChannel

# 1. Setup Email Alerts (Optional)
smtp_config = SMTPConfig(
Expand Down Expand Up @@ -100,7 +100,7 @@ def sum_data_from_csv_and_x(x, a=1, b=2):
# 3. Create the Task Graph (order is irrelevant, that is handled by Process)
tasks = [
Task("t-1", "etl.log", get_previous_working_day),
Task("intependent", "indep.log", indep_task, smtp_config=smtp_config, email_style=email_style), # This task will send email on failure
Task("intependent", "indep.log", indep_task, channels=[EmailChannel(smtp_config, email_style)]), # This task will send email on failure
Task("sum_csv", "etl.log", search_and_sum_csv,
dependencies= [
TaskDependency("t-1",
Expand All @@ -127,7 +127,7 @@ with Process(tasks) as process: # Context Manager ensures correct disposal of lo

## 📧 Customizing the HTML email

When a task with an `smtp_config` raises, the alert is a **styled HTML
When a task with an `EmailChannel` raises, the alert is a **styled HTML
email** built from a bundled template. The body includes the exception,
the traceback (with the user-frame highlighted), the task context, the
list of downstream tasks that were skipped because of the failure, and
Expand All @@ -146,7 +146,7 @@ Email delivery and presentation are configured with two separate dataclasses:
| `traced_vars_frame_filter` | any path substring, or `None` | `None` (outermost user frame) |

```python
from processes import SMTPConfig, HTMLEmailStyle, Task
from processes import SMTPConfig, HTMLEmailStyle, EmailChannel, Task

smtp = SMTPConfig(
mailhost=("smtp.example.com", 587),
Expand All @@ -162,12 +162,12 @@ style = HTMLEmailStyle(
language="es", # en | es | pt | fr | de | it
)

t = Task("task_name", "logfile", func_to_run, smtp_config=smtp, email_style=style)
t = Task("task_name", "logfile", func_to_run, channels=[EmailChannel(smtp, style)])
```

If `smtp_config` is set and `email_style` is omitted, `HTMLEmailStyle()` defaults
(modern, neutral, English) are used. If `smtp_config` is `None`, `email_style` is ignored
and no email handler is attached.
If `style` is omitted, `EmailChannel` defaults to `HTMLEmailStyle()`
(modern, neutral, English). If no `EmailChannel` is included in `channels`,
no email handler is attached.

All assets ship inside the wheel — the styles are Jinja-style HTML
templates at `src/processes/themes/styles/` and the palettes are CSS
Expand Down
3 changes: 1 addition & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ name = "processes"
dynamic = ["version"]
description = "Orchestrate graphs of callables in Python with automatic dependency resolution, parallel execution, retries, timeouts, and HTML email alerts on failure — zero dependencies"
readme = "README.md"
requires-python = ">=3.10"
requires-python = ">=3.11"
license = "MIT"
authors = [
{ name = "Oliver Mohr Bonometti", email = "oliver.mohr.b@gmail.com" }
Expand All @@ -17,7 +17,6 @@ classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
Expand Down
2 changes: 2 additions & 0 deletions src/processes/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
from .exceptions import (
TaskNotFoundError as TaskNotFoundError,
)
from .notification_channels import EmailChannel as EmailChannel
from .notification_channels import NotificationChannel as NotificationChannel
from .process import Process as Process
from .task import Task as Task
from .task import TaskDependency as TaskDependency
Expand Down
146 changes: 146 additions & 0 deletions src/processes/notification_channels.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
from __future__ import annotations

import logging
from abc import ABC, abstractmethod

from ._email_internals import _build_task_email_handler
from ._logfile_formatting import _TaskLogfileFormatter
from .email_config import HTMLEmailStyle, SMTPConfig


class NotificationChannel(ABC):
"""Base class for task notification channels.

A notification channel knows how to build a configured
``logging.Handler`` that delivers a task's log records (and, on
failure, its structured failure context) to some destination. ``Task``
attaches one handler per configured channel to its logger.

Concrete channels wrap a specific delivery mechanism (e.g. a logfile or
an email alert). New channels can be added by subclassing
``NotificationChannel`` and implementing ``build_handler``.
"""

@abstractmethod
def build_handler(self, task_name: str) -> logging.Handler:
"""Build a configured handler for the given task.

Parameters
----------
task_name : str
Name of the task the handler will be attached to.

Returns
-------
logging.Handler
A handler ready to be added to the task's logger.
"""

@property
def frame_filter(self) -> str | None:
"""Substring selecting the traceback frame to trace local variables of.

See ``HTMLEmailStyle.traced_vars_frame_filter``. Channels that don't
influence frame selection return ``None`` (the default).

Returns
-------
str | None
``None`` unless overridden by a subclass.
"""
return None


class _FileChannel(NotificationChannel):
"""Notification channel that writes task log records to a plain-text file.

Attributes
----------
log_path : str
File path the handler writes to.
level : int
Minimum log level handled. Defaults to ``logging.INFO``.

Parameters
----------
log_path : str
File path the handler writes to.
level : int
Minimum log level handled. Defaults to ``logging.INFO``.
"""

def __init__(self, log_path: str, level: int = logging.INFO):
self.log_path = log_path
self.level = level

def build_handler(self, task_name: str) -> logging.Handler:
"""Build a ``FileHandler`` writing to ``log_path``.

Parameters
----------
task_name : str
Name of the task the handler will be attached to. Unused by
this channel, accepted for interface consistency.

Returns
-------
logging.Handler
A ``FileHandler`` at ``level``, formatted with
``_TaskLogfileFormatter``.
"""
handler = logging.FileHandler(self.log_path)
handler.setLevel(self.level)
handler.setFormatter(_TaskLogfileFormatter())
return handler


class EmailChannel(NotificationChannel):
"""Notification channel that sends an HTML email alert on task failure.

Attributes
----------
smtp_config : SMTPConfig
SMTP transport configuration for the alert.
style : HTMLEmailStyle
HTML presentation settings used to render the alert.

Parameters
----------
smtp_config : SMTPConfig
SMTP transport configuration for the alert.
style : HTMLEmailStyle | None
HTML presentation settings used to render the alert. Defaults to
``HTMLEmailStyle()`` (modern, neutral, English) when ``None``.
"""

def __init__(self, smtp_config: SMTPConfig, style: HTMLEmailStyle | None = None):
self.smtp_config = smtp_config
self.style = style or HTMLEmailStyle()

def build_handler(self, task_name: str) -> logging.Handler:
"""Build an HTML email handler bound to ``task_name``.

Parameters
----------
task_name : str
Name of the task the handler will be attached to, used in the
email subject.

Returns
-------
logging.Handler
A handler at ``logging.ERROR`` level that sends a styled HTML
email for each error log record.
"""
return _build_task_email_handler(self.smtp_config, self.style, task_name)

@property
def frame_filter(self) -> str | None:
"""Frame filter sourced from ``style.traced_vars_frame_filter``.

Returns
-------
str | None
The configured ``traced_vars_frame_filter``, or ``None``.
"""
return self.style.traced_vars_frame_filter
Loading
Loading