Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 60 additions & 81 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,84 +1,63 @@
# ml4t-engineer

Feature engineering for financial ML: compute indicators, create labels, sample
alternative bars, and prepare leakage-safe datasets for model training.

## Quick Start

```python
from ml4t.engineer import compute_features, create_dataset_builder
from ml4t.engineer.config import LabelingConfig
from ml4t.engineer.labeling import triple_barrier_labels

features = compute_features(df, ["rsi", "macd", "atr"])
labels = triple_barrier_labels(
features,
config=LabelingConfig.triple_barrier(
upper_barrier=0.02, lower_barrier=0.01, max_holding_period=20,
),
)
builder = create_dataset_builder(
features=labels.select(["rsi", "macd", "atr"]),
labels=labels["label"],
dates=labels["timestamp"],
scaler="robust",
)
```

## Directory Map

| Path | Purpose | Key Surfaces |
|------|---------|--------------|
| `src/ml4t/engineer/features/` | Feature computation and discovery metadata | `compute_features()`, `feature_catalog`, `FeatureCatalog` |
| `src/ml4t/engineer/labeling/` | Supervised label generation and sample weighting | `triple_barrier_labels()`, `atr_triple_barrier_labels()`, `rolling_percentile_binary_labels()` |
| `src/ml4t/engineer/bars/` | Tick, volume, dollar, imbalance, and run bars | `TickBarSampler`, `VolumeBarSampler`, `DollarBarSampler` |
| `src/ml4t/engineer/dataset.py` | Leakage-safe dataset preparation | `MLDatasetBuilder`, `create_dataset_builder()` |
| `src/ml4t/engineer/preprocessing.py` | Train-only scalers and pipelines | `StandardScaler`, `RobustScaler`, `PreprocessingPipeline` |
| `src/ml4t/engineer/config/` | Reusable config objects | `LabelingConfig`, `PreprocessingConfig`, `DataContractConfig` |

## Public Entry Points

```python
from ml4t.engineer import (
compute_features,
create_dataset_builder,
feature_catalog,
FeatureCatalog,
StandardScaler,
RobustScaler,
)
from ml4t.engineer.config import LabelingConfig
from ml4t.engineer.labeling import (
triple_barrier_labels,
atr_triple_barrier_labels,
rolling_percentile_binary_labels,
fixed_time_horizon_labels,
trend_scanning_labels,
)
from ml4t.engineer.bars import (
TickBarSampler,
VolumeBarSampler,
DollarBarSampler,
FixedTickImbalanceBarSampler,
FixedVolumeImbalanceBarSampler,
)
`ml4t-engineer` provides Polars-first feature engineering, labeling, alternative
bars, and leakage-safe dataset preparation for financial machine learning.

## Source orientation

Runtime code lives under `src/ml4t/engineer/`.

- `api.py`, `features/`, and `discovery/` implement registry-backed feature
computation and discovery.
- `labeling/` implements path-dependent, fixed-horizon, percentile, meta-label,
and sample-weighting workflows.
- `bars/` converts trade data into tick, volume, dollar, imbalance, and run bars.
- `dataset.py` and `preprocessing.py` implement train/test preparation and
train-only transformations.
- `core/` and `config/` contain registry metadata, validation, schemas, and
reusable configuration models.
- `artifacts/`, `relationships/`, `store/`, `logging/`, and `utils/` provide
supporting services.

Tests are under `tests/`. Executable examples and repository checks are under
`examples/` and `scripts/`.

## Public workflows

Stable entry points include:

- `ml4t.engineer.compute_features`
- `ml4t.engineer.feature_catalog`
- `ml4t.engineer.create_dataset_builder`
- labeling functions under `ml4t.engineer.labeling`
- bar samplers under `ml4t.engineer.bars`

Start with `README.md` and `docs/getting-started/`. Detailed workflow guidance
is under `docs/user-guide/`, and the generated API reference is under
`docs/api/`.

## Engineering constraints

- Python 3.12, 3.13, and 3.14 are supported.
- Feature computation must preserve the documented DataFrame or LazyFrame
return behavior.
- Changes to registered features must keep implementation, registry metadata,
dependencies, normalization metadata, and lookback behavior consistent.
- Fit preprocessing state only on training data.
- Keep optional dependencies isolated from the base import.
- Shared data-contract changes originate in `ml4t-specs`.

## Quality commands

Run from the repository root:

```bash
uv sync --dev --extra docs --extra ta --extra store --extra viz
uv run ruff check src/ tests/ examples/ scripts/
uv run ruff format --check src/ tests/ examples/ scripts/
uv run ty check
uv run pytest tests/ -q
uv build
uv run python -c "import ml4t.engineer"
uv run mkdocs build --strict
```

## Core Workflows

- `compute_features()` appends indicator columns to an OHLCV DataFrame or LazyFrame
- `LabelingConfig` plus labeling functions create reusable supervised targets
- `MLDatasetBuilder` handles train/test splitting and train-only scaling
- sampler classes in `bars/` turn trade data into non-time bars for downstream use

## Trust Signals

- 120 features across 11 categories
- 60 TA-Lib validated indicators at `1e-6` tolerance
- ~50,000 labels/second for triple-barrier workflows
- shared book integration via `docs/book-guide/index.md`

## Navigation

See [src/ml4t/engineer/AGENTS.md](src/ml4t/engineer/AGENTS.md) for package-level
module orientation and subdirectory entry points.
113 changes: 30 additions & 83 deletions src/ml4t/engineer/AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,83 +1,30 @@
# ml4t.engineer Package

Package-level navigation for the public `ml4t-engineer` surface.

## Main Modules

| Module | Purpose | Key Exports |
|--------|---------|-------------|
| `api.py` | Feature-computation entry point | `compute_features()` |
| `dataset.py` | Leakage-safe dataset preparation | `MLDatasetBuilder`, `create_dataset_builder()` |
| `preprocessing.py` | Train-only scalers and transform pipelines | `StandardScaler`, `MinMaxScaler`, `RobustScaler`, `PreprocessingPipeline` |
| `discovery/catalog.py` | Metadata-driven feature exploration | `FeatureCatalog`, `feature_catalog` |
| `config/` | Reusable config models | `LabelingConfig`, `PreprocessingConfig`, `DataContractConfig` |
| `__init__.py` | Public re-exports and AGENTS discovery helper | `get_agent_docs()` |

## Subdirectories

| Directory | Purpose | AGENTS |
|-----------|---------|--------|
| `features/` | Registry-backed indicator implementations and standalone feature helpers | [features/AGENTS.md](features/AGENTS.md) |
| `labeling/` | Barrier labels, percentile labels, meta-labeling, uniqueness | [labeling/AGENTS.md](labeling/AGENTS.md) |
| `bars/` | Tick, volume, dollar, imbalance, and run bars | [bars/AGENTS.md](bars/AGENTS.md) |
| `core/` | Registry, metadata, schemas, decorators, validation | [core/AGENTS.md](core/AGENTS.md) |
| `config/` | Pydantic configuration models and schema bridges | [config/AGENTS.md](config/AGENTS.md) |
| `discovery/` | Metadata-driven feature search and filtering | [discovery/AGENTS.md](discovery/AGENTS.md) |
| `relationships/` | Correlation helpers and plotting utilities | [relationships/AGENTS.md](relationships/AGENTS.md) |
| `store/` | Offline DuckDB storage helpers | [store/AGENTS.md](store/AGENTS.md) |
| `artifacts/` | Lightweight artifact records for features, labels, predictions | [artifacts/AGENTS.md](artifacts/AGENTS.md) |
| `logging/` | Structured logging configuration | [logging/AGENTS.md](logging/AGENTS.md) |
| `utils/` | optional dependency helpers and low-level utilities | [utils/AGENTS.md](utils/AGENTS.md) |

## Current Public API Shape

```python
from ml4t.engineer import (
compute_features,
create_dataset_builder,
feature_catalog,
MLDatasetBuilder,
StandardScaler,
RobustScaler,
)
from ml4t.engineer.config import LabelingConfig, PreprocessingConfig
from ml4t.engineer.labeling import triple_barrier_labels, atr_triple_barrier_labels
from ml4t.engineer.bars import TickBarSampler, VolumeBarSampler, DollarBarSampler
```

## Core Patterns

### Compute features through the registry

```python
from ml4t.engineer import compute_features

result = compute_features(df, ["rsi", "macd", "atr"])
```

### Discover features through metadata

```python
from ml4t.engineer import feature_catalog

feature_catalog.list(category="momentum")
feature_catalog.search("volatility estimator")
feature_catalog.describe("rsi")
```

### Build train/test data without leakage

```python
from ml4t.engineer import create_dataset_builder

builder = create_dataset_builder(features, labels, scaler="robust")
X_train, X_test, y_train, y_test = builder.train_test_split(train_size=0.8)
```

## Notes

- `FeatureSelector` has moved out of this library and belongs in `ml4t-diagnostic`
- `store/` exists but is lower-priority than the core feature, labeling, bar, and
dataset workflows
- AGENTS files are the current navigation surface; older singular filename references
should be treated as stale
# ml4t.engineer package

These instructions apply within `src/ml4t/engineer/`. Use the repository-root
`AGENTS.md` for setup and quality commands.

## Internal structure

- `core/registry.py` and `core/decorators.py` own feature registration and
metadata.
- `api.py` resolves registered features, their dependencies, and execution
order.
- `features/` contains registered indicators and standalone transforms.
- `labeling/`, `bars/`, `dataset.py`, and `preprocessing.py` implement the
primary downstream workflows.
- `config/` bridges reusable configuration and shared `ml4t-specs` contracts.

## Local rules

- A registered feature change must keep its decorator metadata and first usable
row consistent with its implementation.
- Preserve DataFrame and LazyFrame behavior where the public workflow supports
both.
- Label calculations must not introduce future information before the declared
horizon.
- Preprocessing state is fitted on training observations only.
- Guard optional integrations through the existing dependency helpers so
`import ml4t.engineer` works with base dependencies.

Use the scoped guides in `features/AGENTS.md`, `labeling/AGENTS.md`, and
`bars/AGENTS.md` when working in those subsystems.
8 changes: 0 additions & 8 deletions src/ml4t/engineer/labeling/calendar.py
Original file line number Diff line number Diff line change
Expand Up @@ -326,14 +326,6 @@ def _calendar_session_ids(
return sorted_data, session_ids


# ExchangeCalendar adapter removed - use pandas_market_calendars directly
# See .claude/reference/calendar_libraries.md for rationale:
# - pandas_market_calendars includes ALL exchange_calendars features as dependency
# - Adds critical product-specific calendars (CME_Equity, CME_Agriculture, etc.)
# - Correctly handles CME futures maintenance breaks (4-5 PM CT)
# - Better maintenance, more features, zero downside


def calendar_aware_labels(
data: pl.DataFrame,
config: LabelingConfig,
Expand Down