Add raw Pine Docker backtest harness

.dockerignore

@@ -0,0 +1,23 @@
+.git
+.github
+.venv
+build
+dist
+**/__pycache__
+**/*.pyc
+tests
+vendor/pineforge-engine/.git
+vendor/pineforge-engine/build
+vendor/pineforge-engine/benchmarks
+vendor/pineforge-engine/corpus
+vendor/pineforge-engine/docs
+vendor/pineforge-engine/scripts
+vendor/pineforge-engine/tests
+vendor/pineforge-engine/tutorial
+vendor/pineforge-codegen-oss/.git
+vendor/pineforge-codegen-oss/.github
+vendor/pineforge-codegen-oss/docs
+vendor/pineforge-codegen-oss/gate
+vendor/pineforge-codegen-oss/npm
+vendor/pineforge-codegen-oss/scripts
+vendor/pineforge-codegen-oss/tests

.github/workflows/ci.yml

@@ -28,3 +28,16 @@ jobs:
       - run: pytest
       - if: matrix.python-version == '3.13'
         run: python -m build
+
+  docker:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          submodules: true
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.13"
+          cache: pip
+      - run: python -m pip install -e '.[dev]'
+      - run: PINEFORGE_DOCKER_TEST=1 pytest tests/test_docker_integration.py

.gitmodules

@@ -0,0 +1,6 @@
+[submodule "vendor/pineforge-codegen-oss"]
+	path = vendor/pineforge-codegen-oss
+	url = https://github.com/pineforge-4pass/pineforge-codegen-oss.git
+[submodule "vendor/pineforge-engine"]
+	path = vendor/pineforge-engine
+	url = https://github.com/pineforge-4pass/pineforge-engine.git

CONTRIBUTING.md

@@ -3,6 +3,14 @@
 Community data providers are the reason this repository exists. Keep the core
 contracts small and put vendor behavior behind provider modules.
 
+Docker and initialized codegen/engine submodules are required for raw-Pine
+integration work:
+
+```bash
+git submodule update --init
+docker version
+```
+
 ## Provider checklist
 
 - Implement one or more protocols from `pineforge_data.providers`.
@@ -31,4 +39,5 @@ ruff check .
 mypy src
 pytest
 python -m build
+PINEFORGE_DOCKER_TEST=1 pytest tests/test_docker_integration.py
 ```

README.md

@@ -74,13 +74,36 @@ adapter emits `start_sequence + 1` next.
 
 ## Direct backtest harness
 
-`pineforge-backtest` fetches confirmed OHLCV through a data provider, packs the
-normalized bars into the PineForge C ABI, and calls a compiled strategy library
-directly. It does not create an intermediate CSV.
+The public backtest input is raw PineScript v6. `pineforge-backtest` fetches
+confirmed OHLCV through a data provider and runs this pinned pipeline:
+
+```text
+raw .pine + provider OHLCV
+        ↓ read-only mount
+Docker: Python codegen → C++ strategy → pineforge-engine → JSON report
+```
+
+Docker is a prerequisite. A host C++ compiler and a precompiled strategy
+library are not required.
+
+Clone with both pinned runtime dependencies:
 
 ```bash
-pineforge-backtest \
-  --strategy /path/to/strategy.so \
+git clone --recurse-submodules https://github.com/pineforge-4pass/pineforge-data.git
+cd pineforge-data
+python3 -m venv .venv
+.venv/bin/pip install -e '.[ccxt]'
+```
+
+For an existing checkout:
+
+```bash
+git submodule update --init
+```
+
+```bash
+.venv/bin/pineforge-backtest \
+  --pine strategy.pine \
   --exchange kraken \
   --symbol BTC/USD \
   --timeframe 15m \
@@ -90,6 +113,14 @@ pineforge-backtest \
   --pretty
 ```
 
+The first invocation builds a local image tagged from the container source and
+the exact codegen and engine submodule commits. Later invocations reuse it.
+Pass `--rebuild-image` to force a rebuild or `--no-image-build` to require a
+prebuilt local image.
+Compilation and execution run as a non-root user with networking disabled, all
+Linux capabilities dropped, a read-only root filesystem, and only a read-only
+temporary input mount.
+
 The JSON report contains data provenance, processed-bar counts, every closed
 trade, all/long/short trade statistics, equity statistics, security-feed
 diagnostics, optional trace values, and the complete equity curve. Unix
@@ -99,11 +130,17 @@ Use `--provider-config config.json` for CCXT constructor options and
 `--strategy-params inputs.json` for Pine input overrides. The provider config
 file may contain credentials, so keep it outside version control.
 
+The report records the Pine source hash, generated C++ hash, transpile and
+compile timings, and the exact codegen and engine commits. The OSS codegen is
+source-available under its own PolyForm Noncommercial license and supplemental
+terms; review `vendor/pineforge-codegen-oss/LICENSE` before distribution or
+commercial use. The engine remains Apache-2.0.
+
 Provider implementations are organized by their strongest supported runtime.
 The current Python bucket contains CCXT and the harness; native low-latency
-providers will live in the C++ bucket. Both buckets must emit the same
-normalized records, but an individual provider does not need implementations
-in both languages.
+providers live under `cpp/providers`. Both buckets must emit the same normalized
+records, but an individual provider does not need implementations in both
+languages.
 
 ## Development
 
@@ -113,6 +150,7 @@ python3 -m venv .venv
 .venv/bin/ruff check .
 .venv/bin/mypy src
 .venv/bin/pytest
+PINEFORGE_DOCKER_TEST=1 .venv/bin/pytest tests/test_docker_integration.py
 ```
 
 ## Provider boundary

cpp/providers/README.md

@@ -0,0 +1,9 @@
+# C++ providers
+
+This directory is the native provider bucket for feeds that have a supported
+C++ SDK or a measured latency/throughput requirement.
+
+A native provider must normalize into the same bar and trade field contract as
+the Python package and use shared conformance fixtures. Providers do not need a
+duplicate Python implementation, and provider-specific types must not leak into
+`pineforge-engine`.

docker/Dockerfile

@@ -0,0 +1,63 @@
+# syntax=docker/dockerfile:1@sha256:87999aa3d42bdc6bea60565083ee17e86d1f3339802f543c0d03998580f9cb89
+
+FROM debian:bookworm-slim@sha256:60eac759739651111db372c07be67863818726f754804b8707c90979bda511df AS engine-builder
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        cmake \
+        g++ \
+        libeigen3-dev \
+        ninja-build \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY vendor/pineforge-engine /src/pineforge-engine
+
+RUN cmake -G Ninja -S /src/pineforge-engine -B /tmp/pineforge-build \
+        -DCMAKE_BUILD_TYPE=Release \
+        -DCMAKE_INSTALL_PREFIX=/opt/pineforge \
+        -DPINEFORGE_BUILD_TESTS=OFF \
+        -DPINEFORGE_BUILD_TUTORIAL=OFF \
+        -DPINEFORGE_BUILD_CORPUS_STRATEGIES=OFF \
+    && cmake --build /tmp/pineforge-build --target pineforge -j2 \
+    && cmake --install /tmp/pineforge-build
+
+FROM python:3.13-slim-bookworm@sha256:fcbd8dfc2605ba7c2eca646846c5e892b2931e41f6227985154a596f26ab8ed7
+
+ARG ENGINE_SHA=unknown
+ARG CODEGEN_SHA=unknown
+ARG DATA_SOURCE_DIGEST=unknown
+
+LABEL org.opencontainers.image.title="pineforge-data-backtest" \
+      org.opencontainers.image.description="Raw PineScript and provider OHLCV to PineForge report" \
+      org.opencontainers.image.source="https://github.com/pineforge-4pass/pineforge-data" \
+      org.opencontainers.image.licenses="Apache-2.0 AND LicenseRef-PolyForm-Noncommercial-1.0.0" \
+      io.pineforge.engine.commit="${ENGINE_SHA}" \
+      io.pineforge.codegen.commit="${CODEGEN_SHA}" \
+      io.pineforge.data.source-digest="${DATA_SOURCE_DIGEST}"
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        g++ \
+        libeigen3-dev \
+    && rm -rf /var/lib/apt/lists/* /usr/share/doc /usr/share/man
+
+COPY --from=engine-builder /opt/pineforge /opt/pineforge
+COPY vendor/pineforge-codegen-oss/pineforge_codegen /opt/pineforge-codegen/pineforge_codegen
+COPY vendor/pineforge-codegen-oss/VERSION /opt/pineforge-codegen/VERSION
+COPY src/pineforge_data /opt/pineforge-data/pineforge_data
+COPY docker/entrypoint.py /opt/pineforge/bin/pineforge-data-backtest
+COPY LICENSE /opt/licenses/pineforge-data.LICENSE
+COPY vendor/pineforge-engine/LICENSE /opt/licenses/pineforge-engine.LICENSE
+COPY vendor/pineforge-codegen-oss/LICENSE /opt/licenses/pineforge-codegen.LICENSE
+
+RUN printf '%s\n' "${ENGINE_SHA}" > /opt/pineforge/ENGINE_COMMIT \
+    && printf '%s\n' "${CODEGEN_SHA}" > /opt/pineforge/CODEGEN_COMMIT \
+    && printf '%s\n' "${DATA_SOURCE_DIGEST}" > /opt/pineforge/DATA_SOURCE_DIGEST \
+    && chmod +x /opt/pineforge/bin/pineforge-data-backtest \
+    && useradd --system --uid 10001 --shell /usr/sbin/nologin pineforge
+
+ENV PYTHONPATH=/opt/pineforge-data:/opt/pineforge-codegen \
+    PINEFORGE_PREFIX=/opt/pineforge \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1
+
+USER 10001
+ENTRYPOINT ["/opt/pineforge/bin/pineforge-data-backtest"]

docker/entrypoint.py

@@ -0,0 +1,170 @@
+#!/usr/bin/env python3
+"""Container entrypoint: raw Pine + normalized OHLCV -> JSON backtest report."""
+
+from __future__ import annotations
+
+import csv
+import hashlib
+import json
+import os
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+from typing import cast
+
+from pineforge_codegen import transpile
+from pineforge_codegen.errors import CompileError
+
+from pineforge_data import (
+    BacktestOptions,
+    Bar,
+    Instrument,
+    MagnifierDistribution,
+    PineForgeBacktestRunner,
+)
+from pineforge_data.backtest import JsonValue
+
+WORK = Path("/work")
+PREFIX = Path(os.environ.get("PINEFORGE_PREFIX", "/opt/pineforge"))
+
+
+def _load_request() -> dict[str, object]:
+    value = json.loads((WORK / "request.json").read_text(encoding="utf-8"))
+    if not isinstance(value, dict):
+        raise ValueError("request.json must contain an object")
+    return cast(dict[str, object], value)
+
+
+def _load_bars(path: Path, instrument: Instrument, source: str) -> list[Bar]:
+    bars: list[Bar] = []
+    with path.open(newline="", encoding="utf-8") as handle:
+        for row in csv.DictReader(handle):
+            bars.append(
+                Bar(
+                    instrument,
+                    int(row["timestamp"]),
+                    float(row["open"]),
+                    float(row["high"]),
+                    float(row["low"]),
+                    float(row["close"]),
+                    float(row["volume"]),
+                    source,
+                )
+            )
+    return bars
+
+
+def _read_commit(name: str) -> str:
+    return (PREFIX / name).read_text(encoding="utf-8").strip()
+
+
+def _compile(generated: Path, strategy_library: Path) -> float:
+    command = [
+        "g++",
+        "-std=c++17",
+        "-O2",
+        "-ffp-contract=off",
+        "-fPIC",
+        "-shared",
+        f"-I{PREFIX / 'include'}",
+        "-I/usr/include/eigen3",
+        str(generated),
+        "-Wl,--whole-archive",
+        str(PREFIX / "lib/libpineforge.a"),
+        "-Wl,--no-whole-archive",
+        "-o",
+        str(strategy_library),
+    ]
+    started = time.perf_counter()
+    completed = subprocess.run(command, text=True, capture_output=True, check=False)
+    elapsed = time.perf_counter() - started
+    if completed.returncode != 0:
+        detail = completed.stderr.strip() or completed.stdout.strip() or "unknown compiler error"
+        raise RuntimeError(f"strategy compilation failed:\n{detail}")
+    return elapsed
+
+
+def main() -> int:
+    pine_path = WORK / "strategy.pine"
+    ohlcv_path = WORK / "ohlcv.csv"
+    if not pine_path.is_file() or not ohlcv_path.is_file():
+        print(
+            "error: /work must contain strategy.pine, ohlcv.csv, and request.json",
+            file=sys.stderr,
+        )
+        return 2
+
+    try:
+        request = _load_request()
+        instrument_data = cast(dict[str, object], request["instrument"])
+        options_data = cast(dict[str, object], request["options"])
+        instrument = Instrument(
+            symbol=str(instrument_data["symbol"]),
+            venue=str(instrument_data.get("venue", "")),
+            timezone=str(instrument_data.get("timezone", "UTC")),
+            session=str(instrument_data.get("session", "24x7")),
+            volume_unit=str(instrument_data.get("volume_unit", "base")),
+        )
+        bars = _load_bars(ohlcv_path, instrument, str(request.get("source", "provider")))
+        options = BacktestOptions(
+            input_timeframe=str(options_data.get("input_timeframe", "")),
+            script_timeframe=str(options_data.get("script_timeframe", "")),
+            bar_magnifier=bool(options_data.get("bar_magnifier", False)),
+            magnifier_samples=int(options_data.get("magnifier_samples", 4)),
+            magnifier_distribution=MagnifierDistribution(
+                int(options_data.get("magnifier_distribution", 3))
+            ),
+            trace_enabled=bool(options_data.get("trace_enabled", False)),
+            chart_timezone=cast(str | None, options_data.get("chart_timezone")),
+        )
+        strategy_params = cast(dict[str, JsonValue], request.get("strategy_params", {}))
+        pine_source = pine_path.read_text(encoding="utf-8")
+
+        with tempfile.TemporaryDirectory(prefix="pineforge-") as temporary:
+            temp = Path(temporary)
+            generated = temp / "strategy.cpp"
+            strategy_library = temp / "strategy.so"
+            transpile_started = time.perf_counter()
+            generated_source = transpile(pine_source, filename=pine_path.name)
+            transpile_seconds = time.perf_counter() - transpile_started
+            generated.write_text(generated_source, encoding="utf-8")
+            compile_seconds = _compile(generated, strategy_library)
+            report = PineForgeBacktestRunner.load(strategy_library).run(
+                bars,
+                instrument=instrument,
+                options=options,
+                strategy_params=strategy_params,
+            )
+
+        output: dict[str, object] = {
+            "runtime": {
+                "data_source_digest": _read_commit("DATA_SOURCE_DIGEST"),
+                "engine_commit": _read_commit("ENGINE_COMMIT"),
+                "codegen_commit": _read_commit("CODEGEN_COMMIT"),
+                "codegen_version": (
+                    Path("/opt/pineforge-codegen/VERSION").read_text(encoding="utf-8").strip()
+                ),
+            },
+            "transpile": {
+                "seconds": round(transpile_seconds, 6),
+                "pine_sha256": hashlib.sha256(pine_source.encode()).hexdigest(),
+                "generated_cpp_sha256": hashlib.sha256(generated_source.encode()).hexdigest(),
+                "generated_cpp_bytes": len(generated_source.encode()),
+            },
+            "compile": {"seconds": round(compile_seconds, 6)},
+            "backtest": report.to_dict(),
+        }
+        print(json.dumps(output, separators=(",", ":"), allow_nan=False))
+        return 0
+    except CompileError as exc:
+        print(f"error: PineScript transpilation failed: {exc}", file=sys.stderr)
+        return 5
+    except Exception as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 4
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())