Skip to content

Add schema-mapped local data providers#6

Merged
luisleo526 merged 1 commit into
mainfrom
codex/local-data-providers
Jul 10, 2026
Merged

Add schema-mapped local data providers#6
luisleo526 merged 1 commit into
mainfrom
codex/local-data-providers

Conversation

@luisleo526

Copy link
Copy Markdown
Contributor

Summary

  • add runtime TabularSchema inspection and partial or explicit BarColumnMapping APIs
  • add built-in CSV, read-only SQLite, and SQLAlchemy 2.x historical bar providers
  • register csv, sqlite, and sqlalchemy for the existing raw-Pine backtest harness
  • support numeric and ISO-8601 timestamps, optional symbol/timeframe columns, strict duplicate detection, and normalized provenance
  • add environment-variable database URLs so credentials do not need to live in provider JSON
  • document programmatic and CLI workflows for arbitrary file/table/view schemas

Why

Users should be able to backtest private datasets without migrating them into a PineForge-owned DDL. Runtime reflection plus explicit semantic mapping supports arbitrary source names while avoiding unsafe guesses and raw SQL configuration.

Safety and behavior

  • common OHLCV aliases are inferred; missing or ambiguous mappings fail with available-column guidance
  • SQLite opens files read-only
  • reflected identifiers are validated and safely quoted; symbol, timeframe, and time bounds use bound expressions or parameters
  • SQLAlchemy connection parameters are hidden from logging by default
  • local records are treated as confirmed snapshots and still pass the normal PineForge Bar validation

Validation

  • ruff format --check src tests
  • ruff check .
  • mypy src
  • pytest — 82 passed, 2 expected Docker-gated skips
  • python -m build
  • clean-wheel import without SQLAlchemy installed, confirming the database dependency stays optional

@luisleo526 luisleo526 marked this pull request as ready for review July 10, 2026 15:13
@luisleo526 luisleo526 merged commit 73aa825 into main Jul 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant