Add PLAN.md: detailed strategy for modular ML pipeline - #1
Open
pieris98 wants to merge 7 commits into
Open
Conversation
Outlines architecture for pluggable encoder/decoder registry, pipeline orchestrator with YAML-driven stage execution, third-party plugin interface, and metrics abstraction layer. https://claude.ai/code/session_013WW5ia7wHgxxetTyVhVL1L
Implements ComponentRegistry with register/get/list/decorator APIs for encoders, decoders, metrics, stages, and datasets. Plugin loader supports loading third-party modules from file paths or installed packages via a register(registry) convention. 31 unit tests all passing. https://claude.ai/code/session_013WW5ia7wHgxxetTyVhVL1L
Creates src/decoders/ package with: - BaseDecoder ABC defining forward(), load_checkpoint(), decode_to_sequences() - TransformerDecoderWrapper wrapping existing TransformerDecoder - LMHeadDecoder wrapping encoder lm_heads - Registry integration via decorators - Built-in encoders registered in src/encoders/__init__.py https://claude.ai/code/session_013WW5ia7wHgxxetTyVhVL1L
Creates BaseMetric ABC with compute() interface and concrete wrappers (FIDMetric, MMDMetric, ESMPPPLMetric, PLDDTMetric). Each wrapper delegates to the existing metric functions while providing a uniform interface. Registry setup module registers all built-in metrics. https://claude.ai/code/session_013WW5ia7wHgxxetTyVhVL1L
Implements 7 pipeline stages as registered BaseStage subclasses: - setup_data: dataset download, filtering, and splitting - setup_models: encoder model download and caching - calculate_statistics: normalization statistics computation - train_decoder: optional decoder training - train_diffusion: diffusion model training with DDP support - run_inference: sample generation from trained models - evaluate_metrics: pluggable metric computation via registry Each stage has validate() and run() methods with shared context passing. https://claude.ai/code/session_013WW5ia7wHgxxetTyVhVL1L
Implements PipelineOrchestrator that reads YAML pipeline configs and executes stages in sequence with shared context. Features: - Plugin loading at pipeline start - Stage validation before execution - Dry-run mode (--dry-run) - Configurable on_failure (abort/continue) per stage - Stage-specific parameter overrides - CLI entry point (run_pipeline.py) with --list-components Includes 3 pipeline configs: full_pipeline, inference_only, train_only. https://claude.ai/code/session_013WW5ia7wHgxxetTyVhVL1L
Includes end-to-end tests for: - Plugin loading and component registration - Custom metric computation via plugin - Custom stage execution via plugin - Multi-stage orchestration with context passing - Selective stage enable/disable - Pipeline abort vs continue on failure Example plugin demonstrates the third-party developer interface. Full test suite: 79 tests, all passing. https://claude.ai/code/session_013WW5ia7wHgxxetTyVhVL1L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outlines architecture for pluggable encoder/decoder registry, pipeline
orchestrator with YAML-driven stage execution, third-party plugin
interface, and metrics abstraction layer.
https://claude.ai/code/session_013WW5ia7wHgxxetTyVhVL1L