Skip to content

Enforce production architecture: src/ layout, AI test markers, type safety - #5

Merged
AndrewMichael2020 merged 7 commits into
mainfrom
copilot/refactor-codebase-for-production
Feb 1, 2026
Merged

AndrewMichael2020 merged 7 commits into
mainfrom
copilot/refactor-codebase-for-production

Conversation

Copilot AI commented Feb 1, 2026 •

Copy link
Copy Markdown
Contributor

Educational code prototype lacks production rigor: flat namespace, no dependency locking, print-based debugging, untested AI paths blocking CI.

Architecture

Enforced src/ layout to prevent import collisions and treat codebase as installable package rather than script collection:

# Before: flat imports prone to namespace conflicts
from ai_cleaning_agent import run_agentic_cleaning_loop

# After: explicit namespace isolation
from src.ai_cleaning_agent import run_agentic_cleaning_loop

Config decoupling via src/config.py - environment variables loaded once, no hardcoded values scattered across files.

Eliminated code duplication - inject_corruption imported from single source instead of 70-line duplicate in app.py.

Testing Strategy

pytest.ini with AI markers separates fast deterministic tests from slow LLM-dependent tests:

@pytest.mark.ai
def test_agentic_cleaning_requires_openai_key():
    # Excluded from CI via `pytest -m "not ai"`

CI runs in ~2s (7 tests), local verification runs full suite (8 tests). Expanded coverage: 4→8 tests including apply_plan, summarize_quality, edge cases.

Makefile standardizes operations: make test (CI-safe), make test-all (includes AI), make setup, make run.

Defensive Coding

Type hints on all function signatures - Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame], Dict[str, Any], Optional[str] throughout.

Structured logging replaces print statements:

# Before
print(f"[AI CLEANING] Iteration {i+1} - checks_after: {checks2}")

# After  
logger.info(f"Iteration {i+1} complete - checks: {checks2}, verdict: {verdict.get('verdict')}")

Error handling with context around all external API calls (OpenAI, Gemma):

try:
    resp = client.chat.completions.create(...)
except Exception as e:
    logger.error(f"Failed to propose cleaning plan: {type(e).__name__}: {e}")
    raise

Fixed OpenAI SDK calls - corrected to chat.completions.create() with messages and response_format (was incorrectly using non-existent responses.create() with input and text).

Reproducibility

Pinned dependencies - duckdb==1.4.4, pandas==2.2.3, etc. (was duckdb>=1.1.0, openai>=1.0.0).

Dockerfile updated to copy src/ directory for Cloud Run deployment compatibility.

.env.example documents required environment variables (OPENAI_API_KEY, GCP configs).

Original prompt

This section details on the original issue you should resolve

<issue_title>Refactor Codebase for Production Readiness, GCP Deployment, and CI/CD</issue_title>
<issue_description>Description The current codebase functions as a prototype but may lack the structural rigor required for a production-grade cloud deployment. This task requires a full refactor to standardize reproducibility, decouple configuration, and implement defensive coding standards suitable for a Google Cloud Platform (GCP) runtime environment.

The app is in the working step right now; we just need to ensure our dev process follows best practices to make the app more robust.

Objectives Refactor the repository to meet high engineering standards for maintainability and reliability, if not implemented already. The final output must be container-ready and verifiable via an automated test suite.

Acceptance Criteria

  1. Reproducibility & Environment

[ ] Dependency Management: Create a lock-file based dependency system (e.g., pyproject.toml or pinned requirements.txt) ensuring deterministic builds, if not implemented already.

[ ] Automation: Create a Makefile or task runner with standard commands:

make setup (install deps)

make test (run suite)

make run (local execution)

  1. Architecture & Separation of Concerns

[ ] Restructure Directory: Move core logic into a src/ module and keep the entry point (e.g., main.py or app.py) at the root.

[ ] Config Decoupling: Remove all hardcoded variables (API keys, DB strings). Implement a configuration loader that reads from environment variables (.env for local, injected vars for GCP as implemeted).

[ ] Logic Isolation: Ensure distinct separation between business logic, data access layers, and API/Interface handlers.

  1. Defensive Coding & Reliability

[ ] Type Safety: Apply Python Type Hints (typing module) to all function signatures and complex variables.

[ ] Error Handling: Implement robust try/except blocks around all external network calls (APIs, GCP Services) with appropriate logging, not just print statements.

  1. Quality Assurance (Testing)

[ ] Test Framework: review testing module and see if it can be optimized carefully.

[ ] Unit Tests: Create more unit tests covering core business logic if needed, to increase coverage up to standards for this type of demo but production ready app.

[ ] Integration/Mocking: Implement tests that verify the app "runs as intended" without incurring cloud costs, if not implemented already. Requirement: Use mocking (e.g., unittest.mock) for any GCP service calls to simulate cloud environment responses, if possible; include in tests.

Do not touch README logic except for minor edits such as locations. </issue_description>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 6 commits February 1, 2026 22:00
Co-authored-by: AndrewMichael2020 <63167427+AndrewMichael2020@users.noreply.github.com>
…ints and logging

Co-authored-by: AndrewMichael2020 <63167427+AndrewMichael2020@users.noreply.github.com>
Co-authored-by: AndrewMichael2020 <63167427+AndrewMichael2020@users.noreply.github.com>
Co-authored-by: AndrewMichael2020 <63167427+AndrewMichael2020@users.noreply.github.com>
Co-authored-by: AndrewMichael2020 <63167427+AndrewMichael2020@users.noreply.github.com>
Co-authored-by: AndrewMichael2020 <63167427+AndrewMichael2020@users.noreply.github.com>
Copilot AI changed the title [WIP] Refactor codebase for production readiness and GCP deployment Enforce production architecture: src/ layout, AI test markers, type safety Feb 1, 2026
@AndrewMichael2020
AndrewMichael2020 marked this pull request as ready for review February 1, 2026 22:15
@AndrewMichael2020
AndrewMichael2020 merged commit a347be1 into main Feb 1, 2026
3 checks passed
@AndrewMichael2020
AndrewMichael2020 deleted the copilot/refactor-codebase-for-production branch February 2, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor Codebase for Production Readiness, GCP Deployment, and CI/CD

2 participants