Refactor codebase with modular parsers and enhanced schema for expanded data extraction

[Copilot]

Implemented comprehensive refactoring to expand extracted course information and improve code organization through specialized parsers and enhanced database schema.

Architecture Changes

New modular structure:

• src/config/ - Unified Settings class (28 params) + constants, replaces disparate Config classes
• src/models/ - 11 Pydantic schemas with automatic validation
• src/parsers/ - Specialized HTML parsers with BaseParser abstract class
• src/utils/ - Centralized logging with ContextLogger

Backward compatibility maintained: Legacy Config class available via import alias.

Enhanced Data Extraction

Schema additions (schema_enhanced.sql):

• +21 fields: duration, ECTS, objectives, competencies, outcomes, accreditation, director info, tuition
• +3 tables: curriculum_structure (hierarchical year→semester→module), scraping_log, change_detection

Specialized parsers:

• CourseParser - Extracts 25+ fields from course pages
• CurriculumParser - Parses curriculum tables/lists with automatic year/semester detection
• ModuleParser - Extracts detailed UC information (syllabus, teaching methods, workload, professor)

Usage Example

from bs4 import BeautifulSoup
from src.parsers import CourseParser, CurriculumParser
from src.models import CourseCreate

# Parse course page
soup = BeautifulSoup(html, 'html.parser')
parser = CourseParser(soup, url)
course_data = parser.parse()

# Validate with Pydantic
course = CourseCreate(**course_data)  # Auto-validates 25+ fields

# Parse curriculum structure
curriculum_parser = CurriculumParser(soup, plan_url)
curriculum = curriculum_parser.parse()
# Returns: [{"year": 1, "semester": 1, "modules": [...]}]

Test Coverage

• 61 tests total (46 existing + 15 new parser tests)
• 78% average coverage on parsers
• 0 security vulnerabilities (CodeQL)

Documentation

• REFACTORING_GUIDE.md - Architecture overview, component usage, migration guide
• IMPROVEMENTS_SUMMARY.md - Executive summary with metrics

Next Steps

Integration of parsers into existing scrape_ualg.py and scrape_ualg_supabase.py scrapers to utilize enhanced extraction capabilities.

Original prompt

#codebase crie um plano de melhorias e refinamento do codigo base considerando minimamente os seguintes pontos:

expanda mais a quantidade de informaçao extraida, como por exempoo inserir o plano curricular e outros. refine a organizacao do codigo

[Chronological Review: The conversation began with the user requesting an analysis of the codebase to generate or update `.github/copilot-instructions.md`. The user emphasized the need for actionable instructions for AI coding agents, focusing on the architecture, developer workflows, project-specific conventions, and integration points. The user then requested an exploration and analysis of the project to update all documentation present in the codebase. Following this, the user initiated a semantic search to create a plan for improvements and refinements in the codebase, specifically to expand the information extracted and refine the organization of the code.]

[Intent Mapping:

1. "Analyze this codebase to generate or update .github/copilot-instructions.md for guiding AI coding agents." - The user wants to create or update documentation for AI agents.
2. "explore e analise o projeto e atualize toda documentacao presente no codigo base" - The user seeks a comprehensive analysis and update of all project documentation.
3. "#semantic_search crie um plano de melhorias e refinamento do codigo base" - The user is looking for a plan to improve and refine the codebase, focusing on expanding extracted information and refining code organization.]

[Technical Inventory:

• Repository: Scrape-UAlg-Courses
• Current Branch: feature/supabase
• Default Branch: main
• Tools: semantic_search, various Supabase commands for managing migrations, branches, and SQL execution.]

[Code Archaeology: No specific files or code changes were discussed in detail, but the focus was on documentation and code organization.]

[Progress Assessment: The user has initiated requests for documentation updates and codebase analysis but has not yet received specific outputs or completed tasks.]

[Context Validation: All critical information for continuation is captured, including user requests for documentation updates and codebase improvements.]

[Recent Commands Analysis:

• Last Agent Commands:

  1. "#semantic_search" - The user requested a semantic search to create a plan for codebase improvements.

• Tool Results Summary: No specific results were returned from the semantic search command yet, as it was just initiated.

• Pre-Summary State: The agent was actively working on generating a plan for improvements in the codebase when the token budget was exceeded.

• Operation Context: The recent commands were executed to align with the user's goals of enhancing the codebase's documentation and organization.]

1. Conversation Overview: - Primary Objectives: 1. "Analyze this codebase to generate or update `.github/copilot-instructions.md` for guiding AI coding agents." 2. "explore e analise o projeto e atualize toda documentacao presente no codigo base." 3. "#semantic_search crie um plano de melhorias e refinamento do codigo base." - Session Context: The conversation has focused on improving documentation and code organization within the Scrape-UAlg-Courses repository. - User Intent Evolution: The user initially sought to update AI documentation, then expanded the request to include a comprehensive analysis of the project and finally requested a specific plan for codebase improvements.

2. Technical Foundation:

• Repository: Scrape-UAlg-Courses
• Current Branch: feature/supabase
• Default Branch: main
• Tools: Various Supabase commands for managing migrations, branches, and SQL execution.

3. Codebase Status:

• No specific files or code changes were discussed in detail, but the focus was on documentation and code organization.

4. Problem Resolution:

• Issues Encountered: No specific technical problems were mentioned.
• Solutions Implemented: The user is in the process of generating a plan for improvements.
• Debugging Context: No ongoing troubleshooting efforts were noted.
• Lessons Learned: Insights into the need for better documentation and code organization.

5. Progress Tracking:

• Completed Tasks: None reported yet.
• Partially Complete Work: Documentation updates and codebase analysis are in progress.
• Validated Outcomes: No features or code confirmed working through testing yet.

6. Active Work State:

• Current Focus: The user is focused on generating a plan for improvements in the codebase.
• Recent Context: The last few exchanges involved requests for documentation updates and a semantic search for codebase improvements.
• Working Code: No specific code snippets were discussed recently.
• Immediate Context: The user was addressing the need for expanded information extraction and refined code organization.

7. Recent Operations:

• Last Agent Commands:
  1. "#semantic_search" - Initiated a semantic search for a plan to improve the codebas...

Created from VS Code via the GitHub Pull Request extension.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[marcelo-m7]

@copilot foi adicionado ao ambiente SUPABASE_ANON_KEY para acesso ao Supabase. use e altere somente o schema 'facodi' pois o banco de dados é partilhado. teste e configure corretamente a conexao com o projeto
SUPABASE_PROJECT_URL=https://pkjigvacvddcnlxhvvba.supabase.co