Implement complete scraping, REST API, and web frontend for UAlg course catalog

[Copilot]

Builds a REST API and web platform to centralize Universidad do Algarve course catalog data, extracting courses, modules (UCs), knowledge areas, and documents from https://www.ualg.pt/oferta-formativa.

Scraper Enhancements

• Extract curriculum modules/UCs from tables and lists: code, title, ECTS, year, semester
• Extract and link knowledge areas (many-to-many with courses)
• Download course documents (PDFs) with metadata tracking
• Multiple CSS selector fallbacks for robust HTML parsing

# Enhanced parser extracts structured module data
course_data = scraper.parse_course_page(soup, url)
# Returns: {
#   'modules': [{'code': 'INF101', 'title': 'Programming I', 'ects': 6.0, 'year': 1, 'semester': 1}],
#   'areas': ['Computer Science', 'Engineering'],
#   'documents': [('Curriculum Plan', 'https://...')],
# }

REST API

• 8 endpoints: courses (with filters), course details, levels, schools, areas, stats
• Filtering by level, school, and knowledge area
• Returns courses with full relationships (modules, areas, documents)
• HTML frontend at /, JSON API info at /api

Web Frontend

• Responsive interface with interactive filters (level, school, area)
• Statistics dashboard with Chart.js visualizations
• Real-time course filtering and display
• Modern gradient design with hover effects

Demo System

• scripts/populate_demo_data.py creates 5 example courses with 16 modules
• make demo command for quick start
• Tests functional without live website access

Database

Normalized schema with many-to-many relationships:

• courses ↔ areas via course_area
• courses ↔ modules via module_course
• Document metadata in course_documents

All 36 tests passing. No security issues (CodeQL clean).

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• www.ualg.pt
  • Triggering command: python -m src.scrape_ualg (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Can you welp me to finush these inicial processes to make the project funcional?
I'd like you to analyse the proposal of the project and may check some open soucer github scrapper projects for inspirations, then create a very escructured plan to finislise the project. You have to starting by reding the proposal of the project (be a REST API for integrations with othes plataforms to acces the catalog of couresr of Universidade do Algarve - https://ualg.pt -, with the plus of download the courses documents, UC documentos - unidades curriculares). our main goal is also be ablel go obthain a json estrutured with the full couses plan course.
after, running the project and see the actuals erros, then plan the plan of finalisation based on the riquires your propose, the execuite it or assigne tasks for other agents.

Eu quero criar uma plataforma/banco de dados com sqlite e python que centralize em uma espécie de API todas as informações da grade curricular dos cursos da universidade do Algarve.

o projeto minimante deve ter:
Um mecanismo de scraping que busque, organize e armazene:

1. cursos separados por nível, código, título, descrição
2. documentação minina disponibilizada por curso + plano curricular completo tal como disponibilizado no site em formato cv's
3. dentre os nível, cursos separados por escola, área de conhecimento e matérias em comum
4. mecanismo frontend automatizado que permite vizualizar a informação com base na relação entre as "entidades"

O site ponto de partida do scraping é o seguinte: https://www.ualg.pt/oferta-formativa

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.