Skip to content

Bootstrap Python backend scraper project structure - #1

Merged
marcelo-m7 merged 3 commits into
mainfrom
copilot/update-python-app-workflow
Oct 27, 2025
Merged

marcelo-m7 merged 3 commits into
mainfrom
copilot/update-python-app-workflow

Conversation

Copilot AI commented Oct 27, 2025 •

Copy link
Copy Markdown

Initialize empty repository with complete Python project infrastructure for scraping University of Algarve course data.

Core Implementation

  • Scraper (src/scraper.py): UAlgScraper class with Beautiful Soup parsing, retry logic, and context manager support
  • Configuration (src/config.py): Environment-aware config with customizable timeouts, retries, and user agent
  • Tests (tests/test_scraper.py): 13 unit tests with mocked HTTP requests achieving 99% coverage

Project Infrastructure

  • GitHub Actions: CI pipeline with pytest and flake8
  • Templates: Bug reports, feature requests, and PR template
  • Documentation: README with usage examples, CONTRIBUTING guide, Code of Conduct, MIT license
  • Build tooling: Makefile for common tasks, requirements files, Python .gitignore

Usage

from src.scraper import UAlgScraper

with UAlgScraper() as scraper:
    courses = scraper.scrape_courses()
    for course in courses:
        print(f"{course['code']}: {course['name']}")

Custom configuration:

from src.config import Config

config = Config(timeout=60, max_retries=5)
scraper = UAlgScraper(config)

Notes

  • Logging configuration guarded to prevent interference with consuming applications
  • Chrome user agent set to v120 for modern compatibility
  • HTML parsing structure is placeholder—requires site-specific selectors
Original prompt

monynha-backend-scraper/
├── .github/
│ ├── workflows/
│ │ └── python-app.yml
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ └── PULL_REQUEST_TEMPLATE.md
├── src/
│ ├── init.py
│ ├── scraper.py
│ └── config.py
├── tests/
│ ├── init.py
│ └── test_scraper.py
├── .gitignore
├── README.md
├── LICENSE
├── CONTRIBUTING.md
├── CODE_OF_CONDUCT.md
├── requirements.txt
├── requirements-dev.txt
└── Makefile


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 2 commits October 27, 2025 08:18
Co-authored-by: marcelo-m7 <117441129+marcelo-m7@users.noreply.github.com>
…x Makefile

Co-authored-by: marcelo-m7 <117441129+marcelo-m7@users.noreply.github.com>
Copilot AI changed the title [WIP] Update Python app workflow configuration Bootstrap Python backend scraper project structure Oct 27, 2025
Copilot AI requested a review from marcelo-m7 October 27, 2025 08:25

@marcelo-m7 marcelo-m7 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good

@marcelo-m7
marcelo-m7 marked this pull request as ready for review October 27, 2025 08:28
@marcelo-m7
marcelo-m7 merged commit 303f3c1 into main Oct 27, 2025
2 checks passed
@marcelo-m7
marcelo-m7 deleted the copilot/update-python-app-workflow branch October 27, 2025 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants