diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..c65d947 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,5 @@ +schema/node_modules +.git +.gitignore +*.md +dist \ No newline at end of file diff --git a/.gitignore b/.gitignore index 290d42a..a4ed832 100644 --- a/.gitignore +++ b/.gitignore @@ -4,12 +4,12 @@ *.dump *.gz *.log -*.sql *.tar *.zip # Exceptions !template.csv.zip +!schema/prisma/schema/migrations/*.sql # Directories and system files .DS_Store @@ -17,5 +17,8 @@ .idea .venv venv +node_modules - \ No newline at end of file +schema/prisma/generated +schema/prisma/node_modules +schema/prisma/.env \ No newline at end of file diff --git a/README.md b/README.md index 4ada386..4f4c57a 100644 --- a/README.md +++ b/README.md @@ -1,38 +1,17 @@ # InfoCompanies Data Model -This repository manages the **InfoCompanies** project's data model, and database operations. It provides a complete workflow for company data using a Dockerized PostgreSQL database and robust migration/versioning with Alembic. - ---- - -## πŸ—‚οΈ Repository Structure - -``` -. -β”œβ”€β”€ db.sh # Main orchestration script for DB setup and data loading -β”œβ”€β”€ docker-compose.yml # Docker services for PostgreSQL and PgAdmin -β”œβ”€β”€ requirements-dev.in # Python dependencies for DB scripts -β”œβ”€β”€ README.md # Main usage and feature documentation -β”œβ”€β”€ template.env # Example environment variables -β”œβ”€β”€ parsing/ # Python scripts for web scraping and data enrichment -β”œβ”€β”€ scripts/ # Shell and Python scripts for data loading, backup, export, etc. -β”œβ”€β”€ schema/ # Database schema, Alembic migrations, and SQLAlchemy models -β”‚ β”œβ”€β”€ alembic/ # Alembic migration scripts and config -β”‚ └── app/ # SQLAlchemy models and DB initialization -β”œβ”€β”€ config/ # PgAdmin configuration -β”œβ”€β”€ docs/ # Additional documentation (e.g., autocomplete guide) -└── .github/ # CI/CD workflows -``` - ---- +This repository manages the **InfoCompanies** project's data model and database operations. It provides a complete workflow for company data using a Dockerized PostgreSQL database and modern schema management with Prisma. ## πŸš€ Main Features - **Dockerized PostgreSQL**: Easy local setup with persistent volumes and PgAdmin UI. - **Data Enrichment**: Python scripts for scraping and loading company data. -- **Database Schema Management**: SQLAlchemy models and Alembic migrations for versioned schema evolution. +- **Modern Schema Management**: Prisma ORM with type-safe database access and robust migrations. - **Automated Data Loading**: Bash scripts to orchestrate pulling, unzipping, and importing CSVs into the database. - **Backup & Restore**: Tools for SQL/CSV backup and restore, including gzip support. - **Autocomplete Support**: Extraction and indexing of unique values for fast autocomplete APIs. +- **Database GUI**: Built-in Prisma Studio for visual database exploration and management. +- **Type Safety**: Auto-generated, fully typed database client for multiple languages. - **CI/CD**: GitHub Actions for linting, formatting, and build validation. --- @@ -54,8 +33,13 @@ Copy and edit `.env` from `template.env`: cp template.env .env ``` -### 3. Install Python Dependencies +### 3. Install Dependencies +#### For Prisma (Database ORM) + +See [schema/README.md](schema/README.md) for detailed Prisma setup. + +#### For Python Scripts (Data Processing) ```bash python3 -m venv .venv source .venv/bin/activate @@ -70,26 +54,48 @@ docker compose up -d ### 5. Initialize Database & Load Data -Run the main orchestration script: +#### Set up Prisma and generate client +```bash +cd schema +prisma generate +prisma db push # Push schema to database +``` +#### Run the main orchestration script ```bash ./db.sh ``` This will: - Start Docker containers -- Run Alembic migrations +- Set up database schema with Prisma - Load CSVs from the ETL - Shut down containers +#### Optional: Open Prisma Studio (Database GUI) +```bash +cd schema +prisma studio +``` + --- ## 🧩 Key Components ### Database Schema -- Defined in [schema/app/models/](schema/app/models/) -- Managed and versioned with Alembic ([schema/alembic/](schema/alembic/)) +- **Prisma Schema**: Modern database schema defined in [schema/prisma/schema.prisma](schema/prisma/schema.prisma) +- **Model Documentation**: Individual model references in [schema/prisma/models/](schema/prisma/models/) +- **Type-Safe Client**: Auto-generated Prisma Client for database operations +- **Migration System**: Robust schema versioning and migration management + +### Database Models + +- **Company**: Comprehensive business data with financial information (2018-2023) +- **Leader**: Company leadership and management information +- **Autocomplete Models**: City, Industry Sector, Legal Form, Region reference data +- **User Management**: User quotas and company interaction tracking +- **Configuration**: System settings and configuration data ### Parsing @@ -98,9 +104,18 @@ This will: ### Data Operations +- **Database Management**: Prisma CLI commands for schema and data management - **Backup/Restore**: [scripts/backup.sh](scripts/backup.sh) - **CSV Transfer**: [scripts/util.sh](scripts/util.sh) - **Data Loading**: [scripts/load-csv-to-database.sh](scripts/load-csv-to-database.sh) +- **Database GUI**: Prisma Studio for visual data exploration and editing + +### Development Tools + +- **Prisma Studio**: Visual database browser and editor (`prisma studio`) +- **Type Generation**: Auto-generated type-safe database client +- **Schema Validation**: Built-in schema validation and error checking +- **Migration Management**: Version-controlled database schema evolution ### CI/CD @@ -111,13 +126,37 @@ This will: ## πŸ“š Documentation - [README.md](README.md): Main usage and features -- [schema/README.md](schema/README.md): Alembic and schema management +- [schema/README.md](schema/README.md): Prisma schema management and migration guide +- [schema/prisma/models/README.md](schema/prisma/models/README.md): Database models overview - [docs/AUTOCOMPLETE.md](docs/AUTOCOMPLETE.md): How to add autocomplete support ---- +## πŸš€ Quick Start Commands + +```bash +# Start database services +docker compose up -d + +# Generate Prisma client (after schema changes) +cd schema && prisma generate + +# Push schema to database (development) +cd schema && prisma db push + +# Create and apply migrations (production) +cd schema && prisma migrate dev --name "your_migration_name" + +# Open database GUI +cd schema && prisma studio + +# Load data +./db.sh +``` ## πŸ“ Notes - All scripts assume a Unix-like environment and require Docker. - Data files (`.csv`, `.dump`, etc.) are git-ignored by default. - For troubleshooting, check logs in the output pane or use `docker logs`. +- **Prisma Client**: Generated client is located in `schema/generated/prisma/` +- **Environment**: Ensure `DATABASE_URL` is properly configured in your `.env` file +- **Development**: Use `prisma db push` for quick schema changes, `prisma migrate dev` for production-ready migrations diff --git a/dev.Dockerfile b/dev.Dockerfile index 2ee056f..22a0a14 100644 --- a/dev.Dockerfile +++ b/dev.Dockerfile @@ -8,13 +8,15 @@ ENV POSTGRES_PASSWORD=root ENV POSTGRES_DB=postgres ENV PGDATA=/var/lib/postgresql/data -# Install Python & dependencies for Alembic +# Install Node.js and Python RUN apt-get update && apt-get install -y --no-install-recommends \ - python3 python3-venv python3-pip postgresql-client \ + curl \ + && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \ + && apt-get install -y nodejs npm postgresql-client python3 python3-pip python3-venv \ && python3 -m venv /opt/venv \ && rm -rf /var/lib/apt/lists/* -ENV PATH="/opt/venv/bin:$PATH" + ENV PATH="/opt/venv/bin:$PATH" # Copy requirements and install COPY requirements/build.in /tmp/requirements.in @@ -27,8 +29,15 @@ COPY scripts ./scripts COPY scripts/setup-db.sh /app/setup-db.sh COPY final.csv leaders.csv fichier_combine_updated_big_fixed.csv ./ +# Install dependencies +RUN npm install -g --no-fund pnpm@10.10.0 +WORKDIR /app/schema +RUN pnpm install --frozen-lockfile +WORKDIR /app + # Make ./data writable by postgres user RUN mkdir -p /app/data && chown -R postgres:postgres /app/data +RUN chown -R postgres:postgres /app/schema/ # Start Postgres for migrations and CSV loading USER postgres diff --git a/prod.Dockerfile b/prod.Dockerfile index 4d87a0e..3ba2c0c 100644 --- a/prod.Dockerfile +++ b/prod.Dockerfile @@ -1,22 +1,20 @@ -FROM python:3.11-slim AS migrations +FROM node:22-slim AS migrations -# Install Postgres client (optional: for raw SQL commands or debugging) -RUN apt-get update && apt-get install -y --no-install-recommends \ - && rm -rf /var/lib/apt/lists/* +WORKDIR /app + +# Enable pnpm via corepack -# Create virtualenv -RUN python -m venv /opt/venv -ENV PATH="/opt/venv/bin:$PATH" +# Copy only package files first +COPY schema/package.json schema/pnpm-lock.yaml ./ # Install dependencies -COPY requirements/build.in /tmp/requirements.in -RUN pip install --no-cache-dir -r /tmp/requirements.in +RUN npm install -g --no-fund pnpm@10.10.0 \ + pnpm install --frozen-lockfile -# Copy app/migrations code -WORKDIR /app -COPY schema ./schema -COPY scripts ./scripts +# Now copy the rest of the schema code +COPY schema . + +# Prisma checks +RUN pnpm exec prisma generate -# Default command runs alembic inside schema folder -WORKDIR /app/schema -ENTRYPOINT ["alembic", "upgrade", "head"] +ENTRYPOINT ["pnpm", "exec", "prisma", "db", "push"] diff --git a/requirements/build.in b/requirements/build.in index c6d5c17..0f792c2 100644 --- a/requirements/build.in +++ b/requirements/build.in @@ -1,3 +1 @@ -alembic~=1.15.2 -psycopg2-binary~=2.9.10 -sqlalchemy~=2.0.41 \ No newline at end of file +psycopg2-binary~=2.9.10 \ No newline at end of file diff --git a/schema/README.md b/schema/README.md index be28370..3d1f660 100644 --- a/schema/README.md +++ b/schema/README.md @@ -1,83 +1,148 @@ +# πŸ“¦ Database Schema Management with Prisma - -# πŸ“¦ Database Migrations with Alembic - -This project uses **Alembic** for managing SQLAlchemy schema migrations. +This project uses **Prisma** for database schema management, type-safe database access, and migrations. --- -## πŸ“ Directory structure +## πŸ› οΈ Setup + +1. **Install Prisma CLI**: -``` -InfoCompanies-Data-Model/ -β”œβ”€β”€ app/ -β”‚ β”œβ”€β”€ models/ -β”‚ β”‚ └── company.py # Your model(s) -β”œβ”€β”€ schema/ -β”‚ └── alembic/ # Alembic config and migrations -β”‚ └── env.py # Configure metadata here -β”‚ └── database.py # Init the database +```bash +npm install -g prisma +# or using your preferred package manager ``` ---- +2. **Install dependencies**: -## πŸ› οΈ Setup +```bash +cd schema +pnpm install +``` -1. **Install dependencies**: +3. **Set up environment variables**: + Make sure your `DATABASE_URL` is properly configured in your environment: ```bash -pip install alembic psycopg2-binary +export DATABASE_URL="postgresql://username:password@localhost:5432/your_database" ``` -2. **Initialize Alembic** (only needed once): +4. **Generate Prisma Client**: ```bash cd schema -alembic upgrade head -PYTHONPATH=. python3 app/database/database.py +prisma generate ``` -3. **Edit `alembic/env.py`** to link Alembic to your models: +--- -```python -# At the top of env.py -import sys -import os +## πŸš€ Common Commands -sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))) -from app.db.database import Base -from app.models import company # force model import +### Generate Prisma Client (after schema changes) -# Then set: -target_metadata = Base.metadata +```bash +cd schema +prisma generate ``` ---- +### Create and apply migrations -## πŸš€ Common Commands +```bash +cd schema +# Create a new migration +prisma migrate dev --name "description_of_changes" + +# Apply migrations to production +prisma migrate deploy +``` + +### Database introspection (sync schema with existing DB) + +```bash +cd schema +prisma db pull +``` -### Create a new migration (auto-generate) +### Push schema changes without migrations (for development) ```bash cd schema -alembic revision --autogenerate -m "add company model" +prisma db push ``` -### Apply migrations (upgrade database) +### Reset database (careful in production!) ```bash -alembic upgrade head +cd schema +prisma migrate reset ``` -### Downgrade (undo last migration) +### Open Prisma Studio (database GUI) ```bash -alembic downgrade -1 +cd schema +prisma studio ``` --- -## 🧠 Tips +## πŸ“Š Database Models + +This schema includes the following models: + +### Core Business Models +- **Company** - Comprehensive company data with financial information (2018-2023) +- **Leader** - Company leadership and management information + +### Reference Data Models +- **City** - City reference data for autocomplete +- **IndustrySector** - Industry sector classifications +- **LegalForm** - Legal form types +- **Region** - Geographic regions + +### User Management Models +- **UserCompanyStatus** - User interaction tracking with companies +- **UserQuota** - User quota management system + +### Configuration Models +- **Config** - System configuration settings + +--- + +## 🧠 Migration from Alembic + +This project has been migrated from SQLAlchemy + Alembic to Prisma. Key benefits: + +- **Type Safety**: Auto-generated, fully typed database client +- **Better DX**: Intuitive query API and excellent IntelliSense +- **Database GUI**: Built-in Prisma Studio for data exploration +- **Migration Management**: Robust migration system with rollback support +- **Multi-language Support**: Works with JavaScript, TypeScript, Python, and more +- **I don't like Python**: I understand Python but I don't like it and I prefer TypeScript + +### Key Changes: +- All SQLAlchemy models converted to Prisma schema +- Preserved all existing indexes and constraints +- Maintained database compatibility (same table names and structure) +- Enhanced with proper relations between models + +--- + +## 🧭 Tips + +- **Always run `prisma generate`** after modifying the schema file +- **Use `prisma migrate dev`** during development for schema changes +- **Use `prisma migrate deploy`** for production deployments +- **Backup your database** before running `prisma migrate reset` +- **Check the generated client** in `./generated/prisma` for available methods +- **Use Prisma Studio** (`prisma studio`) for easy data visualization and editing +- **Make sure the correct `DATABASE_URL` is set** in your environment variables + +--- + +## πŸ“š Useful Resources -- Always **import all models** before running `alembic revision --autogenerate`, or Alembic won’t detect them. -- Make sure the correct `DATABASE_URL` is set in your `database.py`. \ No newline at end of file +- [Prisma Documentation](https://www.prisma.io/docs) +- [Prisma Schema Reference](https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference) +- [Prisma Client API Reference](https://www.prisma.io/docs/reference/api-reference/prisma-client-reference) +- [Migration Guide](https://www.prisma.io/docs/guides/migrate-to-prisma) diff --git a/schema/alembic.ini b/schema/alembic.ini deleted file mode 100644 index 441fdd3..0000000 --- a/schema/alembic.ini +++ /dev/null @@ -1,118 +0,0 @@ -# A generic, single database configuration. - -[alembic] -# path to migration scripts -# Use forward slashes (/) also on windows to provide an os agnostic path -script_location = alembic - -# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s -# Uncomment the line below if you want the files to be prepended with date and time -# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file -# for all available tokens -# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s - -# sys.path path, will be prepended to sys.path if present. -# defaults to the current working directory. -prepend_sys_path = . - -# timezone to use when rendering the date within the migration file -# as well as the filename. -# If specified, requires the python>=3.9 or backports.zoneinfo library and tzdata library. -# Any required deps can installed by adding `alembic[tz]` to the pip requirements -# string value is passed to ZoneInfo() -# leave blank for localtime -# timezone = - -# max length of characters to apply to the "slug" field -# truncate_slug_length = 40 - -# set to 'true' to run the environment during -# the 'revision' command, regardless of autogenerate -# revision_environment = false - -# set to 'true' to allow .pyc and .pyo files without -# a source .py file to be detected as revisions in the -# versions/ directory -# sourceless = false - -# version location specification; This defaults -# to alembic/versions. When using multiple version -# directories, initial revisions must be specified with --version-path. -# The path separator used here should be the separator specified by "version_path_separator" below. -# version_locations = %(here)s/bar:%(here)s/bat:alembic/versions - -# version path separator; As mentioned above, this is the character used to split -# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep. -# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas. -# Valid values for version_path_separator are: -# -# version_path_separator = : -# version_path_separator = ; -# version_path_separator = space -# version_path_separator = newline -# -# Use os.pathsep. Default configuration used for new projects. -version_path_separator = os - -# set to 'true' to search source files recursively -# in each "version_locations" directory -# new in Alembic version 1.10 -# recursive_version_locations = false - -# the output encoding used when revision files -# are written from script.py.mako -# output_encoding = utf-8 - -sqlalchemy.url = - -[post_write_hooks] -# post_write_hooks defines scripts or Python functions that are run -# on newly generated revision scripts. See the documentation for further -# detail and examples - -# format using "black" - use the console_scripts runner, against the "black" entrypoint -# hooks = black -# black.type = console_scripts -# black.entrypoint = black -# black.options = -l 79 REVISION_SCRIPT_FILENAME - -# lint with attempts to fix using "ruff" - use the exec runner, execute a binary -# hooks = ruff -# ruff.type = exec -# ruff.executable = %(here)s/.venv/bin/ruff -# ruff.options = check --fix REVISION_SCRIPT_FILENAME - -# Logging configuration -[loggers] -keys = root,sqlalchemy,alembic - -[handlers] -keys = console - -[formatters] -keys = generic - -[logger_root] -level = WARNING -handlers = console -qualname = - -[logger_sqlalchemy] -level = WARNING -handlers = -qualname = sqlalchemy.engine - -[logger_alembic] -level = INFO -handlers = -qualname = alembic - -[handler_console] -class = StreamHandler -args = (sys.stderr,) -level = NOTSET -formatter = generic - -[formatter_generic] -format = %(levelname)-5.5s [%(name)s] %(message)s -datefmt = %H:%M:%S diff --git a/schema/alembic/__pycache__/env.cpython-312.pyc b/schema/alembic/__pycache__/env.cpython-312.pyc deleted file mode 100644 index 74ed6e5..0000000 Binary files a/schema/alembic/__pycache__/env.cpython-312.pyc and /dev/null differ diff --git a/schema/alembic/env.py b/schema/alembic/env.py deleted file mode 100644 index 20a46d2..0000000 --- a/schema/alembic/env.py +++ /dev/null @@ -1,90 +0,0 @@ -from logging.config import fileConfig -import os - -from alembic import context -from app.models.autocomplete import Base as AutocompleteBase -from app.models.companies import Base as CompaniesBase -from app.models.config import Base as ConfigBase -from app.models.leaders import Base as LeadersBase -from app.models.user_company_status import Base as UserCompanyStatusBase -from app.models.user_quota import Base as UserQuotaBase -from sqlalchemy import engine_from_config, pool - -# this is the Alembic Config object, which provides -# access to the values within the .ini file in use. -config = context.config - -# Interpret the config file for Python logging. -# This line sets up loggers basically. -if config.config_file_name is not None: - fileConfig(config.config_file_name) - -database_url = os.getenv("DATABASE_URL") -if database_url: - config.set_main_option("sqlalchemy.url", database_url) - -# add your model's MetaData object here -target_metadata = [ - CompaniesBase.metadata, - LeadersBase.metadata, - ConfigBase.metadata, - AutocompleteBase.metadata, - UserQuotaBase.metadata, - UserCompanyStatusBase.metadata, -] - -# other values from the config, defined by the needs of env.py, -# can be acquired: -# my_important_option = config.get_main_option("my_important_option") -# ... etc. - - -def run_migrations_offline() -> None: - """Run migrations in 'offline' mode. - - This configures the context with just a URL - and not an Engine, though an Engine is acceptable - here as well. By skipping the Engine creation - we don't even need a DBAPI to be available. - - Calls to context.execute() here emit the given string to the - script output. - - """ - url = config.get_main_option("sqlalchemy.url") - context.configure( - url=url, - target_metadata=target_metadata, - literal_binds=True, - dialect_opts={"paramstyle": "named"}, - ) - - with context.begin_transaction(): - context.run_migrations() - - -def run_migrations_online() -> None: - """Run migrations in 'online' mode. - - In this scenario we need to create an Engine - and associate a connection with the context. - - """ - connectable = engine_from_config( - config.get_section(config.config_ini_section, {}), - prefix="sqlalchemy.", - poolclass=pool.NullPool, - ) - - with connectable.connect() as connection: - - context.configure(connection=connection, target_metadata=target_metadata) - - with context.begin_transaction(): - context.run_migrations() - - -if context.is_offline_mode(): - run_migrations_offline() -else: - run_migrations_online() diff --git a/schema/alembic/script.py.mako b/schema/alembic/script.py.mako deleted file mode 100644 index 480b130..0000000 --- a/schema/alembic/script.py.mako +++ /dev/null @@ -1,28 +0,0 @@ -"""${message} - -Revision ID: ${up_revision} -Revises: ${down_revision | comma,n} -Create Date: ${create_date} - -""" -from typing import Sequence, Union - -from alembic import op -import sqlalchemy as sa -${imports if imports else ""} - -# revision identifiers, used by Alembic. -revision: str = ${repr(up_revision)} -down_revision: Union[str, None] = ${repr(down_revision)} -branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)} -depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)} - - -def upgrade() -> None: - """Upgrade schema.""" - ${upgrades if upgrades else "pass"} - - -def downgrade() -> None: - """Downgrade schema.""" - ${downgrades if downgrades else "pass"} diff --git a/schema/alembic/versions/57a5a84e79d0_init_db.py b/schema/alembic/versions/57a5a84e79d0_init_db.py deleted file mode 100644 index d052a9f..0000000 --- a/schema/alembic/versions/57a5a84e79d0_init_db.py +++ /dev/null @@ -1,374 +0,0 @@ -"""init db - -Revision ID: 57a5a84e79d0 -Revises: -Create Date: 2025-05-11 21:29:06.925736 - -""" - -from typing import Sequence, Union - -import sqlalchemy as sa -from alembic import op -from sqlalchemy.dialects import postgresql - -# revision identifiers, used by Alembic. -revision: str = "57a5a84e79d0" -down_revision: Union[str, None] = None -branch_labels: Union[str, Sequence[str], None] = None -depends_on: Union[str, Sequence[str], None] = None - - -def upgrade() -> None: - """Upgrade schema.""" - # ### commands auto generated by Alembic - please adjust! ### - op.create_table( - "companies", - sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), - sa.Column("company_name", sa.String(), nullable=True), - sa.Column("siren_number", sa.String(), nullable=True), - sa.Column("nic_number", sa.String(), nullable=True), - sa.Column("legal_form", sa.String(), nullable=True), - sa.Column("ape_code", sa.String(), nullable=True), - sa.Column("ape_label", sa.String(), nullable=True), - sa.Column("address", sa.String(), nullable=True), - sa.Column("postal_code", sa.String(), nullable=True), - sa.Column("department_number", sa.String(), nullable=True), - sa.Column("department", sa.String(), nullable=True), - sa.Column("city", sa.String(), nullable=True), - sa.Column("region", sa.String(), nullable=True), - sa.Column("trade_name", sa.String(), nullable=True), - sa.Column("registration_date", sa.Date(), nullable=True), - sa.Column("deregistration_date", sa.Date(), nullable=True), - sa.Column("closing_date_2018_1", sa.Date(), nullable=True), - sa.Column("revenue_2018_1", sa.Float(), nullable=True), - sa.Column("turnover_2018_1", sa.Float(), nullable=True), - sa.Column("closing_date_2018_2", sa.Date(), nullable=True), - sa.Column("revenue_2018_2", sa.Float(), nullable=True), - sa.Column("turnover_2018_2", sa.Float(), nullable=True), - sa.Column("closing_date_2018_3", sa.Date(), nullable=True), - sa.Column("revenue_2018_3", sa.Float(), nullable=True), - sa.Column("turnover_2018_3", sa.Float(), nullable=True), - sa.Column("closing_date_2019_1", sa.Date(), nullable=True), - sa.Column("revenue_2019_1", sa.Float(), nullable=True), - sa.Column("turnover_2019_1", sa.Float(), nullable=True), - sa.Column("closing_date_2019_2", sa.Date(), nullable=True), - sa.Column("revenue_2019_2", sa.Float(), nullable=True), - sa.Column("turnover_2019_2", sa.Float(), nullable=True), - sa.Column("closing_date_2019_3", sa.Date(), nullable=True), - sa.Column("revenue_2019_3", sa.Float(), nullable=True), - sa.Column("turnover_2019_3", sa.Float(), nullable=True), - sa.Column("closing_date_2020_1", sa.Date(), nullable=True), - sa.Column("revenue_2020_1", sa.Float(), nullable=True), - sa.Column("turnover_2020_1", sa.Float(), nullable=True), - sa.Column("closing_date_2020_2", sa.Date(), nullable=True), - sa.Column("revenue_2020_2", sa.Float(), nullable=True), - sa.Column("turnover_2020_2", sa.Float(), nullable=True), - sa.Column("closing_date_2020_3", sa.Date(), nullable=True), - sa.Column("revenue_2020_3", sa.Float(), nullable=True), - sa.Column("turnover_2020_3", sa.Float(), nullable=True), - sa.Column("closing_date_2021_1", sa.Date(), nullable=True), - sa.Column("revenue_2021_1", sa.Float(), nullable=True), - sa.Column("turnover_2021_1", sa.Float(), nullable=True), - sa.Column("closing_date_2021_2", sa.Date(), nullable=True), - sa.Column("revenue_2021_2", sa.Float(), nullable=True), - sa.Column("turnover_2021_2", sa.Float(), nullable=True), - sa.Column("closing_date_2021_3", sa.Date(), nullable=True), - sa.Column("revenue_2021_3", sa.Float(), nullable=True), - sa.Column("turnover_2021_3", sa.Float(), nullable=True), - sa.Column("closing_date_2022_1", sa.Date(), nullable=True), - sa.Column("revenue_2022_1", sa.Float(), nullable=True), - sa.Column("turnover_2022_1", sa.Float(), nullable=True), - sa.Column("closing_date_2022_2", sa.Date(), nullable=True), - sa.Column("revenue_2022_2", sa.Float(), nullable=True), - sa.Column("turnover_2022_2", sa.Float(), nullable=True), - sa.Column("closing_date_2022_3", sa.Date(), nullable=True), - sa.Column("revenue_2022_3", sa.Float(), nullable=True), - sa.Column("turnover_2022_3", sa.Float(), nullable=True), - sa.Column("closing_date_2023_1", sa.Date(), nullable=True), - sa.Column("revenue_2023_1", sa.Float(), nullable=True), - sa.Column("turnover_2023_1", sa.Float(), nullable=True), - sa.Column("closing_date_2023_2", sa.Date(), nullable=True), - sa.Column("revenue_2023_2", sa.Float(), nullable=True), - sa.Column("turnover_2023_2", sa.Float(), nullable=True), - sa.Column("closing_date_2023_3", sa.Date(), nullable=True), - sa.Column("revenue_2023_3", sa.Float(), nullable=True), - sa.Column("turnover_2023_3", sa.Float(), nullable=True), - sa.Column("industry_sector", sa.String(), nullable=True), - sa.Column("phone_number", sa.String(), nullable=True), - sa.Column("website", sa.String(length=3000), nullable=True), - sa.Column("reviews", postgresql.JSONB(astext_type=sa.Text()), nullable=True), - sa.Column("schedule", postgresql.JSONB(astext_type=sa.Text()), nullable=True), - sa.Column("instagram", sa.String(length=10000), nullable=True), - sa.Column("facebook", sa.String(length=3000), nullable=True), - sa.Column("twitter", sa.String(length=3000), nullable=True), - sa.Column("linkedin", sa.String(length=3000), nullable=True), - sa.Column("youtube", sa.String(length=3000), nullable=True), - sa.Column("email", sa.String(length=3000), nullable=True), - sa.Column("scraping_date", sa.Date(), nullable=True), - sa.Column("date_creation", sa.Date(), nullable=True), - sa.Column("last_processing_date", sa.Date(), nullable=True), - sa.Column("number_of_employee", sa.Integer(), nullable=True), - sa.Column("company_category", sa.String(), nullable=True), - sa.PrimaryKeyConstraint("id"), - ) - op.create_index("ix_companies_city", "companies", ["city"], unique=False) - op.create_index( - "ix_companies_city_industry_sector", - "companies", - ["city", "industry_sector"], - unique=False, - ) - op.create_index( - "ix_companies_city_industry_sector_legal_form", - "companies", - ["city", "industry_sector", "legal_form"], - unique=False, - ) - op.create_index( - "ix_companies_city_legal_form", - "companies", - ["city", "legal_form"], - unique=False, - ) - op.create_index( - "ix_companies_company_name", "companies", ["company_name"], unique=False - ) - op.create_index("ix_companies_email", "companies", ["email"], unique=False) - op.create_index("ix_companies_facebook", "companies", ["facebook"], unique=False) - op.create_index( - "ix_companies_industry_sector", "companies", ["industry_sector"], unique=False - ) - op.create_index( - "ix_companies_industry_sector_legal_form", - "companies", - ["industry_sector", "legal_form"], - unique=False, - ) - op.create_index( - "ix_companies_industry_sector_number_of_employee", - "companies", - ["industry_sector", "number_of_employee"], - unique=False, - ) - op.create_index("ix_companies_instagram", "companies", ["instagram"], unique=False) - op.create_index( - "ix_companies_legal_form", "companies", ["legal_form"], unique=False - ) - op.create_index("ix_companies_linkedin", "companies", ["linkedin"], unique=False) - op.create_index( - "ix_companies_number_of_employee", - "companies", - ["number_of_employee"], - unique=False, - ) - op.create_index( - "ix_companies_phone_number", "companies", ["phone_number"], unique=False - ) - op.create_index("ix_companies_region", "companies", ["region"], unique=False) - op.create_index( - "ix_companies_region_city", "companies", ["region", "city"], unique=False - ) - op.create_index( - "ix_companies_region_city_industry_sector", - "companies", - ["region", "city", "industry_sector"], - unique=False, - ) - op.create_index( - "ix_companies_region_city_industry_sector_legal_form", - "companies", - ["region", "city", "industry_sector", "legal_form"], - unique=False, - ) - op.create_index( - "ix_companies_region_city_legal_form", - "companies", - ["region", "city", "legal_form"], - unique=False, - ) - op.create_index( - "ix_companies_region_industry_sector", - "companies", - ["region", "industry_sector"], - unique=False, - ) - op.create_index( - "ix_companies_region_industry_sector_legal_form", - "companies", - ["region", "industry_sector", "legal_form"], - unique=False, - ) - op.create_index( - "ix_companies_region_legal_form", - "companies", - ["region", "legal_form"], - unique=False, - ) - op.create_index( - "ix_companies_siren_number", "companies", ["siren_number"], unique=False - ) - op.create_index("ix_companies_twitter", "companies", ["twitter"], unique=False) - op.create_index("ix_companies_website", "companies", ["website"], unique=False) - op.create_index("ix_companies_youtube", "companies", ["youtube"], unique=False) - op.create_table( - "leaders", - sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), - sa.Column("siren", sa.String(), nullable=True), - sa.Column("role", sa.String(), nullable=True), - sa.Column("last_name", sa.String(), nullable=True), - sa.Column("first_name", sa.String(), nullable=True), - sa.Column("gestion_number", sa.String(), nullable=True), - sa.Column("type", sa.String(), nullable=True), - sa.Column("event_name", sa.String(), nullable=True), - sa.Column("usage_name", sa.String(), nullable=True), - sa.Column("pseudo", sa.String(), nullable=True), - sa.Column("company_name", sa.String(length=3000), nullable=True), - sa.Column("legal_form", sa.String(), nullable=True), - sa.Column("id_data", sa.String(), nullable=True), - sa.PrimaryKeyConstraint("id"), - ) - op.create_index( - "idx_leader_company_name", "leaders", ["company_name"], unique=False - ) - op.create_index("idx_leader_first_name", "leaders", ["first_name"], unique=False) - op.create_index("idx_leader_last_name", "leaders", ["last_name"], unique=False) - op.create_index("idx_leader_role", "leaders", ["role"], unique=False) - op.create_index("idx_leader_siren", "leaders", ["siren"], unique=False) - op.create_table( - "config", - sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), - sa.Column("last_reset_quota_date", sa.Date(), nullable=True), - sa.PrimaryKeyConstraint("id"), - ) - op.create_table( - "city", - sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), - sa.Column("name", sa.String(), nullable=True), - sa.PrimaryKeyConstraint("id"), - ) - op.create_index("ix_city_name", "city", ["name"], unique=False) - op.create_table( - "industry_sector", - sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), - sa.Column("name", sa.String(), nullable=True), - sa.PrimaryKeyConstraint("id"), - ) - op.create_index( - "ix_industry_sector_name", "industry_sector", ["name"], unique=False - ) - op.create_table( - "legal_form", - sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), - sa.Column("name", sa.String(), nullable=True), - sa.PrimaryKeyConstraint("id"), - ) - op.create_index("ix_legal_form_name", "legal_form", ["name"], unique=False) - op.create_table( - "region", - sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), - sa.Column("name", sa.String(), nullable=True), - sa.PrimaryKeyConstraint("id"), - ) - op.create_index("ix_region_name", "region", ["name"], unique=False) - op.create_table( - "user_quota", - sa.Column("user_id", sa.String(), nullable=False), - sa.Column("quota_allocated", sa.Integer(), nullable=True), - sa.Column("quota_used", sa.Integer(), nullable=True), - sa.PrimaryKeyConstraint("user_id"), - ) - op.create_index( - "ix_user_quota_quota_allocated", "user_quota", ["quota_allocated"], unique=False - ) - op.create_index( - "ix_user_quota_quota_used", "user_quota", ["quota_used"], unique=False - ) - op.create_index("ix_user_quota_user_id", "user_quota", ["user_id"], unique=False) - op.create_table( - "user_company_status", - sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), - sa.Column("user_id", sa.String(), nullable=True), - sa.Column( - "status", sa.Enum("NOT_DONE", "TO_DO", "DONE", name="status"), nullable=True - ), - sa.Column("company_id", sa.Integer(), nullable=True), - sa.PrimaryKeyConstraint("id"), - ) - op.create_index( - "ix_user_company_status_company_id", - "user_company_status", - ["company_id"], - unique=False, - ) - op.create_index( - "ix_user_company_status_user_id", - "user_company_status", - ["user_id"], - unique=False, - ) - # ### end Alembic commands ### - - -def downgrade() -> None: - """Downgrade schema.""" - # ### commands auto generated by Alembic - please adjust! ### - op.drop_index("ix_user_company_status_user_id", table_name="user_company_status") - op.drop_index("ix_user_company_status_company_id", table_name="user_company_status") - op.drop_table("user_company_status") - op.drop_index("ix_user_quota_user_id", table_name="user_quota") - op.drop_index("ix_user_quota_quota_used", table_name="user_quota") - op.drop_index("ix_user_quota_quota_allocated", table_name="user_quota") - op.drop_table("user_quota") - op.drop_index("ix_region_name", table_name="region") - op.drop_table("region") - op.drop_index("ix_legal_form_name", table_name="legal_form") - op.drop_table("legal_form") - op.drop_index("ix_industry_sector_name", table_name="industry_sector") - op.drop_table("industry_sector") - op.drop_index("ix_city_name", table_name="city") - op.drop_table("city") - op.drop_table("config") - op.drop_index("idx_leader_siren", table_name="leaders") - op.drop_index("idx_leader_role", table_name="leaders") - op.drop_index("idx_leader_last_name", table_name="leaders") - op.drop_index("idx_leader_first_name", table_name="leaders") - op.drop_index("idx_leader_company_name", table_name="leaders") - op.drop_table("leaders") - op.drop_index("ix_companies_youtube", table_name="companies") - op.drop_index("ix_companies_website", table_name="companies") - op.drop_index("ix_companies_twitter", table_name="companies") - op.drop_index("ix_companies_siren_number", table_name="companies") - op.drop_index("ix_companies_region_legal_form", table_name="companies") - op.drop_index( - "ix_companies_region_industry_sector_legal_form", table_name="companies" - ) - op.drop_index("ix_companies_region_industry_sector", table_name="companies") - op.drop_index("ix_companies_region_city_legal_form", table_name="companies") - op.drop_index( - "ix_companies_region_city_industry_sector_legal_form", table_name="companies" - ) - op.drop_index("ix_companies_region_city_industry_sector", table_name="companies") - op.drop_index("ix_companies_region_city", table_name="companies") - op.drop_index("ix_companies_region", table_name="companies") - op.drop_index("ix_companies_phone_number", table_name="companies") - op.drop_index("ix_companies_number_of_employee", table_name="companies") - op.drop_index("ix_companies_linkedin", table_name="companies") - op.drop_index("ix_companies_legal_form", table_name="companies") - op.drop_index("ix_companies_instagram", table_name="companies") - op.drop_index( - "ix_companies_industry_sector_number_of_employee", table_name="companies" - ) - op.drop_index("ix_companies_industry_sector_legal_form", table_name="companies") - op.drop_index("ix_companies_industry_sector", table_name="companies") - op.drop_index("ix_companies_facebook", table_name="companies") - op.drop_index("ix_companies_email", table_name="companies") - op.drop_index("ix_companies_company_name", table_name="companies") - op.drop_index("ix_companies_city_legal_form", table_name="companies") - op.drop_index( - "ix_companies_city_industry_sector_legal_form", table_name="companies" - ) - op.drop_index("ix_companies_city_industry_sector", table_name="companies") - op.drop_index("ix_companies_city", table_name="companies") - op.drop_table("companies") - # ### end Alembic commands ### diff --git a/schema/alembic/versions/__pycache__/57a5a84e79d0_init_db.cpython-312.pyc b/schema/alembic/versions/__pycache__/57a5a84e79d0_init_db.cpython-312.pyc deleted file mode 100644 index 0f1c799..0000000 Binary files a/schema/alembic/versions/__pycache__/57a5a84e79d0_init_db.cpython-312.pyc and /dev/null differ diff --git a/schema/app/__init__.py b/schema/app/__init__.py deleted file mode 100644 index 0650744..0000000 --- a/schema/app/__init__.py +++ /dev/null @@ -1 +0,0 @@ -from . import models diff --git a/schema/app/__pycache__/__init__.cpython-312.pyc b/schema/app/__pycache__/__init__.cpython-312.pyc deleted file mode 100644 index d31152d..0000000 Binary files a/schema/app/__pycache__/__init__.cpython-312.pyc and /dev/null differ diff --git a/schema/app/database/database.py b/schema/app/database/database.py deleted file mode 100644 index 5fa0774..0000000 --- a/schema/app/database/database.py +++ /dev/null @@ -1,63 +0,0 @@ -import configparser # Use ConfigParser for reading config - -from app.models.autocomplete import Base as AutocompleteBase -from app.models.companies import Base as CompaniesBase -from app.models.config import Base as ConfigBase -from app.models.leaders import Base as LeadersBase -from app.models.user_company_status import Base as UserCompanyStatusBase -from app.models.user_quota import Base as UserQuotaBase -from sqlalchemy import create_engine, text - -# Replace Alembic context with custom config handling -config = configparser.ConfigParser() - -# Load the configuration file (you can change this path to your actual config file) -config.read("alembic.ini") # Replace with the correct path if necessary - -# Retrieve the sqlalchemy.url from the config file -url = config.get( - "alembic", "sqlalchemy.url" -) # Assuming your config has a section [database] with sqlalchemy.url - -# SQLAlchemy engine setup -engine = create_engine(url) - - -def ensure_pg_trgm(connection): - if connection.dialect.name == "postgresql": - connection.execute(text("CREATE EXTENSION IF NOT EXISTS pg_trgm")) - - -def create_trgm_index(connection): - if connection.dialect.name == "postgresql": - connection.execute( - text( - """ - CREATE INDEX IF NOT EXISTS idx_companies_company_name_trgm - ON companies USING gin (LOWER(company_name) gin_trgm_ops); - """ - ) - ) - - -def init_db(): - # Create tables - bases = [ - CompaniesBase, - LeadersBase, - ConfigBase, - AutocompleteBase, - UserQuotaBase, - UserCompanyStatusBase, - ] - - for base in bases: - base.metadata.create_all(engine) - - with engine.begin() as conn: - ensure_pg_trgm(conn) - create_trgm_index(conn) - - -if __name__ == "__main__": - init_db() diff --git a/schema/app/models/__pycache__/autocomplete.cpython-312.pyc b/schema/app/models/__pycache__/autocomplete.cpython-312.pyc deleted file mode 100644 index 218a51b..0000000 Binary files a/schema/app/models/__pycache__/autocomplete.cpython-312.pyc and /dev/null differ diff --git a/schema/app/models/__pycache__/companies.cpython-312.pyc b/schema/app/models/__pycache__/companies.cpython-312.pyc deleted file mode 100644 index 414a665..0000000 Binary files a/schema/app/models/__pycache__/companies.cpython-312.pyc and /dev/null differ diff --git a/schema/app/models/__pycache__/config.cpython-312.pyc b/schema/app/models/__pycache__/config.cpython-312.pyc deleted file mode 100644 index 2d1a714..0000000 Binary files a/schema/app/models/__pycache__/config.cpython-312.pyc and /dev/null differ diff --git a/schema/app/models/__pycache__/leaders.cpython-312.pyc b/schema/app/models/__pycache__/leaders.cpython-312.pyc deleted file mode 100644 index a172625..0000000 Binary files a/schema/app/models/__pycache__/leaders.cpython-312.pyc and /dev/null differ diff --git a/schema/app/models/__pycache__/user.cpython-312.pyc b/schema/app/models/__pycache__/user.cpython-312.pyc deleted file mode 100644 index dab3359..0000000 Binary files a/schema/app/models/__pycache__/user.cpython-312.pyc and /dev/null differ diff --git a/schema/app/models/__pycache__/user_company_status.cpython-312.pyc b/schema/app/models/__pycache__/user_company_status.cpython-312.pyc deleted file mode 100644 index b786757..0000000 Binary files a/schema/app/models/__pycache__/user_company_status.cpython-312.pyc and /dev/null differ diff --git a/schema/app/models/__pycache__/user_quota.cpython-312.pyc b/schema/app/models/__pycache__/user_quota.cpython-312.pyc deleted file mode 100644 index 98a6bac..0000000 Binary files a/schema/app/models/__pycache__/user_quota.cpython-312.pyc and /dev/null differ diff --git a/schema/app/models/autocomplete.py b/schema/app/models/autocomplete.py deleted file mode 100644 index 9f0dd16..0000000 --- a/schema/app/models/autocomplete.py +++ /dev/null @@ -1,40 +0,0 @@ -from sqlalchemy import Column, Index, Integer, String -from sqlalchemy.ext.declarative import declarative_base - -Base = declarative_base() - - -class City(Base): - __tablename__ = "city" - - id = Column(Integer, primary_key=True, autoincrement=True) - name = Column(String) - - __table_args__ = (Index("ix_city_name", "name"),) - - -class IndustrySector(Base): - __tablename__ = "industry_sector" - - id = Column(Integer, primary_key=True, autoincrement=True) - name = Column(String) - - __table_args__ = (Index("ix_industry_sector_name", "name"),) - - -class LegalForm(Base): - __tablename__ = "legal_form" - - id = Column(Integer, primary_key=True, autoincrement=True) - name = Column(String) - - __table_args__ = (Index("ix_legal_form_name", "name"),) - - -class Region(Base): - __tablename__ = "region" - - id = Column(Integer, primary_key=True, autoincrement=True) - name = Column(String) - - __table_args__ = (Index("ix_region_name", "name"),) diff --git a/schema/app/models/companies.py b/schema/app/models/companies.py deleted file mode 100644 index a3bf9eb..0000000 --- a/schema/app/models/companies.py +++ /dev/null @@ -1,173 +0,0 @@ -from sqlalchemy import Column, Date, Float, Index, Integer, String -from sqlalchemy.dialects.postgresql import JSONB -from sqlalchemy.ext.declarative import declarative_base - -Base = declarative_base() - - -class Company(Base): - __tablename__ = "companies" - - id = Column(Integer, primary_key=True, autoincrement=True) - company_name = Column(String) - siren_number = Column(String) - nic_number = Column(String) - legal_form = Column(String) - ape_code = Column(String) - ape_label = Column(String) - address = Column(String) - postal_code = Column(String) - department_number = Column(String) - department = Column(String) - city = Column(String) - region = Column(String) - trade_name = Column(String) - - registration_date = Column(Date) - deregistration_date = Column(Date) - - # 2018 - closing_date_2018_1 = Column(Date) - revenue_2018_1 = Column(Float) - turnover_2018_1 = Column(Float) - closing_date_2018_2 = Column(Date) - revenue_2018_2 = Column(Float) - turnover_2018_2 = Column(Float) - closing_date_2018_3 = Column(Date) - revenue_2018_3 = Column(Float) - turnover_2018_3 = Column(Float) - - # 2019 - closing_date_2019_1 = Column(Date) - revenue_2019_1 = Column(Float) - turnover_2019_1 = Column(Float) - closing_date_2019_2 = Column(Date) - revenue_2019_2 = Column(Float) - turnover_2019_2 = Column(Float) - closing_date_2019_3 = Column(Date) - revenue_2019_3 = Column(Float) - turnover_2019_3 = Column(Float) - - # 2020 - closing_date_2020_1 = Column(Date) - revenue_2020_1 = Column(Float) - turnover_2020_1 = Column(Float) - closing_date_2020_2 = Column(Date) - revenue_2020_2 = Column(Float) - turnover_2020_2 = Column(Float) - closing_date_2020_3 = Column(Date) - revenue_2020_3 = Column(Float) - turnover_2020_3 = Column(Float) - - # 2021 - closing_date_2021_1 = Column(Date) - revenue_2021_1 = Column(Float) - turnover_2021_1 = Column(Float) - closing_date_2021_2 = Column(Date) - revenue_2021_2 = Column(Float) - turnover_2021_2 = Column(Float) - closing_date_2021_3 = Column(Date) - revenue_2021_3 = Column(Float) - turnover_2021_3 = Column(Float) - - # 2022 - closing_date_2022_1 = Column(Date) - revenue_2022_1 = Column(Float) - turnover_2022_1 = Column(Float) - closing_date_2022_2 = Column(Date) - revenue_2022_2 = Column(Float) - turnover_2022_2 = Column(Float) - closing_date_2022_3 = Column(Date) - revenue_2022_3 = Column(Float) - turnover_2022_3 = Column(Float) - - # 2023 - closing_date_2023_1 = Column(Date) - revenue_2023_1 = Column(Float) - turnover_2023_1 = Column(Float) - closing_date_2023_2 = Column(Date) - revenue_2023_2 = Column(Float) - turnover_2023_2 = Column(Float) - closing_date_2023_3 = Column(Date) - revenue_2023_3 = Column(Float) - turnover_2023_3 = Column(Float) - - industry_sector = Column(String) - phone_number = Column(String) - website = Column(String(3000)) - - reviews = Column(JSONB) - schedule = Column(JSONB) - - instagram = Column(String(10000)) - facebook = Column(String(3000)) - twitter = Column(String(3000)) - linkedin = Column(String(3000)) - youtube = Column(String(3000)) - email = Column(String(3000)) - - scraping_date = Column(Date) - date_creation = Column(Date) - last_processing_date = Column(Date) - number_of_employee = Column(Integer) - company_category = Column(String) - - __table_args__ = ( - # Single column indexes - Index("ix_companies_siren_number", "siren_number"), - Index("ix_companies_company_name", "company_name"), - Index("ix_companies_legal_form", "legal_form"), - Index("ix_companies_industry_sector", "industry_sector"), - Index("ix_companies_region", "region"), - Index("ix_companies_city", "city"), - Index("ix_companies_phone_number", "phone_number"), - Index("ix_companies_website", "website"), - Index("ix_companies_email", "email"), - Index("ix_companies_number_of_employee", "number_of_employee"), - Index("ix_companies_linkedin", "linkedin"), - Index("ix_companies_twitter", "twitter"), - Index("ix_companies_facebook", "facebook"), - Index("ix_companies_instagram", "instagram"), - Index("ix_companies_youtube", "youtube"), - # - # Composite indexes - Index( - "ix_companies_region_city_industry_sector_legal_form", - "region", - "city", - "industry_sector", - "legal_form", - ), - Index( - "ix_companies_region_city_industry_sector", - "region", - "city", - "industry_sector", - ), - Index("ix_companies_region_city_legal_form", "region", "city", "legal_form"), - Index("ix_companies_region_city", "region", "city"), - Index("ix_companies_region_industry_sector", "region", "industry_sector"), - Index("ix_companies_region_legal_form", "region", "legal_form"), - Index( - "ix_companies_city_industry_sector_legal_form", - "city", - "industry_sector", - "legal_form", - ), - Index("ix_companies_city_industry_sector", "city", "industry_sector"), - Index("ix_companies_city_legal_form", "city", "legal_form"), - Index( - "ix_companies_region_industry_sector_legal_form", - "region", - "industry_sector", - "legal_form", - ), - Index( - "ix_companies_industry_sector_legal_form", "industry_sector", "legal_form" - ), - Index( - "ix_companies_industry_sector_number_of_employee", - "industry_sector", - "number_of_employee", - ), - ) diff --git a/schema/app/models/config.py b/schema/app/models/config.py deleted file mode 100644 index e66b787..0000000 --- a/schema/app/models/config.py +++ /dev/null @@ -1,11 +0,0 @@ -from sqlalchemy import Column, Date, Integer -from sqlalchemy.ext.declarative import declarative_base - -Base = declarative_base() - - -class Config(Base): - __tablename__ = "config" - - id = Column(Integer, primary_key=True, autoincrement=True) - last_reset_quota_date = Column(Date) diff --git a/schema/app/models/leaders.py b/schema/app/models/leaders.py deleted file mode 100644 index 6944fc3..0000000 --- a/schema/app/models/leaders.py +++ /dev/null @@ -1,30 +0,0 @@ -from sqlalchemy import Column, Index, Integer, String -from sqlalchemy.ext.declarative import declarative_base - -Base = declarative_base() - - -class Leader(Base): - __tablename__ = "leaders" - - id = Column(Integer, primary_key=True, autoincrement=True) - siren = Column(String) - role = Column(String) - last_name = Column(String) - first_name = Column(String) - gestion_number = Column(String) - type = Column(String) - event_name = Column(String) - usage_name = Column(String) - pseudo = Column(String) - company_name = Column(String(3000)) - legal_form = Column(String) - id_data = Column(String) - - __table_args__ = ( - Index("idx_leader_siren", "siren"), - Index("idx_leader_company_name", "company_name"), - Index("idx_leader_first_name", "first_name"), - Index("idx_leader_last_name", "last_name"), - Index("idx_leader_role", "role"), - ) diff --git a/schema/app/models/user_company_status.py b/schema/app/models/user_company_status.py deleted file mode 100644 index 3d49bc1..0000000 --- a/schema/app/models/user_company_status.py +++ /dev/null @@ -1,26 +0,0 @@ -import enum - -from sqlalchemy import Column, Enum, Index, Integer, String -from sqlalchemy.ext.declarative import declarative_base - -Base = declarative_base() - - -class Status(enum.Enum): - NOT_DONE = "NOT_DONE" - TO_DO = "TO_DO" - DONE = "DONE" - - -class UserCompanyStatus(Base): - __tablename__ = "user_company_status" - - id = Column(Integer, primary_key=True, autoincrement=True) - user_id = Column(String) - status = Column(Enum(Status)) - company_id = Column(Integer) - - __table_args__ = ( - Index("ix_user_company_status_user_id", "user_id"), - Index("ix_user_company_status_company_id", "company_id"), - ) diff --git a/schema/app/models/user_quota.py b/schema/app/models/user_quota.py deleted file mode 100644 index 23f9b4f..0000000 --- a/schema/app/models/user_quota.py +++ /dev/null @@ -1,18 +0,0 @@ -from sqlalchemy import Column, Index, Integer, String -from sqlalchemy.ext.declarative import declarative_base - -Base = declarative_base() - - -class UserQuota(Base): - __tablename__ = "user_quota" - - user_id = Column(String, primary_key=True) - quota_allocated = Column(Integer) - quota_used = Column(Integer) - - __table_args__ = ( - Index("ix_user_quota_user_id", "user_id"), - Index("ix_user_quota_quota_allocated", "quota_allocated"), - Index("ix_user_quota_quota_used", "quota_used"), - ) diff --git a/schema/package.json b/schema/package.json new file mode 100644 index 0000000..303efa7 --- /dev/null +++ b/schema/package.json @@ -0,0 +1,9 @@ +{ + "devDependencies": { + "prisma": "^6.16.1" + }, + "dependencies": { + "@prisma/client": "^6.16.1", + "dotenv": "^17.2.2" + } +} diff --git a/schema/pnpm-lock.yaml b/schema/pnpm-lock.yaml new file mode 100644 index 0000000..a777f9e --- /dev/null +++ b/schema/pnpm-lock.yaml @@ -0,0 +1,299 @@ +lockfileVersion: '9.0' + +settings: + autoInstallPeers: true + excludeLinksFromLockfile: false + +importers: + + .: + dependencies: + '@prisma/client': + specifier: ^6.16.1 + version: 6.16.1(prisma@6.16.1) + dotenv: + specifier: ^17.2.2 + version: 17.2.2 + devDependencies: + prisma: + specifier: ^6.16.1 + version: 6.16.1 + +packages: + + '@prisma/client@6.16.1': + resolution: {integrity: sha512-QaBCOY29lLAxEFFJgBPyW3WInCW52fJeQTmWx/h6YsP5u0bwuqP51aP0uhqFvhK9DaZPwvai/M4tSDYLVE9vRg==} + engines: {node: '>=18.18'} + peerDependencies: + prisma: '*' + typescript: '>=5.1.0' + peerDependenciesMeta: + prisma: + optional: true + typescript: + optional: true + + '@prisma/config@6.16.1': + resolution: {integrity: sha512-sz3uxRPNL62QrJ0EYiujCFkIGZ3hg+9hgC1Ae1HjoYuj0BxCqHua4JNijYvYCrh9LlofZDZcRBX3tHBfLvAngA==} + + '@prisma/debug@6.16.1': + resolution: {integrity: sha512-RWv/VisW5vJE4cDRTuAHeVedtGoItXTnhuLHsSlJ9202QKz60uiXWywBlVcqXVq8bFeIZoCoWH+R1duZJPwqLw==} + + '@prisma/engines-version@6.16.0-7.1c57fdcd7e44b29b9313256c76699e91c3ac3c43': + resolution: {integrity: sha512-ThvlDaKIVrnrv97ujNFDYiQbeMQpLa0O86HFA2mNoip4mtFqM7U5GSz2ie1i2xByZtvPztJlNRgPsXGeM/kqAA==} + + '@prisma/engines@6.16.1': + resolution: {integrity: sha512-EOnEM5HlosPudBqbI+jipmaW/vQEaF0bKBo4gVkGabasINHR6RpC6h44fKZEqx4GD8CvH+einD2+b49DQrwrAg==} + + '@prisma/fetch-engine@6.16.1': + resolution: {integrity: sha512-fl/PKQ8da5YTayw86WD3O9OmKJEM43gD3vANy2hS5S1CnfW2oPXk+Q03+gUWqcKK306QqhjjIHRFuTZ31WaosQ==} + + '@prisma/get-platform@6.16.1': + resolution: {integrity: sha512-kUfg4vagBG7dnaGRcGd1c0ytQFcDj2SUABiuveIpL3bthFdTLI6PJeLEia6Q8Dgh+WhPdo0N2q0Fzjk63XTyaA==} + + '@standard-schema/spec@1.0.0': + resolution: {integrity: sha512-m2bOd0f2RT9k8QJx1JN85cZYyH1RqFBdlwtkSlf4tBDYLCiiZnv1fIIwacK6cqwXavOydf0NPToMQgpKq+dVlA==} + + c12@3.1.0: + resolution: {integrity: sha512-uWoS8OU1MEIsOv8p/5a82c3H31LsWVR5qiyXVfBNOzfffjUWtPnhAb4BYI2uG2HfGmZmFjCtui5XNWaps+iFuw==} + peerDependencies: + magicast: ^0.3.5 + peerDependenciesMeta: + magicast: + optional: true + + chokidar@4.0.3: + resolution: {integrity: sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==} + engines: {node: '>= 14.16.0'} + + citty@0.1.6: + resolution: {integrity: sha512-tskPPKEs8D2KPafUypv2gxwJP8h/OaJmC82QQGGDQcHvXX43xF2VDACcJVmZ0EuSxkpO9Kc4MlrA3q0+FG58AQ==} + + confbox@0.2.2: + resolution: {integrity: sha512-1NB+BKqhtNipMsov4xI/NnhCKp9XG9NamYp5PVm9klAT0fsrNPjaFICsCFhNhwZJKNh7zB/3q8qXz0E9oaMNtQ==} + + consola@3.4.2: + resolution: {integrity: sha512-5IKcdX0nnYavi6G7TtOhwkYzyjfJlatbjMjuLSfE2kYT5pMDOilZ4OvMhi637CcDICTmz3wARPoyhqyX1Y+XvA==} + engines: {node: ^14.18.0 || >=16.10.0} + + deepmerge-ts@7.1.5: + resolution: {integrity: sha512-HOJkrhaYsweh+W+e74Yn7YStZOilkoPb6fycpwNLKzSPtruFs48nYis0zy5yJz1+ktUhHxoRDJ27RQAWLIJVJw==} + engines: {node: '>=16.0.0'} + + defu@6.1.4: + resolution: {integrity: sha512-mEQCMmwJu317oSz8CwdIOdwf3xMif1ttiM8LTufzc3g6kR+9Pe236twL8j3IYT1F7GfRgGcW6MWxzZjLIkuHIg==} + + destr@2.0.5: + resolution: {integrity: sha512-ugFTXCtDZunbzasqBxrK93Ik/DRYsO6S/fedkWEMKqt04xZ4csmnmwGDBAb07QWNaGMAmnTIemsYZCksjATwsA==} + + dotenv@16.6.1: + resolution: {integrity: sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==} + engines: {node: '>=12'} + + dotenv@17.2.2: + resolution: {integrity: sha512-Sf2LSQP+bOlhKWWyhFsn0UsfdK/kCWRv1iuA2gXAwt3dyNabr6QSj00I2V10pidqz69soatm9ZwZvpQMTIOd5Q==} + engines: {node: '>=12'} + + effect@3.16.12: + resolution: {integrity: sha512-N39iBk0K71F9nb442TLbTkjl24FLUzuvx2i1I2RsEAQsdAdUTuUoW0vlfUXgkMTUOnYqKnWcFfqw4hK4Pw27hg==} + + empathic@2.0.0: + resolution: {integrity: sha512-i6UzDscO/XfAcNYD75CfICkmfLedpyPDdozrLMmQc5ORaQcdMoc21OnlEylMIqI7U8eniKrPMxxtj8k0vhmJhA==} + engines: {node: '>=14'} + + exsolve@1.0.7: + resolution: {integrity: sha512-VO5fQUzZtI6C+vx4w/4BWJpg3s/5l+6pRQEHzFRM8WFi4XffSP1Z+4qi7GbjWbvRQEbdIco5mIMq+zX4rPuLrw==} + + fast-check@3.23.2: + resolution: {integrity: sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A==} + engines: {node: '>=8.0.0'} + + giget@2.0.0: + resolution: {integrity: sha512-L5bGsVkxJbJgdnwyuheIunkGatUF/zssUoxxjACCseZYAVbaqdh9Tsmmlkl8vYan09H7sbvKt4pS8GqKLBrEzA==} + hasBin: true + + jiti@2.5.1: + resolution: {integrity: sha512-twQoecYPiVA5K/h6SxtORw/Bs3ar+mLUtoPSc7iMXzQzK8d7eJ/R09wmTwAjiamETn1cXYPGfNnu7DMoHgu12w==} + hasBin: true + + node-fetch-native@1.6.7: + resolution: {integrity: sha512-g9yhqoedzIUm0nTnTqAQvueMPVOuIY16bqgAJJC8XOOubYFNwz6IER9qs0Gq2Xd0+CecCKFjtdDTMA4u4xG06Q==} + + nypm@0.6.2: + resolution: {integrity: sha512-7eM+hpOtrKrBDCh7Ypu2lJ9Z7PNZBdi/8AT3AX8xoCj43BBVHD0hPSTEvMtkMpfs8FCqBGhxB+uToIQimA111g==} + engines: {node: ^14.16.0 || >=16.10.0} + hasBin: true + + ohash@2.0.11: + resolution: {integrity: sha512-RdR9FQrFwNBNXAr4GixM8YaRZRJ5PUWbKYbE5eOsrwAjJW0q2REGcf79oYPsLyskQCZG1PLN+S/K1V00joZAoQ==} + + pathe@2.0.3: + resolution: {integrity: sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==} + + perfect-debounce@1.0.0: + resolution: {integrity: sha512-xCy9V055GLEqoFaHoC1SoLIaLmWctgCUaBaWxDZ7/Zx4CTyX7cJQLJOok/orfjZAh9kEYpjJa4d0KcJmCbctZA==} + + pkg-types@2.3.0: + resolution: {integrity: sha512-SIqCzDRg0s9npO5XQ3tNZioRY1uK06lA41ynBC1YmFTmnY6FjUjVt6s4LoADmwoig1qqD0oK8h1p/8mlMx8Oig==} + + prisma@6.16.1: + resolution: {integrity: sha512-MFkMU0eaDDKAT4R/By2IA9oQmwLTxokqv2wegAErr9Rf+oIe7W2sYpE/Uxq0H2DliIR7vnV63PkC1bEwUtl98w==} + engines: {node: '>=18.18'} + hasBin: true + peerDependencies: + typescript: '>=5.1.0' + peerDependenciesMeta: + typescript: + optional: true + + pure-rand@6.1.0: + resolution: {integrity: sha512-bVWawvoZoBYpp6yIoQtQXHZjmz35RSVHnUOTefl8Vcjr8snTPY1wnpSPMWekcFwbxI6gtmT7rSYPFvz71ldiOA==} + + rc9@2.1.2: + resolution: {integrity: sha512-btXCnMmRIBINM2LDZoEmOogIZU7Qe7zn4BpomSKZ/ykbLObuBdvG+mFq11DL6fjH1DRwHhrlgtYWG96bJiC7Cg==} + + readdirp@4.1.2: + resolution: {integrity: sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==} + engines: {node: '>= 14.18.0'} + + tinyexec@1.0.1: + resolution: {integrity: sha512-5uC6DDlmeqiOwCPmK9jMSdOuZTh8bU39Ys6yidB+UTt5hfZUPGAypSgFRiEp+jbi9qH40BLDvy85jIU88wKSqw==} + +snapshots: + + '@prisma/client@6.16.1(prisma@6.16.1)': + optionalDependencies: + prisma: 6.16.1 + + '@prisma/config@6.16.1': + dependencies: + c12: 3.1.0 + deepmerge-ts: 7.1.5 + effect: 3.16.12 + empathic: 2.0.0 + transitivePeerDependencies: + - magicast + + '@prisma/debug@6.16.1': {} + + '@prisma/engines-version@6.16.0-7.1c57fdcd7e44b29b9313256c76699e91c3ac3c43': {} + + '@prisma/engines@6.16.1': + dependencies: + '@prisma/debug': 6.16.1 + '@prisma/engines-version': 6.16.0-7.1c57fdcd7e44b29b9313256c76699e91c3ac3c43 + '@prisma/fetch-engine': 6.16.1 + '@prisma/get-platform': 6.16.1 + + '@prisma/fetch-engine@6.16.1': + dependencies: + '@prisma/debug': 6.16.1 + '@prisma/engines-version': 6.16.0-7.1c57fdcd7e44b29b9313256c76699e91c3ac3c43 + '@prisma/get-platform': 6.16.1 + + '@prisma/get-platform@6.16.1': + dependencies: + '@prisma/debug': 6.16.1 + + '@standard-schema/spec@1.0.0': {} + + c12@3.1.0: + dependencies: + chokidar: 4.0.3 + confbox: 0.2.2 + defu: 6.1.4 + dotenv: 16.6.1 + exsolve: 1.0.7 + giget: 2.0.0 + jiti: 2.5.1 + ohash: 2.0.11 + pathe: 2.0.3 + perfect-debounce: 1.0.0 + pkg-types: 2.3.0 + rc9: 2.1.2 + + chokidar@4.0.3: + dependencies: + readdirp: 4.1.2 + + citty@0.1.6: + dependencies: + consola: 3.4.2 + + confbox@0.2.2: {} + + consola@3.4.2: {} + + deepmerge-ts@7.1.5: {} + + defu@6.1.4: {} + + destr@2.0.5: {} + + dotenv@16.6.1: {} + + dotenv@17.2.2: {} + + effect@3.16.12: + dependencies: + '@standard-schema/spec': 1.0.0 + fast-check: 3.23.2 + + empathic@2.0.0: {} + + exsolve@1.0.7: {} + + fast-check@3.23.2: + dependencies: + pure-rand: 6.1.0 + + giget@2.0.0: + dependencies: + citty: 0.1.6 + consola: 3.4.2 + defu: 6.1.4 + node-fetch-native: 1.6.7 + nypm: 0.6.2 + pathe: 2.0.3 + + jiti@2.5.1: {} + + node-fetch-native@1.6.7: {} + + nypm@0.6.2: + dependencies: + citty: 0.1.6 + consola: 3.4.2 + pathe: 2.0.3 + pkg-types: 2.3.0 + tinyexec: 1.0.1 + + ohash@2.0.11: {} + + pathe@2.0.3: {} + + perfect-debounce@1.0.0: {} + + pkg-types@2.3.0: + dependencies: + confbox: 0.2.2 + exsolve: 1.0.7 + pathe: 2.0.3 + + prisma@6.16.1: + dependencies: + '@prisma/config': 6.16.1 + '@prisma/engines': 6.16.1 + transitivePeerDependencies: + - magicast + + pure-rand@6.1.0: {} + + rc9@2.1.2: + dependencies: + defu: 6.1.4 + destr: 2.0.5 + + readdirp@4.1.2: {} + + tinyexec@1.0.1: {} diff --git a/schema/prisma.config.ts b/schema/prisma.config.ts new file mode 100644 index 0000000..a77d031 --- /dev/null +++ b/schema/prisma.config.ts @@ -0,0 +1,6 @@ +import 'dotenv/config' +import { defineConfig } from 'prisma/config' + +export default defineConfig({ + schema: './prisma/schema/', +}) \ No newline at end of file diff --git a/schema/prisma/schema/autocomplete.prisma b/schema/prisma/schema/autocomplete.prisma new file mode 100644 index 0000000..86faf61 --- /dev/null +++ b/schema/prisma/schema/autocomplete.prisma @@ -0,0 +1,31 @@ +model City { + id Int @id @default(autoincrement()) + name String? + + @@map("city") + @@index([name], map: "ix_city_name") +} + +model IndustrySector { + id Int @id @default(autoincrement()) + name String? + + @@map("industry_sector") + @@index([name], map: "ix_industry_sector_name") +} + +model LegalForm { + id Int @id @default(autoincrement()) + name String? + + @@map("legal_form") + @@index([name], map: "ix_legal_form_name") +} + +model Region { + id Int @id @default(autoincrement()) + name String? + + @@map("region") + @@index([name], map: "ix_region_name") +} diff --git a/schema/prisma/schema/company.prisma b/schema/prisma/schema/company.prisma new file mode 100644 index 0000000..31bbf6a --- /dev/null +++ b/schema/prisma/schema/company.prisma @@ -0,0 +1,133 @@ +model Company { + id Int @id @default(autoincrement()) + company_name String? + siren_number String? + nic_number String? + legal_form String? + ape_code String? + ape_label String? + address String? + postal_code String? + department_number String? + department String? + city String? + region String? + trade_name String? + registration_date DateTime? @db.Date + deregistration_date DateTime? @db.Date + + // 2018 financial data + closing_date_2018_1 DateTime? @db.Date + revenue_2018_1 Float? + turnover_2018_1 Float? + closing_date_2018_2 DateTime? @db.Date + revenue_2018_2 Float? + turnover_2018_2 Float? + closing_date_2018_3 DateTime? @db.Date + revenue_2018_3 Float? + turnover_2018_3 Float? + + // 2019 financial data + closing_date_2019_1 DateTime? @db.Date + revenue_2019_1 Float? + turnover_2019_1 Float? + closing_date_2019_2 DateTime? @db.Date + revenue_2019_2 Float? + turnover_2019_2 Float? + closing_date_2019_3 DateTime? @db.Date + revenue_2019_3 Float? + turnover_2019_3 Float? + + // 2020 financial data + closing_date_2020_1 DateTime? @db.Date + revenue_2020_1 Float? + turnover_2020_1 Float? + closing_date_2020_2 DateTime? @db.Date + revenue_2020_2 Float? + turnover_2020_2 Float? + closing_date_2020_3 DateTime? @db.Date + revenue_2020_3 Float? + turnover_2020_3 Float? + + // 2021 financial data + closing_date_2021_1 DateTime? @db.Date + revenue_2021_1 Float? + turnover_2021_1 Float? + closing_date_2021_2 DateTime? @db.Date + revenue_2021_2 Float? + turnover_2021_2 Float? + closing_date_2021_3 DateTime? @db.Date + revenue_2021_3 Float? + turnover_2021_3 Float? + + // 2022 financial data + closing_date_2022_1 DateTime? @db.Date + revenue_2022_1 Float? + turnover_2022_1 Float? + closing_date_2022_2 DateTime? @db.Date + revenue_2022_2 Float? + turnover_2022_2 Float? + closing_date_2022_3 DateTime? @db.Date + revenue_2022_3 Float? + turnover_2022_3 Float? + + // 2023 financial data + closing_date_2023_1 DateTime? @db.Date + revenue_2023_1 Float? + turnover_2023_1 Float? + closing_date_2023_2 DateTime? @db.Date + revenue_2023_2 Float? + turnover_2023_2 Float? + closing_date_2023_3 DateTime? @db.Date + revenue_2023_3 Float? + turnover_2023_3 Float? + + industry_sector String? + phone_number String? + website String? @db.VarChar(3000) + reviews Json? + schedule Json? + instagram String? @db.VarChar(10000) + facebook String? @db.VarChar(3000) + twitter String? @db.VarChar(3000) + linkedin String? @db.VarChar(3000) + youtube String? @db.VarChar(3000) + email String? @db.VarChar(3000) + scraping_date DateTime? @db.Date + date_creation DateTime? @db.Date + last_processing_date DateTime? @db.Date + number_of_employee Int? + company_category String? + + // Relations + user_company_statuses UserCompanyStatus[] + + @@map("companies") + @@index([siren_number], map: "ix_companies_siren_number") + @@index([company_name], map: "ix_companies_company_name") + @@index([legal_form], map: "ix_companies_legal_form") + @@index([industry_sector], map: "ix_companies_industry_sector") + @@index([region], map: "ix_companies_region") + @@index([city], map: "ix_companies_city") + @@index([phone_number], map: "ix_companies_phone_number") + @@index([website], map: "ix_companies_website") + @@index([email], map: "ix_companies_email") + @@index([number_of_employee], map: "ix_companies_number_of_employee") + @@index([linkedin], map: "ix_companies_linkedin") + @@index([twitter], map: "ix_companies_twitter") + @@index([facebook], map: "ix_companies_facebook") + @@index([instagram], map: "ix_companies_instagram") + @@index([youtube], map: "ix_companies_youtube") + @@index([region, city, industry_sector, legal_form], map: "ix_companies_region_city_industry_sector_legal_form") + @@index([region, city, industry_sector], map: "ix_companies_region_city_industry_sector") + @@index([region, city, legal_form], map: "ix_companies_region_city_legal_form") + @@index([region, city], map: "ix_companies_region_city") + @@index([region, industry_sector], map: "ix_companies_region_industry_sector") + @@index([region, legal_form], map: "ix_companies_region_legal_form") + @@index([city, industry_sector, legal_form], map: "ix_companies_city_industry_sector_legal_form") + @@index([city, industry_sector], map: "ix_companies_city_industry_sector") + @@index([city, legal_form], map: "ix_companies_city_legal_form") + @@index([region, industry_sector, legal_form], map: "ix_companies_region_industry_sector_legal_form") + @@index([industry_sector, legal_form], map: "ix_companies_industry_sector_legal_form") + @@index([industry_sector, number_of_employee], map: "ix_companies_industry_sector_number_of_employee") +} \ No newline at end of file diff --git a/schema/prisma/schema/config.prisma b/schema/prisma/schema/config.prisma new file mode 100644 index 0000000..8992344 --- /dev/null +++ b/schema/prisma/schema/config.prisma @@ -0,0 +1,7 @@ +model Config { + id Int @id @default(autoincrement()) + last_reset_quota_date DateTime? @db.Date + + @@map("config") +} + diff --git a/schema/prisma/schema/leader.prisma b/schema/prisma/schema/leader.prisma new file mode 100644 index 0000000..4fc4faa --- /dev/null +++ b/schema/prisma/schema/leader.prisma @@ -0,0 +1,22 @@ +model Leader { + id Int @id @default(autoincrement()) + siren String? + role String? + last_name String? + first_name String? + gestion_number String? + type String? + event_name String? + usage_name String? + pseudo String? + company_name String? @db.VarChar(3000) + legal_form String? + id_data String? + + @@map("leaders") + @@index([siren], map: "idx_leader_siren") + @@index([company_name], map: "idx_leader_company_name") + @@index([first_name], map: "idx_leader_first_name") + @@index([last_name], map: "idx_leader_last_name") + @@index([role], map: "idx_leader_role") +} diff --git a/schema/prisma/schema/migrations/20250916093415_first_migration/migration.sql b/schema/prisma/schema/migrations/20250916093415_first_migration/migration.sql new file mode 100644 index 0000000..c433c8f --- /dev/null +++ b/schema/prisma/schema/migrations/20250916093415_first_migration/migration.sql @@ -0,0 +1,298 @@ +-- CreateEnum +CREATE TYPE "public"."Status" AS ENUM ('NOT_DONE', 'TO_DO', 'DONE'); + +-- CreateTable +CREATE TABLE "public"."city" ( + "id" SERIAL NOT NULL, + "name" TEXT, + + CONSTRAINT "city_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "public"."industry_sector" ( + "id" SERIAL NOT NULL, + "name" TEXT, + + CONSTRAINT "industry_sector_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "public"."legal_form" ( + "id" SERIAL NOT NULL, + "name" TEXT, + + CONSTRAINT "legal_form_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "public"."region" ( + "id" SERIAL NOT NULL, + "name" TEXT, + + CONSTRAINT "region_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "public"."companies" ( + "id" SERIAL NOT NULL, + "company_name" TEXT, + "siren_number" TEXT, + "nic_number" TEXT, + "legal_form" TEXT, + "ape_code" TEXT, + "ape_label" TEXT, + "address" TEXT, + "postal_code" TEXT, + "department_number" TEXT, + "department" TEXT, + "city" TEXT, + "region" TEXT, + "trade_name" TEXT, + "registration_date" DATE, + "deregistration_date" DATE, + "closing_date_2018_1" DATE, + "revenue_2018_1" DOUBLE PRECISION, + "turnover_2018_1" DOUBLE PRECISION, + "closing_date_2018_2" DATE, + "revenue_2018_2" DOUBLE PRECISION, + "turnover_2018_2" DOUBLE PRECISION, + "closing_date_2018_3" DATE, + "revenue_2018_3" DOUBLE PRECISION, + "turnover_2018_3" DOUBLE PRECISION, + "closing_date_2019_1" DATE, + "revenue_2019_1" DOUBLE PRECISION, + "turnover_2019_1" DOUBLE PRECISION, + "closing_date_2019_2" DATE, + "revenue_2019_2" DOUBLE PRECISION, + "turnover_2019_2" DOUBLE PRECISION, + "closing_date_2019_3" DATE, + "revenue_2019_3" DOUBLE PRECISION, + "turnover_2019_3" DOUBLE PRECISION, + "closing_date_2020_1" DATE, + "revenue_2020_1" DOUBLE PRECISION, + "turnover_2020_1" DOUBLE PRECISION, + "closing_date_2020_2" DATE, + "revenue_2020_2" DOUBLE PRECISION, + "turnover_2020_2" DOUBLE PRECISION, + "closing_date_2020_3" DATE, + "revenue_2020_3" DOUBLE PRECISION, + "turnover_2020_3" DOUBLE PRECISION, + "closing_date_2021_1" DATE, + "revenue_2021_1" DOUBLE PRECISION, + "turnover_2021_1" DOUBLE PRECISION, + "closing_date_2021_2" DATE, + "revenue_2021_2" DOUBLE PRECISION, + "turnover_2021_2" DOUBLE PRECISION, + "closing_date_2021_3" DATE, + "revenue_2021_3" DOUBLE PRECISION, + "turnover_2021_3" DOUBLE PRECISION, + "closing_date_2022_1" DATE, + "revenue_2022_1" DOUBLE PRECISION, + "turnover_2022_1" DOUBLE PRECISION, + "closing_date_2022_2" DATE, + "revenue_2022_2" DOUBLE PRECISION, + "turnover_2022_2" DOUBLE PRECISION, + "closing_date_2022_3" DATE, + "revenue_2022_3" DOUBLE PRECISION, + "turnover_2022_3" DOUBLE PRECISION, + "closing_date_2023_1" DATE, + "revenue_2023_1" DOUBLE PRECISION, + "turnover_2023_1" DOUBLE PRECISION, + "closing_date_2023_2" DATE, + "revenue_2023_2" DOUBLE PRECISION, + "turnover_2023_2" DOUBLE PRECISION, + "closing_date_2023_3" DATE, + "revenue_2023_3" DOUBLE PRECISION, + "turnover_2023_3" DOUBLE PRECISION, + "industry_sector" TEXT, + "phone_number" TEXT, + "website" VARCHAR(3000), + "reviews" JSONB, + "schedule" JSONB, + "instagram" VARCHAR(10000), + "facebook" VARCHAR(3000), + "twitter" VARCHAR(3000), + "linkedin" VARCHAR(3000), + "youtube" VARCHAR(3000), + "email" VARCHAR(3000), + "scraping_date" DATE, + "date_creation" DATE, + "last_processing_date" DATE, + "number_of_employee" INTEGER, + "company_category" TEXT, + + CONSTRAINT "companies_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "public"."config" ( + "id" SERIAL NOT NULL, + "last_reset_quota_date" DATE, + + CONSTRAINT "config_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "public"."leaders" ( + "id" SERIAL NOT NULL, + "siren" TEXT, + "role" TEXT, + "last_name" TEXT, + "first_name" TEXT, + "gestion_number" TEXT, + "type" TEXT, + "event_name" TEXT, + "usage_name" TEXT, + "pseudo" TEXT, + "company_name" VARCHAR(3000), + "legal_form" TEXT, + "id_data" TEXT, + + CONSTRAINT "leaders_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "public"."user_company_status" ( + "id" SERIAL NOT NULL, + "user_id" TEXT, + "status" "public"."Status", + "company_id" INTEGER, + + CONSTRAINT "user_company_status_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "public"."user_quota" ( + "user_id" TEXT NOT NULL, + "quota_allocated" INTEGER, + "quota_used" INTEGER, + + CONSTRAINT "user_quota_pkey" PRIMARY KEY ("user_id") +); + +-- CreateIndex +CREATE INDEX "ix_city_name" ON "public"."city"("name"); + +-- CreateIndex +CREATE INDEX "ix_industry_sector_name" ON "public"."industry_sector"("name"); + +-- CreateIndex +CREATE INDEX "ix_legal_form_name" ON "public"."legal_form"("name"); + +-- CreateIndex +CREATE INDEX "ix_region_name" ON "public"."region"("name"); + +-- CreateIndex +CREATE INDEX "ix_companies_siren_number" ON "public"."companies"("siren_number"); + +-- CreateIndex +CREATE INDEX "ix_companies_company_name" ON "public"."companies"("company_name"); + +-- CreateIndex +CREATE INDEX "ix_companies_legal_form" ON "public"."companies"("legal_form"); + +-- CreateIndex +CREATE INDEX "ix_companies_industry_sector" ON "public"."companies"("industry_sector"); + +-- CreateIndex +CREATE INDEX "ix_companies_region" ON "public"."companies"("region"); + +-- CreateIndex +CREATE INDEX "ix_companies_city" ON "public"."companies"("city"); + +-- CreateIndex +CREATE INDEX "ix_companies_phone_number" ON "public"."companies"("phone_number"); + +-- CreateIndex +CREATE INDEX "ix_companies_website" ON "public"."companies"("website"); + +-- CreateIndex +CREATE INDEX "ix_companies_email" ON "public"."companies"("email"); + +-- CreateIndex +CREATE INDEX "ix_companies_number_of_employee" ON "public"."companies"("number_of_employee"); + +-- CreateIndex +CREATE INDEX "ix_companies_linkedin" ON "public"."companies"("linkedin"); + +-- CreateIndex +CREATE INDEX "ix_companies_twitter" ON "public"."companies"("twitter"); + +-- CreateIndex +CREATE INDEX "ix_companies_facebook" ON "public"."companies"("facebook"); + +-- CreateIndex +CREATE INDEX "ix_companies_instagram" ON "public"."companies"("instagram"); + +-- CreateIndex +CREATE INDEX "ix_companies_youtube" ON "public"."companies"("youtube"); + +-- CreateIndex +CREATE INDEX "ix_companies_region_city_industry_sector_legal_form" ON "public"."companies"("region", "city", "industry_sector", "legal_form"); + +-- CreateIndex +CREATE INDEX "ix_companies_region_city_industry_sector" ON "public"."companies"("region", "city", "industry_sector"); + +-- CreateIndex +CREATE INDEX "ix_companies_region_city_legal_form" ON "public"."companies"("region", "city", "legal_form"); + +-- CreateIndex +CREATE INDEX "ix_companies_region_city" ON "public"."companies"("region", "city"); + +-- CreateIndex +CREATE INDEX "ix_companies_region_industry_sector" ON "public"."companies"("region", "industry_sector"); + +-- CreateIndex +CREATE INDEX "ix_companies_region_legal_form" ON "public"."companies"("region", "legal_form"); + +-- CreateIndex +CREATE INDEX "ix_companies_city_industry_sector_legal_form" ON "public"."companies"("city", "industry_sector", "legal_form"); + +-- CreateIndex +CREATE INDEX "ix_companies_city_industry_sector" ON "public"."companies"("city", "industry_sector"); + +-- CreateIndex +CREATE INDEX "ix_companies_city_legal_form" ON "public"."companies"("city", "legal_form"); + +-- CreateIndex +CREATE INDEX "ix_companies_region_industry_sector_legal_form" ON "public"."companies"("region", "industry_sector", "legal_form"); + +-- CreateIndex +CREATE INDEX "ix_companies_industry_sector_legal_form" ON "public"."companies"("industry_sector", "legal_form"); + +-- CreateIndex +CREATE INDEX "ix_companies_industry_sector_number_of_employee" ON "public"."companies"("industry_sector", "number_of_employee"); + +-- CreateIndex +CREATE INDEX "idx_leader_siren" ON "public"."leaders"("siren"); + +-- CreateIndex +CREATE INDEX "idx_leader_company_name" ON "public"."leaders"("company_name"); + +-- CreateIndex +CREATE INDEX "idx_leader_first_name" ON "public"."leaders"("first_name"); + +-- CreateIndex +CREATE INDEX "idx_leader_last_name" ON "public"."leaders"("last_name"); + +-- CreateIndex +CREATE INDEX "idx_leader_role" ON "public"."leaders"("role"); + +-- CreateIndex +CREATE INDEX "ix_user_company_status_user_id" ON "public"."user_company_status"("user_id"); + +-- CreateIndex +CREATE INDEX "ix_user_company_status_company_id" ON "public"."user_company_status"("company_id"); + +-- CreateIndex +CREATE INDEX "ix_user_quota_user_id" ON "public"."user_quota"("user_id"); + +-- CreateIndex +CREATE INDEX "ix_user_quota_quota_allocated" ON "public"."user_quota"("quota_allocated"); + +-- CreateIndex +CREATE INDEX "ix_user_quota_quota_used" ON "public"."user_quota"("quota_used"); + +-- AddForeignKey +ALTER TABLE "public"."user_company_status" ADD CONSTRAINT "user_company_status_company_id_fkey" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE SET NULL ON UPDATE CASCADE; diff --git a/schema/prisma/schema/migrations/migration_lock.toml b/schema/prisma/schema/migrations/migration_lock.toml new file mode 100644 index 0000000..044d57c --- /dev/null +++ b/schema/prisma/schema/migrations/migration_lock.toml @@ -0,0 +1,3 @@ +# Please do not edit this file manually +# It should be added in your version-control system (e.g., Git) +provider = "postgresql" diff --git a/schema/prisma/schema/schema.prisma b/schema/prisma/schema/schema.prisma new file mode 100644 index 0000000..ef6f328 --- /dev/null +++ b/schema/prisma/schema/schema.prisma @@ -0,0 +1,28 @@ +/** +TODO: add this +def ensure_pg_trgm(connection): + if connection.dialect.name == "postgresql": + connection.execute(text("CREATE EXTENSION IF NOT EXISTS pg_trgm")) + + +def create_trgm_index(connection): + if connection.dialect.name == "postgresql": + connection.execute( + text( + """ + CREATE INDEX IF NOT EXISTS idx_companies_company_name_trgm + ON companies USING gin (LOWER(company_name) gin_trgm_ops); + """ + ) + ) +**/ + +generator client { + provider = "prisma-client-js" + output = "../generated/prisma" +} + +datasource db { + provider = "postgresql" + url = env("DATABASE_URL") +} diff --git a/schema/prisma/schema/user.prisma b/schema/prisma/schema/user.prisma new file mode 100644 index 0000000..9e02a3d --- /dev/null +++ b/schema/prisma/schema/user.prisma @@ -0,0 +1,31 @@ +enum Status { + NOT_DONE + TO_DO + DONE +} + +model UserCompanyStatus { + id Int @id @default(autoincrement()) + user_id String? + status Status? + company_id Int? + + // Relations + company Company? @relation(fields: [company_id], references: [id]) + + @@map("user_company_status") + @@index([user_id], map: "ix_user_company_status_user_id") + @@index([company_id], map: "ix_user_company_status_company_id") +} + +model UserQuota { + user_id String @id + quota_allocated Int? + quota_used Int? + + @@map("user_quota") + @@index([user_id], map: "ix_user_quota_user_id") + @@index([quota_allocated], map: "ix_user_quota_quota_allocated") + @@index([quota_used], map: "ix_user_quota_quota_used") +} + diff --git a/scripts/docker/docker-build-init.sh b/scripts/docker/docker-build-init.sh deleted file mode 100755 index d9da14c..0000000 --- a/scripts/docker/docker-build-init.sh +++ /dev/null @@ -1,57 +0,0 @@ -#!/bin/bash -set -e - -echo ">>> Initializing Postgres data at build time..." - -# Init database cluster -initdb -D /var/lib/postgresql/data - -# Prepend Docker subnet access to pg_hba.conf -# This ensures it is checked BEFORE any default scram-sha-256 lines -sed -i "1ihost all all 172.19.0.0/16 md5" /var/lib/postgresql/data/pg_hba.conf - -# Start Postgres on IPv4 localhost -pg_ctl -D /var/lib/postgresql/data \ - -o "-c listen_addresses='*' -c unix_socket_directories='/var/run/postgresql'" \ - -w start - -# Wait until Postgres is ready -until pg_isready -U postgres -h 127.0.0.1 -p 5432; do - echo "Waiting for Postgres to be ready..." - sleep 1 -done - -# Before running migrations / CSV load -psql -v ON_ERROR_STOP=1 --username=postgres <<-EOSQL - ALTER SYSTEM SET max_wal_size = '6GB'; - ALTER SYSTEM SET checkpoint_timeout = '30min'; - ALTER SYSTEM SET synchronous_commit = off; - ALTER SYSTEM SET fsync = off; - ALTER SYSTEM SET full_page_writes = off; -EOSQL -pg_ctl -D /var/lib/postgresql/data reload - -# Apply migrations -export DATABASE_URL=postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@127.0.0.1:5432/$POSTGRES_DB -cd /app/schema -PYTHONPATH=/app alembic upgrade head - -# Populate with CSV data -PYTHONPATH=/app python /app/app/database/database.py -cd .. -/app/scripts/load-csv-to-database.sh - -# Restore safer defaults -psql -v ON_ERROR_STOP=1 --username=postgres <<-EOSQL - ALTER SYSTEM RESET max_wal_size; - ALTER SYSTEM RESET checkpoint_timeout; - ALTER SYSTEM RESET synchronous_commit; - ALTER SYSTEM RESET fsync; - ALTER SYSTEM RESET full_page_writes; -EOSQL -pg_ctl -D /var/lib/postgresql/data reload - -# Stop Postgres -pg_ctl -D /var/lib/postgresql/data -m fast -w stop - -echo ">>> Database populated and baked into image." diff --git a/scripts/docker/run-migrations.sh b/scripts/docker/run-migrations.sh deleted file mode 100755 index 9025103..0000000 --- a/scripts/docker/run-migrations.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -# shellcheck disable=SC1091 -source ./scripts/util.sh - -log_info "Initializing the database." - -cd ./schema -alembic upgrade head - -log_info "Database initialized." \ No newline at end of file diff --git a/scripts/setup-db.sh b/scripts/setup-db.sh index 9616c66..ac7f422 100755 --- a/scripts/setup-db.sh +++ b/scripts/setup-db.sh @@ -8,17 +8,14 @@ docker-entrypoint.sh postgres & echo "Waiting for Postgres..." sleep 5 -export PGHOST=localhost -export PGUSER=postgres -export PGPASSWORD=root +export DATABASE_URL="postgresql://postgres:root@localhost:5432/postgres" # Test connection psql -d postgres -c "SELECT 1;" # Run migrations cd /app/schema -alembic upgrade head -PYTHONPATH=. python ./app/database/database.py +pnpm exec prisma db push cd /app psql -v ON_ERROR_STOP=1 --username="$POSTGRES_USER" <