GraphAI and Celery

GraphAI and Celery

1. Objects shared across tasks

Shared objects are made available for tasks by passing them through the task decorator. A function will take care of initialising those objects that need some initialisation (graph, ontology, whisper model, caching system, etc.), and that function will be called from the on_event(”startup”) in the main FastAPI file. Thus we avoid instantiating them twice (one for the gunicorn process and another for the celery process) and also prevent the lazy loading to slow down the first request after startup, as the FastAPI application is not ready until the listener is done executing.

2. Master functions

Master functions are those that orchestrate celery tasks. By convention, we assume that the endpoint functions are master functions, which organise and dispatch the jobs, then return either a task_id if the endpoint is asynchronous or the actual result if it is synchronous.

3. Tasks

Tasks are modular pieces of code that run under the celery process. There are two types: API-related (caching, result handling, etc.) and API-agnostic (wikisearch, fingerprinting, etc.). Initially we thought that we could use this separation to place them under the api or core subpackages respectively. However, we decide to keep them all under api/celery_tasks for the moment.

4. Project structure

The current project structure has two main subpackages: api and core, and in their subpackages we often have a subpackages/submodules separation depending on the functionality (e.g. video, text, audio, etc.). Another approach would be to have a subpackage for each such functionality and everything related to it (router, schemas, etc.) inside it. Here’s a link explaining this second approach. The idea would be to go from

/graphai
  /api
    /common
    /celery_tasks
      /ontology.py
      /text.py
      /video.py
      ...
    /routers
      /ontology.py
      /text.py
      /video.py
      ...
    /schemas
      /ontology.py
      /text.py
      /video.py
      ...
  /core
    /common
    /ontology
    /text
    /video
    ...

to something like

/graphai
  /common
    ...
  /ontology
    /tasks.py
    /router.py
    /schemas.py
    ...
  /text
    /tasks.py
    /router.py
    /schemas.py
    ...
  /video
    /tasks.py
    /router.py
    /schemas.py
    ...
  /main.py

However, we postpone the decision of refactoring the whole project until we have a more stable version.