Sub task/post models model id embedding processes core 556

[irumvanselme]

Fixes

• CORE-556

[Copilot]

Pull request overview

Adds backend + API-spec support for triggering an embeddings generation process for a released model (CORE-556), including queue job validation and persistence of an embedding-process state.

Changes:

• Introduces POST /models/{modelId}/embedding-processes handler + OpenAPI wiring, including schema validation and role-based access.
• Adds embeddings queue job schema/validation used by both the producer (SQS sender) and consumer (embeddings lambda).
• Adds persistence layer for EmbeddingProcessState and registers it in repository/service registries.

Reviewed changes

Copilot reviewed 60 out of 61 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
backend/yarn.lock	Locks new AWS SQS SDK dependency.
backend/package.json	Adds @aws-sdk/client-sqs dependency.
backend/src/validator.ts	Registers AJV schema for the embedding-process POST request payload.
backend/src/server/httpUtils.ts	Adds HTTP 409 CONFLICT status code.
backend/src/server/repositoryRegistry/repositoryRegistry.ts	Registers EmbeddingProcessStateRepository and creates indexes.
backend/src/server/repositoryRegistry/repositoryRegistry.test.ts	Asserts embeddingProcessState repository is initialized.
backend/src/server/serviceRegistry/serviceRegistry.ts	Wires EmbeddingProcessService + SQS client into service registry.
backend/src/server/serviceRegistry/serviceRegistry.test.ts	Tests SQS client construction and service initialization.
backend/src/index.ts	Routes /models/{id}/embedding-processes to the new handler.
backend/src/modelInfo/embeddingProcesses/POST/index.ts	Implements POST handler: validation, auth, service call, error mapping.
backend/src/modelInfo/embeddingProcesses/POST/index.test.ts	Unit tests for POST handler responses and error cases.
backend/src/modelInfo/embeddingProcesses/POST/index.integration.test.ts	Integration test with in-memory DB + mocked SQS send asserting queued jobs.
backend/src/modelInfo/embeddingProcesses/POST/transform.ts	Maps persisted embedding-process state to API response payload.
backend/src/modelInfo/embeddingProcesses/POST/transform.test.ts	Tests transform output + required properties.
backend/src/modelInfo/embeddingProcesses/index.test.ts	Tests dispatching to the POST handler.
backend/src/embeddings/specs/queueJob.schema.ts	Adds backend-only AJV schema + validator for embedding queue jobs.
backend/src/embeddings/specs/queueJob.schema.test.ts	Tests queue job schema validation rules.
backend/src/embeddings/service/types.ts	Updates task type to fields: EmbeddableField[].
backend/src/embeddings/service/client.ts	Implements SQS-based embedding task producer with schema validation.
backend/src/embeddings/service/client.test.ts	Tests client sends messages, rejects invalid jobs, wraps send failures.
backend/src/embeddings/service/service.test.ts	Adds basic test coverage for EmbeddingService.processTask.
backend/src/embeddings/models/gemini/geminiService.test.ts	Adds tests for Gemini embedding service configuration/behavior.
backend/src/embeddings/handler/index.ts	Validates queue jobs on the consumer side and skips invalid records.
backend/src/embeddings/handler/index.test.ts	Tests consumer behavior for valid/invalid/unparseable SQS records.
backend/src/embeddings/embeddingProcessState/embeddingProcessState.types.ts	Defines types/specs for embedding-process state persistence.
backend/src/embeddings/embeddingProcessState/embeddingProcessStateModel.ts	Adds mongoose schema/model for embedding-process state.
backend/src/embeddings/embeddingProcessState/embeddingProcessStateModel.test.ts	Tests mongoose schema validation + toObject transform behavior.
backend/src/embeddings/embeddingProcessState/embeddingProcessStateRepository.ts	Adds repository methods (create/update/find/findPending).
backend/src/embeddings/embeddingProcessState/embeddingProcessStateRepository.test.ts	Tests repository CRUD behavior with in-memory MongoDB.
backend/src/embeddings/embeddingProcess/errors.ts	Adds embedding-process domain errors.
backend/src/embeddings/embeddingProcess/embeddingProcess.service.types.ts	Defines embedding-process service interface.
backend/src/embeddings/embeddingProcess/embeddingProcess.service.ts	Implements embedding-process orchestration: checks, persist state, enqueue tasks.
backend/src/embeddings/embeddingProcess/embeddingProcess.service.test.ts	Unit tests for service orchestration and error handling.
backend/openapi/generateOpenApiDoc.ts	Registers embedding-process schemas and endpoint-specific error schemas.
api-specifications/src/modelInfo/index.ts	Exposes instance-level ModelInfo namespace (via [id]).
api-specifications/src/modelInfo/[id]/index.ts	Adds [id] concept aggregator for model instance APIs.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/index.ts	Adds embedding-process-states concept aggregator.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/index.test.ts	Tests module export + snapshot.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/enums.ts	Defines embedding-process status enum.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/POST/index.ts	Defines POST operation exports (schemas/types/enums/constants).
api-specifications/src/modelInfo/[id]/embeddingProcessStates/POST/types.ts	Defines request/response types for POST.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/POST/schema.request.ts	Defines request JSON schema for POST.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/POST/schema.request.test.ts	Tests request schema validation.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/POST/schema.response.ts	Defines response JSON schema for POST.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/POST/schema.response.test.ts	Tests response schema validation.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/POST/enums.ts	Defines POST-specific status + error codes enums.
api-specifications/src/modelInfo/[id]/embeddingProcessStates/POST/constants.ts	Defines POST constants (payload length).
api-specifications/src/modelInfo/[id]/embeddingProcessStates/snapshots/index.test.ts.snap	Snapshot for embedding-process-states module exports.
api-specifications/src/modelInfo/snapshots/index.test.ts.snap	Updates modelInfo snapshot to include instance-level exports.
api-specifications/src/error/schema.ts	Extends allowed error codes with embedding-process error codes.
api-specifications/src/error/schema.test.ts	Extends error-schema tests for the new error codes.
api-specifications/src/error/snapshots/index.test.ts.snap	Updates error module snapshot with new codes.
api-specifications/src/embeddings/package.json	Adds embeddings module package entrypoint.
api-specifications/src/embeddings/index.ts	Introduces embeddings API-spec module (constants + schema).
api-specifications/src/embeddings/index.test.ts	Tests embeddings module import + snapshot.
api-specifications/src/embeddings/constants.ts	Adds embedding-model registry constants.
api-specifications/src/embeddings/constants.test.ts	Tests constants registry uniqueness + snapshots.
api-specifications/src/embeddings/schema.ts	Adds embedding-model JSON schema.
api-specifications/src/embeddings/schema.test.ts	Tests embedding-model schema + validates registered models.
api-specifications/src/embeddings/snapshots/index.test.ts.snap	Snapshot for embeddings module exports.
api-specifications/src/embeddings/snapshots/constants.test.ts.snap	Snapshot for embeddings constants.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[musayann]

This validation implementation is repeated multiple times across different files. I would suggest to convert it into a helper function such such validateEvent then it can be reused everywhere instead of repeating the same behaviour

[musayann]

same for this schema validation I think we should perhaps create another task for cleaning up this but we should have something like validateSchema that would receive the schema as a parameter and the payload and it would do the the whole validation

[sentry]

Bug: The query in findPendingByModelId incorrectly uses a bare array on the scalar status field. This relies on undocumented Mongoose/driver behavior instead of the standard $in operator.
_{Severity: MEDIUM}

Suggested Fix

Refactor the query to use the explicit and standard $in operator to ensure correct and maintainable behavior. The query filter should be changed to { status: { $in: [EmbeddingProcessStateStatus.PENDING, EmbeddingProcessStateStatus.IN_PROGRESS] } }.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location:
backend/src/embeddings/embeddingProcessState/embeddingProcessStateRepository.ts#L127-L135

Potential issue: The `findPendingByModelId` function constructs a MongoDB query using a
bare array `[PENDING, IN_PROGRESS]` for the `status` field. However, the `status` field
is a scalar string, not an array. According to MongoDB documentation, a bare array query
on a scalar field attempts an exact match for the entire array, which will never find a
document with a single string status. While tests currently pass, this is likely due to
undocumented Mongoose or driver-level query rewriting. Relying on this non-standard
behavior is semantically incorrect and creates a maintenance risk, as future library
updates could break this functionality.