diff --git a/README.md b/README.md index 389abe5e..e0595d2d 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ The seed database of the `embabel/hub` image is kept in step by the Export Seed ## Switching the Graph Database -`GraphObjectManagerStore` (from `embabel-agent-rag-graph`) supports four backends: **Neo4j** (default), **FalkorDB**, **Memgraph** and **Oracle 23ai**. The active backend is selected by Spring profile in `src/main/resources/application.yml` — each profile sets `database.dataSources.neo.type`, and Drivine speaks to whichever engine that names. +`GraphObjectManagerStore` (from `embabel-agent-rag-graph`) supports five backends: **Neo4j** (default), **FalkorDB**, **Memgraph**, **Oracle 23ai** and **Postgres**. The active backend is selected by Spring profile in `src/main/resources/application.yml` — each profile sets `database.dataSources.neo.type`, and Drivine speaks to whichever engine that names. | Backend | Profile | Default port | Compose profile | |----------|------------|--------------|-----------------| @@ -94,8 +94,9 @@ The seed database of the `embabel/hub` image is kept in step by the Export Seed | FalkorDB | `falkordb` | `6379` | `falkordb` | | Memgraph | `memgraph` | `7688` (bolt)| `memgraph` | | Oracle 23ai ![New](https://img.shields.io/badge/NEW-ff3e00?style=flat-square) | `oracle` | `1521` | `oracle` | +| Postgres ![New](https://img.shields.io/badge/NEW-ff3e00?style=flat-square) | `postgres` | `5432` | `postgres` | -[embabel-chat-store](https://github.com/embabel/embabel-chat-store) and [embabel-agent-rag-graph](https://github.com/embabel/embabel-agent-rag-graph) (see [`GraphObjectManagerStore`](https://github.com/embabel/embabel-agent-rag-graph/blob/main/src/main/kotlin/com/embabel/agent/rag/graph/GraphObjectManagerStore.kt)) use Drivine's `StatelessGraphObjectManager`, which generates openCypher in each engine's dialect, or SQL/PGQ for Oracle. +[embabel-chat-store](https://github.com/embabel/embabel-chat-store) and [embabel-agent-rag-graph](https://github.com/embabel/embabel-agent-rag-graph) (see [`GraphObjectManagerStore`](https://github.com/embabel/embabel-agent-rag-graph/blob/main/src/main/kotlin/com/embabel/agent/rag/graph/GraphObjectManagerStore.kt)) use Drivine's `StatelessGraphObjectManager`, which generates openCypher in each engine's dialect, SQL/PGQ for Oracle, or plain SQL for Postgres. ### Switching at startup @@ -113,12 +114,15 @@ Change `spring.profiles.active` in `application.yml`, or override at launch: # Oracle ./mvnw spring-boot:run -Dspring-boot.run.profiles=oracle + +# Postgres +./mvnw spring-boot:run -Dspring-boot.run.profiles=postgres ``` ### Starting the matching container ```bash -docker compose --profile neo4j up -d # or falkordb / memgraph / oracle +docker compose --profile neo4j up -d # or falkordb / memgraph / oracle / postgres ``` Memgraph maps host port `7688` → container `7687` to avoid clashing with Neo4j, so you can run them side-by-side. FalkorDB uses Redis protocol on `6379`. @@ -135,6 +139,18 @@ Oracle runs from `gvenzl/oracle-free:23-faststart` and wants about 2 GB of memor The user is created only when the `oracle_data` volume is new: after changing the username or password, stop the container and remove its volume (`docker rm -f embabel-oracle && docker volume rm guide_oracle_data`, which deletes the Oracle data), or create the user yourself. `docker compose down -v` would remove the Neo4j volume as well. Guide makes its tables on startup, and the RAG store is filled by an ingest as on any other backend. +Postgres runs from `pgvector/pgvector:pg17`: Postgres 17 with the pgvector extension, which vector search uses. On first start the image creates the user and its database, and the `postgres` Spring profile connects as that user. On startup Guide makes its tables in that database, and the `vector` and `pg_trgm` extensions (pg_trgm backs full-text search). The profile reads all of these; compose reads the port, username, password and database: + +| Variable | Default | +|---------------------|-------------| +| `POSTGRES_HOST` | `localhost` | +| `POSTGRES_PORT` | `5432` | +| `POSTGRES_USERNAME` | `guide` | +| `POSTGRES_PASSWORD` | `brahmsian` | +| `POSTGRES_DATABASE` | `guide` | + +As with Oracle, the user and database are created only when the `postgres_data` volume is new: after changing them, stop the container and remove its volume (`docker rm -f embabel-postgres && docker volume rm guide_postgres_data`, which deletes the Postgres data), or create them yourself. + ### Browsing data - **Neo4j**: http://localhost:7474/browser/ (user `neo4j`, password `brahmsian`) @@ -149,6 +165,15 @@ The user is created only when the `oracle_data` volume is new: after changing th ```sql select count(*) from drivine_node where json_exists(labels, '$[*]?(@ == "Chunk")'); ``` +- **Postgres**: psql in the container. The tables are as on Oracle, with `labels` and `props` as `jsonb`: + + ```bash + docker exec -it embabel-postgres psql -U guide -d guide + ``` + + ```sql + select count(*) from drivine_node where labels @> '["Chunk"]'; + ``` ## Viewing and Deleting Data diff --git a/codegen-gradle/build.gradle.kts b/codegen-gradle/build.gradle.kts index 910f6078..cc2eda8f 100644 --- a/codegen-gradle/build.gradle.kts +++ b/codegen-gradle/build.gradle.kts @@ -19,7 +19,7 @@ plugins { group = "com.embabel.guide" version = "0.1.0-SNAPSHOT" -val drivineVersion = "0.1.6" +val drivineVersion = "0.2.0" repositories { mavenCentral() diff --git a/compose.yaml b/compose.yaml index 346bbd41..5833b6ec 100644 --- a/compose.yaml +++ b/compose.yaml @@ -106,6 +106,28 @@ services: networks: - embabel-network + # Postgres 17 with pgvector, for the `postgres` Spring profile. The image makes the user and its database. + postgres: + profiles: + - postgres + image: pgvector/pgvector:${POSTGRES_VERSION:-pg17} + container_name: embabel-postgres + ports: + - "${POSTGRES_PORT:-5432}:5432" + environment: + - POSTGRES_USER=${POSTGRES_USERNAME:-guide} + - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-brahmsian} + - POSTGRES_DB=${POSTGRES_DATABASE:-guide} + volumes: + - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"] + interval: 10s + timeout: 5s + retries: 20 + networks: + - embabel-network + memgraph: profiles: - memgraph @@ -159,6 +181,8 @@ volumes: driver: local oracle_data: driver: local + postgres_data: + driver: local memgraph_data: driver: local memgraph_log: diff --git a/pom.xml b/pom.xml index 64a62115..103be29e 100644 --- a/pom.xml +++ b/pom.xml @@ -74,8 +74,9 @@ org.drivine drivine4j-spring-boot-starter - 0.1.6 + in codegen-gradle/build.gradle.kts: code compiled against one Drivine minor line need not link + with another (0.0.x does not link with 0.1.x). --> + 0.2.0 @@ -85,6 +86,13 @@ runtime + + + org.postgresql + postgresql + runtime + + diff --git a/src/main/java/com/embabel/guide/rag/RagConfiguration.java b/src/main/java/com/embabel/guide/rag/RagConfiguration.java index 5807610a..3fe155e8 100644 --- a/src/main/java/com/embabel/guide/rag/RagConfiguration.java +++ b/src/main/java/com/embabel/guide/rag/RagConfiguration.java @@ -80,7 +80,7 @@ HierarchicalContentReader hierarchicalContentReader(GuideProperties guidePropert /** * The RAG store, on Drivine's stateless object manager. It is the one implementation - * embabel-agent-rag-graph ships, and runs on every engine Drivine supports, Oracle included. + * embabel-agent-rag-graph ships, and runs on every engine Drivine supports, Oracle and Postgres included. */ @Bean @Primary diff --git a/src/main/kotlin/com/embabel/guide/migration/RelationshipRepairRunner.kt b/src/main/kotlin/com/embabel/guide/migration/RelationshipRepairRunner.kt index d770439c..f143c13d 100644 --- a/src/main/kotlin/com/embabel/guide/migration/RelationshipRepairRunner.kt +++ b/src/main/kotlin/com/embabel/guide/migration/RelationshipRepairRunner.kt @@ -45,7 +45,7 @@ import org.springframework.stereotype.Component * `guide.repair.fields` (`View.field`, comma separated). An ambiguous finding is not repaired. * * Run it once against each database, `report` first, after backing the store up. The tools are - * written in Cypher and have nothing to repair on Oracle. + * written in Cypher and have nothing to repair on Oracle or Postgres. */ @Component @ConditionalOnProperty(name = ["guide.repair.mode"]) diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml index ad73721f..4c60a2e5 100644 --- a/src/main/resources/application.yml +++ b/src/main/resources/application.yml @@ -118,7 +118,7 @@ spring: ansi: enabled: always profiles: - active: neo4j # Change to 'neo4j', 'falkordb', 'memgraph', 'oracle', or 'default' for TestContainers + active: neo4j # Change to 'neo4j', 'falkordb', 'memgraph', 'oracle', 'postgres', or 'default' for TestContainers config: activate: on-profile: neo4j @@ -249,3 +249,23 @@ database: userName: ${ORACLE_USERNAME:guide} password: ${ORACLE_PASSWORD:brahmsian} databaseName: ${ORACLE_SERVICE:FREEPDB1} + +--- +# Postgres profile — activate with: spring.profiles.active=postgres +# Postgres 14 or later, with pgvector for vector search (docker compose --profile postgres up). +# Drivine reads the graph with plain SQL and makes its tables in this database. The datasource keeps +# the name 'neo', which the beans are qualified by. +spring: + config: + activate: + on-profile: postgres + +database: + dataSources: + neo: + type: POSTGRES + host: ${POSTGRES_HOST:localhost} + port: ${POSTGRES_PORT:5432} + userName: ${POSTGRES_USERNAME:guide} + password: ${POSTGRES_PASSWORD:brahmsian} + databaseName: ${POSTGRES_DATABASE:guide} diff --git a/src/test/kotlin/com/embabel/guide/domain/GuideUserServicePostgresTest.kt b/src/test/kotlin/com/embabel/guide/domain/GuideUserServicePostgresTest.kt new file mode 100644 index 00000000..a44927df --- /dev/null +++ b/src/test/kotlin/com/embabel/guide/domain/GuideUserServicePostgresTest.kt @@ -0,0 +1,7 @@ +package com.embabel.guide.domain + +import org.springframework.test.context.ActiveProfiles + +/** [GuideUserServiceTest] on Postgres, which Drivine reads with plain SQL and pgvector. */ +@ActiveProfiles("test-postgres") +class GuideUserServicePostgresTest : GuideUserServiceTest() diff --git a/src/test/kotlin/com/embabel/guide/domain/GuideUserServiceTest.kt b/src/test/kotlin/com/embabel/guide/domain/GuideUserServiceTest.kt index 1a389aed..b2aade26 100644 --- a/src/test/kotlin/com/embabel/guide/domain/GuideUserServiceTest.kt +++ b/src/test/kotlin/com/embabel/guide/domain/GuideUserServiceTest.kt @@ -31,7 +31,7 @@ import org.springframework.transaction.annotation.Transactional @ContextConfiguration(initializers = [Neo4jPropertiesInitializer::class]) @ImportAutoConfiguration(exclude = [McpClientAutoConfiguration::class]) @Transactional -class GuideUserServiceTest { +open class GuideUserServiceTest { @Autowired private lateinit var guideUserService: GuideUserService diff --git a/src/test/kotlin/com/embabel/guide/domain/drivine/GuideUserRepositoryDefaultImplPostgresTest.kt b/src/test/kotlin/com/embabel/guide/domain/drivine/GuideUserRepositoryDefaultImplPostgresTest.kt new file mode 100644 index 00000000..cb2b580b --- /dev/null +++ b/src/test/kotlin/com/embabel/guide/domain/drivine/GuideUserRepositoryDefaultImplPostgresTest.kt @@ -0,0 +1,7 @@ +package com.embabel.guide.domain.drivine + +import org.springframework.test.context.ActiveProfiles + +/** [GuideUserRepositoryDefaultImplTest] on Postgres, which Drivine reads with plain SQL and pgvector. */ +@ActiveProfiles("test-postgres") +class GuideUserRepositoryDefaultImplPostgresTest : GuideUserRepositoryDefaultImplTest() diff --git a/src/test/kotlin/com/embabel/guide/domain/drivine/GuideUserRepositoryDefaultImplTest.kt b/src/test/kotlin/com/embabel/guide/domain/drivine/GuideUserRepositoryDefaultImplTest.kt index 34d8c605..7f515367 100644 --- a/src/test/kotlin/com/embabel/guide/domain/drivine/GuideUserRepositoryDefaultImplTest.kt +++ b/src/test/kotlin/com/embabel/guide/domain/drivine/GuideUserRepositoryDefaultImplTest.kt @@ -42,7 +42,7 @@ import java.util.* @ContextConfiguration(initializers = [Neo4jPropertiesInitializer::class]) @ImportAutoConfiguration(exclude = [McpClientAutoConfiguration::class]) @Transactional -class GuideUserRepositoryDefaultImplTest { +open class GuideUserRepositoryDefaultImplTest { @Autowired private lateinit var repository: GuideUserRepositoryDefaultImpl diff --git a/src/test/kotlin/com/embabel/guide/domain/drivine/PersonaRepositoryImplPostgresTest.kt b/src/test/kotlin/com/embabel/guide/domain/drivine/PersonaRepositoryImplPostgresTest.kt new file mode 100644 index 00000000..6b351afa --- /dev/null +++ b/src/test/kotlin/com/embabel/guide/domain/drivine/PersonaRepositoryImplPostgresTest.kt @@ -0,0 +1,7 @@ +package com.embabel.guide.domain.drivine + +import org.springframework.test.context.ActiveProfiles + +/** [PersonaRepositoryImplTest] on Postgres, which Drivine reads with plain SQL and pgvector. */ +@ActiveProfiles("test-postgres") +class PersonaRepositoryImplPostgresTest : PersonaRepositoryImplTest() diff --git a/src/test/kotlin/com/embabel/guide/domain/drivine/PersonaRepositoryImplTest.kt b/src/test/kotlin/com/embabel/guide/domain/drivine/PersonaRepositoryImplTest.kt index c2891d1b..8a9366de 100644 --- a/src/test/kotlin/com/embabel/guide/domain/drivine/PersonaRepositoryImplTest.kt +++ b/src/test/kotlin/com/embabel/guide/domain/drivine/PersonaRepositoryImplTest.kt @@ -22,7 +22,7 @@ import java.util.UUID @ContextConfiguration(initializers = [Neo4jPropertiesInitializer::class]) @ImportAutoConfiguration(exclude = [McpClientAutoConfiguration::class]) @Transactional -class PersonaRepositoryImplTest { +open class PersonaRepositoryImplTest { @Autowired private lateinit var repository: PersonaRepositoryImpl diff --git a/src/test/kotlin/com/embabel/guide/rag/DataManagerLoadReferencesIntegrationTest.kt b/src/test/kotlin/com/embabel/guide/rag/DataManagerLoadReferencesIntegrationTest.kt index 9d88b50f..43dbd241 100644 --- a/src/test/kotlin/com/embabel/guide/rag/DataManagerLoadReferencesIntegrationTest.kt +++ b/src/test/kotlin/com/embabel/guide/rag/DataManagerLoadReferencesIntegrationTest.kt @@ -44,7 +44,7 @@ import org.springframework.test.context.TestPropertySource "guide.directories[0]=./src/test/resources/sample-repo-for-ingestion" ] ) -class DataManagerLoadReferencesIntegrationTest { +open class DataManagerLoadReferencesIntegrationTest { @Autowired private lateinit var dataManager: DataManager diff --git a/src/test/kotlin/com/embabel/guide/rag/DataManagerLoadReferencesPostgresTest.kt b/src/test/kotlin/com/embabel/guide/rag/DataManagerLoadReferencesPostgresTest.kt new file mode 100644 index 00000000..1c5cc551 --- /dev/null +++ b/src/test/kotlin/com/embabel/guide/rag/DataManagerLoadReferencesPostgresTest.kt @@ -0,0 +1,7 @@ +package com.embabel.guide.rag + +import org.springframework.test.context.ActiveProfiles + +/** [DataManagerLoadReferencesIntegrationTest] on Postgres, which Drivine reads with plain SQL and pgvector. */ +@ActiveProfiles("test-postgres") +class DataManagerLoadReferencesPostgresTest : DataManagerLoadReferencesIntegrationTest() diff --git a/src/test/kotlin/com/embabel/guide/stats/GuideStatsServicePostgresTest.kt b/src/test/kotlin/com/embabel/guide/stats/GuideStatsServicePostgresTest.kt new file mode 100644 index 00000000..40f7df12 --- /dev/null +++ b/src/test/kotlin/com/embabel/guide/stats/GuideStatsServicePostgresTest.kt @@ -0,0 +1,7 @@ +package com.embabel.guide.stats + +import org.springframework.test.context.ActiveProfiles + +/** [GuideStatsServiceTest] on Postgres, which Drivine reads with plain SQL and pgvector. */ +@ActiveProfiles("test-postgres") +class GuideStatsServicePostgresTest : GuideStatsServiceTest() diff --git a/src/test/kotlin/com/embabel/guide/stats/GuideStatsServiceTest.kt b/src/test/kotlin/com/embabel/guide/stats/GuideStatsServiceTest.kt index d5982427..cdb04269 100644 --- a/src/test/kotlin/com/embabel/guide/stats/GuideStatsServiceTest.kt +++ b/src/test/kotlin/com/embabel/guide/stats/GuideStatsServiceTest.kt @@ -31,7 +31,7 @@ import java.util.UUID @ContextConfiguration(initializers = [Neo4jPropertiesInitializer::class]) @ImportAutoConfiguration(exclude = [McpClientAutoConfiguration::class]) @Transactional -class GuideStatsServiceTest { +open class GuideStatsServiceTest { @Autowired private lateinit var statsService: GuideStatsService diff --git a/src/test/resources/application-test-postgres.yml b/src/test/resources/application-test-postgres.yml new file mode 100644 index 00000000..a6cbe327 --- /dev/null +++ b/src/test/resources/application-test-postgres.yml @@ -0,0 +1,10 @@ +# The test datasource, on Postgres. Active beside `test` in the *PostgresTest classes. +# @EnableDrivineTestConfig starts Postgres with pgvector in a container and connects as its user to its database. +database: + datasources: + neo: + type: POSTGRES + port: 5432 + user-name: drivine + password: drivine + database-name: drivine