Add MariaDB Vector DocumentStore integration#3565
Open
SyedShahmeerAli12 wants to merge 8 commits into
Open
Conversation
github-actions
Bot
added
topic:CI
integration:tavily
type:documentation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Contributor
Coverage report (tavily)
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
Implements MariaDBDocumentStore backed by MariaDB 11.7+ native VECTOR support with MHNSW indexing and full-text keyword search. - Full DocumentStore protocol: write_documents (FAIL/OVERWRITE/SKIP), filter_documents, delete_documents, count_documents - Vector similarity via VEC_DISTANCE_COSINE / VEC_DISTANCE_EUCLIDEAN - Full-text keyword search via MATCH ... AGAINST (NATURAL LANGUAGE MODE) - Haystack metadata filtering converted to JSON_EXTRACT SQL expressions - MariaDBEmbeddingRetriever and MariaDBKeywordRetriever with FilterPolicy - 80 tests: 68 unit + 12 integration (all verified against MariaDB 11.7) - GitHub Actions workflow with MariaDB 11.7 service container Closes deepset-ai#2340
- Make mariadb C extension import lazy so API reference builds without requiring libmariadb-dev in the docs environment - Fix Docker health check to use --su-mysql flag required by MariaDB 11.7 - Add mariadb (LGPL-2.1) to license compliance exclusion list
SyedShahmeerAli12
force-pushed
the
feat/mariadb-document-store-2340
branch
from
f71d1cd to
ff12b6a
Compare
- Add skip-install=true to default hatch env so docs build does not attempt to compile the mariadb C extension (libmariadb-dev not available in the API reference runner). The pydoc search_path already points to src/ so modules are importable without installation. - Switch service container health check from healthcheck.sh (unreliable in some MariaDB 11.7 images) to mysqladmin ping which is more robust.
…THCHECK The mariadb:11.7 Docker image ships with a HEALTHCHECK instruction. GitHub Actions automatically waits for it when no custom options override it. Custom health-cmd variants (healthcheck.sh, mysqladmin) were failing because the slim runner environment handles them differently.
Contributor
Author
|
there are Some CI failures i will check them Later Till then the rest of the code can be reviewed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #2340
Implements a complete MariaDB document store integration using MariaDB 11.7+ native
VECTORsupport.MariaDBDocumentStorefull DocumentStore protocol (write_documents,filter_documents,delete_documents,count_documents)VEC_DISTANCE_COSINE/VEC_DISTANCE_EUCLIDEANwithMHNSWindexingMATCH ... AGAINST (IN NATURAL LANGUAGE MODE)on aFULLTEXTindexJSON_UNQUOTE(JSON_EXTRACT(...))SQL expressions with parameterized queriesMariaDBEmbeddingRetrieverandMariaDBKeywordRetrieverwithFilterPolicysupportDuplicatePolicysupport:FAIL(INSERT),OVERWRITE(upsert viaON DUPLICATE KEY UPDATE),SKIP(INSERT IGNORE)Tests
80 tests total all passing:
CI
Added
.github/workflows/mariadb.ymlwith a MariaDB 11.7 service container, matching the pattern used bypgvector.How to run locally