Fix slowness issue when performing codegraph context - #10
Closed
hungpham10 wants to merge 61 commits into
Closed
hungpham10 wants to merge 61 commits into
hungpham10 wants to merge 61 commits into
Conversation
…-index-algorithm Replace bfs with search index algorithm
* Remove old data structure * Benchmark * Remove temporary codegraph-viz and move benches into .github * Fix issue codspeed * Fix benchmark * Implement sandbox and diff * Add sandbox execution for diff and origin * Remove unused unittest * Fix lint * Remove unused unittest * Temporal remove another environments
* Implement new storage to improve performance * style: apply rustfmt * Fix lint * Fix issue multiple reading in multiple streams
* Finetune to reduce LLM token * style: apply rustfmt * Fix lint
…es (#6) * Implement MCP server for GA * Setup unit-tests to new storages * style: apply rustfmt * Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Add tests * Fix tests * Fix tests * Fix tests * Fix lint --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Implement new embed vector and similar search * style: apply rustfmt * Fix unit test * Add badges * Fix coverage
Add a security policy document outlining supported versions and vulnerability reporting.
* Truncate APIs and support graphql with mermaid * style: apply rustfmt
* Truncate APIs and support graphql with mermaid * Implement to support windows * style: apply rustfmt * Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Fix lint --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Implement flow to install codegraph to MacOS and linux * Add missing pipeline to support releasing * Fix issue with doctor * style: apply rustfmt
…rce code as possible (#13) * Bump version to v2.0.3 * Update ci * Update ci again * Remove coverage * Setup codecov.yml
* Improve code to be more maintainable * style: apply rustfmt * Fix lint
- Add support to build graph from binary using radare2 for reverse binary code - Fix issue when working with lambda function which is a core feature to analyze obfuscated code
* Fix issue missing function name and crash when picking to wrong function * Bump version to v2.1.1 * style: apply rustfmt
* Improve by showing more meaningful object naming from r2 * Increase code-coverage * style: apply rustfmt
* Implement tool to convert structure documents into graph * Bump version to v2.1.3
* Integrate codegraph-docs into main flow * style: apply rustfmt * Fix lint * style: apply rustfmt
* Persistent document graph into disk * Inplement nginx parser * style: apply rustfmt * Fix lint * style: apply rustfmt * Bump version to v2.1.5
* Split storage of binary graph into different database * Fix lint
* Support YAML multiple files * style: apply rustfmt * Bump version to v2.1.6
* Show progress bar while indexing documents * Improve performance when starting * Bump version to v2.1.7 * style: apply rustfmt * Fix lint
* Fix document graph pipeline and wire real pattern/value search The document tools were unusable end-to-end due to four latent bugs: - parsers: parent->children links were never persisted (nodes cloned into Document.nodes before children wiring), so hydrate could never descend. - graph: path token chains were built in reverse order (root ended up last), so full-path queries never matched; also node cache was materialized after trie insertion, so the first ingest interned no keys. - graph: the four trie projections shared one storage without namespace — radix root pointers collided per shard, leaving only the last-written trie reachable. Add shard_bias to Radix/Search (default 0) and give each docs trie a distinct shard range. - graph: the string interner was RAM-only while tries persisted, so interned token payloads dangled after restart. Persist the interner blob alongside docs and restore it in open(); doc ids move to their own id range (>=6e11) so they no longer collide with node ids. MCP/CLI wiring: - doc_search now parses dotted patterns into token chains via the interner, with a case-insensitive key-scan fallback for non full-path queries. - new tools: doc_search_value (scalar substring scan), doc_ingest_dir (recursive bulk ingest with limit), doc_remove. - doc_hydrate takes max_depth to keep LLM payloads small; doc_list returns per-doc metadata (doc_id, path, format, root_node_id, nodes) instead of bare counts; search results include key/index. - CLI: fix `doc ingest` clap panic (positional `path` clashed with the global --path arg, renamed to `file`); doc search uses the same real pattern resolution. * Bump version to v2.1.8 * style: apply rustfmt
#30) * Add fuzzy key match, pattern mining (P#) and IDF ranking to document graph Phase 3 of the structured document graph: fuzzy retrieval, cross-document pattern mining and rarity-based ranking (rare structure = high information, IDF-style scoring). - Fuzzy key match: search_key_fuzzy scores distinct keys by exact/prefix/ contains/Levenshtein similarity with an IDF bonus (rarer keys score higher). `doc_search` accepts `~segment` to force fuzzy and falls back exact-substring -> fuzzy when the full-path trie misses. - Pattern mining: mine_patterns() counts kind chains (wildcard FIELD/IDX payloads) ending at scalar leaves over a max_depth window, assigns stable pattern ids (P#, registry persisted in storage and restored on open) and indexes chains into the previously dead pattern_trie. Results carry node_count / doc_count / doc_freq, sorted by document frequency ascending so characteristic patterns surface first and background (~1.0) sinks. - Structural search: search_kind_chain() matches nodes whose ancestor kind window ends with the query chain (e.g. "MAP, FIELD, NUMBER"); results are ranked by pattern uniqueness IDF. search_path_scan() complements the radix trie whose leaf holds a single record per chain, returning all nodes with the same key path across documents (spec.replicas: 97 hits on the infra repo instead of 1). - Depth semantics fixed in doc_search: depth counts extra levels BELOW the pattern, so the radix key-length filter gets pattern_len + depth. - New MCP tools: doc_mine_patterns, doc_list_patterns, doc_search_struct. New CLI commands: `codegraph doc patterns`, `codegraph doc struct`. * style: apply rustfmt * Fix lint * Bump to version v2.1.9
#31) * Fix binary callees/callers/flow always empty: route id >= bin_base to BinaryGraph (#26 regression) After the storage split, binary symbols live only in binary.sqlite (id range bin_base = 2e9) while codegraph_callees/callers/flow still queried the main GraphIndex, so every binary returned empty results. Now those tools route to BinaryGraph when the node id falls in the binary range, with a fallback to the old path if the binary DB is unavailable. - BinaryGraph: add callees (call records resolved by name within the same binary), callers (BFS over a new caller_of:{name} reverse index built at ingest) and flow (chain render with CFG markers + call sites, same shape as GraphIndex::flow). - Ingest: stop corrupting chain entries — CFG markers (id < SYMBOL_BASE) and unresolved-call placeholders (0) are kept as-is; only real symbol ids are remapped into the bin_base range. - MCP: dispatch_binary_graph routes callees/callers/impact/flow for binary ids. - Config template: document the [bingraph] section (enabled/bin_base/storage) with a warning against overlapping id ranges. - Docs: binary-analysis.md updated for the split-storage architecture. - Tests: bingraph unit tests extended + end-to-end test compiling a real shared library and running it through r2 extraction and queries. * style: apply rustfmt
r2 sometimes fails to recover call ops for a shared library (entrypoint detection fails on Mach-O dylibs), producing chains without call sites. Retry extraction up to 3 times with the extract cache disabled and only accept a run whose chains contain at least one resolved call.
- codegraph_graphcode_*: class, list_types, function_scope, search_by_annotation, files, dependencies, sandbox, diff, diff_simulate, origin_simulate + new codegraph_graphcode_stats - codegraph_doc_* -> codegraph_graphdoc_* (all 11 tools) - codegraph_binary_list/addr/stats -> codegraph_graphbin_*; binary_search merged into codegraph_search_symbol via new `source` arg (all|code|binary) - Shared/session tools keep their names (symbol, search_symbol, callers, callees, impact, flow, context, references, mermaid, search_flow, init/deinit/index, query_usage_report) - codegraph_status now aggregates all three datasets (code + doc + binary, null when absent) - Minimize everywhere: doc/binary outputs routed through emit_value (omit_defaults); graphbin list/addr support `format` with fixed-order row arrays - Docs updated: server-instructions.md, README, architecture, binary-analysis
* Update graphql to support new models * Bump version to v2.2.1
* Add subcommand `clean` and remove subcommand `doc` * Bump version to v2.2.2
Owner
|
? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.