Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request establishes a robust framework for benchmarking Named Entity Recognition (NER) capabilities of different large language models within the biomedical domain. It integrates a new dataset, a data processing utility, and a suite of evaluation results and analysis tools. The changes aim to provide a clear and quantifiable assessment of model performance, highlighting the impact of human-in-the-loop processes on NER accuracy and ontology alignment. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a new benchmarking dataset for StructSense, including a Python script (bio_txt.py) to convert BIO-tagged data into continuous text and JSONL formats. It also adds numerous evaluation results for Named Entity Recognition (NER) tasks, specifically for 'Integrating brainstem' and 'Latent-circuit' papers, comparing various LLMs (Claude 3.7 Sonnet, DeepSeek V3 0324, GPT-4o-mini) both with and without Human-in-the-Loop (HIL). These results are presented in new CSV and JSON files, detailing token usage, costs, speed, entity detection, ontology mapping, and judge scores. Additionally, new Python scripts (ner_comprehensive_summary.py, ner_data_loader.py, ner_entity_pool_analysis.py, ner_judge_score_analysis.py, ner_label_distribution.py, ner_location_analysis.py, ner_ontology_analysis.py) are added to analyze and visualize these NER evaluation metrics, along with a README.md for token cost analysis. Review comments highlight several issues: redundant nested judge_ner_terms keys in JSON output files, an inconsistency in schema definitions where paper_title is specified as a string but examples show it as an array, hardcoded file paths in analysis scripts, and extraneous \ No newline at end of file artifacts in CSV files.
Note: Security Review is unavailable for this PR.
...old/evaluation/Latent-circuit/without_hil/ner_config_gpt_s41593-025-01869-7_without_hil.json
Show resolved
Hide resolved
...on/Integrating-brainstem/without_hil/ner_config_deepseek_s41593-024-01787-0_without_hil.json
Show resolved
Hide resolved
...luation/Integrating-brainstem/without_hil/ner_config_gpt_s41593-024-01787-0_without_hil.json
Show resolved
Hide resolved
.../old/evaluation/Latent-circuit/with_hil/ner_config_deepseek_s41593-025-01869-7_with_hil.json
Show resolved
Hide resolved
| structsense_root = Path(__file__).parent.parent.parent | ||
| output_dir = structsense_root / "evaluation/ner/evaluation/Latent-circuit/results" |
There was a problem hiding this comment.
| structsense_root = Path(__file__).parent.parent.parent | ||
| output_dir = structsense_root / "evaluation/ner/evaluation/Latent-circuit/results" |
There was a problem hiding this comment.
| structsense_root = Path(__file__).parent.parent.parent | ||
| output_dir = structsense_root / "evaluation/ner/evaluation/Latent-circuit/results" |
There was a problem hiding this comment.
| "Jun 27, 12:53 PM",Favicon for OpenAI,GPT-4o-mini,liteLLM,1236,495,0.000482,47.0 tps,stop,"Multi-animal pose estimation, identification and tracking with DeepLabCut",Resource extraction,NO | ||
| "Jun 27, 12:53 PM",Favicon for OpenAI,GPT-4o-mini,liteLLM,300,262,0.000202,69.8 tps,stop,"Multi-animal pose estimation, identification and tracking with DeepLabCut",Resource extraction,NO | ||
| "Jun 27, 12:53 PM",Favicon for OpenAI,GPT-4o-mini,liteLLM,59016,634,0.00923,67.6 tps,stop,"Multi-animal pose estimation, identification and tracking with DeepLabCut",Resource extraction,NO | ||
| "Jun 27, 12:53 PM",Favicon for OpenAI,GPT-4o-mini,liteLLM,59239,262,0.00904,80.6 tps,stop,"Multi-animal pose estimation, identification and tracking with DeepLabCut",Resource extraction,NO No newline at end of file |
There was a problem hiding this comment.
The file ends with the text \ No newline at end of file. This appears to be an artifact from a version control or diff tool and should be removed. This extra text can cause parsing errors in tools that expect a clean CSV format.
"Jun 27, 12:53 PM",Favicon for OpenAI,GPT-4o-mini,liteLLM,59239,262,0.00904,80.6,tps,stop,"Multi-animal pose estimation, identification and tracking with DeepLabCut",Resource extraction,NO
|
@puja-trivedi could you confirm that you've tested and everything is working as expected? |
|
@tekrajchhetri - what is the stage of thi PR. |
|
|
And do we know what is the issue? At the beginning I understood that this is issue with passing the information form agent to another agent/output due to some keys mismatch etc. Is this the case here? Or possibly the agent is just not finding anything interesting? |
…ful for scientific articles
|
@tekrajchhetri - even after merging #65 this PR has 277 files changed. Does anything else should be merge first? Could you please create a PR that has only relevant changes? |
@djarecka the 277 file changes is coming from the Regarding the changes, some of the things are |
Yes, we do and they are:
|
|
Update: After PR 67 & 69 merge we no longer have 277 changes as before. |
|
@puja-trivedi @djarecka can you test other pdf from your side so that we can merge this? |
The parameter was renamed to input_source in the signature but the method body still referenced source, causing a TypeError when called from the CLI.
|
@tekrajchhetri - There is an inconsistency with the parameter naming |
Fix StructSenseFlow.__init__ parameter name: input_source -> source
|
Since @puja-trivedi is able to produce output, we can merge this PR to improvement and improvement to main. The question of less number of entities, depends on the model choice and other parameter--issues independent of output being able to save. Let me know if @djarecka @puja-trivedi have any concerns? |
|
@tekrajchhetri - the test stopped working due to the issue with poetry, can you please updat |
@djarecka can you share the logs because it's unclear what's causing the issue. Also, this means that we will always have an issue if test is failing just because we updated the dependencies. |
|
before merging this PR we should merge the following PR in order.
|
Resolving issue #84
This should be merged after #65 is merged