Implement multilingual support for lift ranges editor#21
Draft
google-labs-jules[bot] wants to merge 117 commits intomainfrom
Draft
Implement multilingual support for lift ranges editor#21google-labs-jules[bot] wants to merge 117 commits intomainfrom
google-labs-jules[bot] wants to merge 117 commits intomainfrom
Conversation
…o ranges-loader.js Co-authored-by: milekpl <1446470+milekpl@users.noreply.github.com>
…5-17c1d257977f Fix JavaScript error: Add missing populateSelectWithFallback method to ranges-loader.js
…ntries - Added a policy ensuring that invalid entries can always be loaded for viewing and editing. - Updated validation rules to display errors as guidance rather than blocking editing. - Enhanced user experience by ensuring invalid entries appear in search results and entry lists. test(editing): Add comprehensive tests for invalid entry editability - Created tests to verify that invalid entries can be loaded for editing without validation blocking. - Ensured that validation errors are shown as guidance, not barriers, during editing. - Verified that invalid entries are included in search results. fix(validation): Adjust validation rules for variant entries - Updated validation rules to allow variant entries without senses. - Ensured that regular entries still require at least one sense for validation. - Implemented detailed tests to confirm the correct behavior of variant entry validation. debug(validation): Added debugging scripts for entry validation and JSONPath behavior - Introduced scripts to debug entry instantiation and validation processes. - Added tests to understand validation errors and ensure proper handling of invalid entries.
…tton to header Co-authored-by: milekpl <1446470+milekpl@users.noreply.github.com>
…8-7aa92a70f52c Fix Etymology section visual consistency: add tooltip and move Add button to header
This commit introduces a mechanism to generate a static Markdown document (`API_DOCUMENTATION.md`) that lists all Flask API endpoints, their HTTP methods, and the specific source file and line number where each endpoint handler function is defined. Includes: - `inspect_routes.py`: A script to inspect the Flask app and extract route definitions using `app.url_map` and the `inspect` module. - `generate_endpoint_doc.py`: A script to process the JSON output from `inspect_routes.py` and generate the Markdown documentation. - `API_DOCUMENTATION.md`: The generated static documentation file.
feat: Create static API documentation with endpoint source locations
Modified app/static/js/entry-form.js to ensure that creating and updating entries is done via the dedicated JSON API endpoints: - POST /api/entries/ for creating new entries. - PUT /api/entries/<entry_id> for updating existing entries. This change addresses an issue where the form might have been submitting to general-purpose view routes that could return HTML instead of JSON, leading to save operation failures on the client-side. The JavaScript now correctly determines the appropriate API endpoint and HTTP method (POST/PUT) based on whether an entry ID is present, and handles the expected JSON responses accordingly.
- Created app/data/languages.yaml for a comprehensive, user-editable list of languages. - Installed PyYAML. - Updated app.utils.language_utils.load_available_languages to parse languages.yaml, with caching and error handling. - SettingsForm now dynamically populates language choices from this YAML file. - Added Select2 CDN links to settings.html and initialized Select2 for searchable language dropdowns. - Updated tests for settings route to attempt mocking of dynamic language loading. - Fixed Markup import in language_utils (from markupsafe import Markup). - Ensured app/utils and app/forms are treated as packages by adding __init__.py files. Note: Test failures persist for test_settings_route.py related to proving the mock of load_available_languages is called. This seems to be a complex interaction with unittest.mock.patch and module import timing in the test environment. The core application logic for loading and using these languages is expected to work. Other tests related to ConfigManager pass. Many other pre-existing tests in the suite are failing due to missing dependencies (pytest, requests) or BaseX connection issues, which are outside the scope of this task.
…structure - Implemented `getFieldValidationState` method in `ValidationUI` class to retrieve the validation state of a field by its ID. - Modified entry form HTML to change the form action to `javascript:void(0);` for better handling of form submissions. - Cleaned up the senses section by removing unnecessary comments and restructuring the sense item display for improved usability. - Updated JavaScript initialization for various managers (Pronunciation, Variant Forms, Relations, Etymology, Multilingual Notes) to streamline their setup and ensure proper functionality. - Enhanced drag-and-drop functionality for senses with a dedicated drag handle and improved cursor feedback during dragging.
…in entry form - Updated entry_form.html to replace single input fields with multilingual support for definitions and glosses. - Added functionality to dynamically add and remove language-specific inputs. - Enhanced form serialization to handle complex nested data structures using Web Workers. - Improved form submission process with progress indicators and better error handling. - Created multilingual-sense-fields.js to manage multilingual input fields. - Added form-serializer-worker.js for background processing of form serialization. - Updated documentation to reflect changes in form structure and functionality.
…g sense reordering functionality
…age handling - Updated name attributes in multilingual sense fields to use nested structure for definitions and glosses. - Enhanced language selection logic to include non-project languages with appropriate labeling. - Improved HTML generation for language options to ensure original languages are correctly displayed. - Added tests to verify that non-project languages appear correctly in the edit form. - Cleaned up and organized test cases for better readability and maintainability. - Removed outdated refactoring specification document. - Added repository rules for project consistency and development practices.
- Updated test_multilingual_editing.py to handle multilingual notes as nested dictionaries. - Modified test_postgresql_real_integration.py to accommodate changes in database schema for multilingual fields. - Adjusted test_real_integration.py to ensure proper handling of multilingual entries and senses. - Enhanced test_serialization_error.py with logging for better debugging of serialization issues. - Added unit tests for parseFieldPath to validate field name parsing logic. - Created a progress log for multilingual refactor in multilang.md. - Implemented a minimal setup_postgresql.py for test discovery. - Introduced test_backend_dict_field_error.py to validate error logging for malformed dictionary fields. - Developed test_lift_parser_senses.py to ensure correct parsing of LIFT entries with senses.
…ize realistic LIFT XML parsing. Update test cases to support multilingual lexical units and ensure proper handling of Unicode characters in JavaScript initialization. Enhance validation rules to accommodate multilanguage structures and improve error messaging for various validation scenarios. Add a TODO file outlining outstanding project issues and feature enhancements.
- Implemented integration tests for working coverage on database connectors, search integration, and parser modules in `test_working_coverage.py`. - Created integration tests for workset management API in `test_workset_api.py`, covering creation, retrieval, updating, deletion, and performance requirements. - Added unit tests for etymology UI functionality in `test_etymology_ui_unit.py`, focusing on object creation, serialization, and handling of etymologies. - Established pytest fixtures for unit testing in `unit_conftest.py`, including mocks for database connections and services to facilitate isolated testing.
…idation - Implement tests for parsing LIFT entries with single and multiple senses. - Add tests for extracting variant types from LIFT traits and language codes. - Create tests for entry creation with various field types to ensure list handling. - Introduce tests for POS inheritance functionality in entry forms. - Add tests for the multilingual form processor to validate JSON structure. - Implement tests for the Scholastic Assessment Test entry validation. - Create comprehensive unit tests for system status functionality. - Add unit tests for base exporter and utility classes, including exception handling.
- Backend:
- Extended API sort_by options for entries to include:
citation_form, part_of_speech, gloss, definition.
- Updated DictionaryService XQuery to support sorting by these new fields,
including case-insensitive sorting and consistent handling of empty values.
- Frontend (Entries Page):
- Implemented dynamic column visibility:
- You can show/hide columns via a dropdown menu.
- Preferences are saved in LocalStorage.
- Ensures at least one column remains visible.
- Table headers and rows are now dynamically rendered by JavaScript.
- Implemented sorting for all configurable columns directly via table headers.
- Sort state (field and order) is saved in LocalStorage.
- Updated styling and icons for new controls and sort indicators.
- Introduced a new DeclarativeValidationEngine to handle validation rules defined in JSON format. - Added comprehensive validation rules in validation_rules_v2.json for entry, sense, and note levels. - Updated tests to accommodate changes in validation structure and ensure proper functionality. - Modified existing test cases to reflect new data structures for definitions and glosses. - Enhanced language selection forms to utilize a comprehensive language database. - Improved user experience by providing dropdowns and searchable fields for language selection.
…d management scripts - Added `init-basex-data.sh` to initialize BaseX with sample LIFT data. - Created `reset_basex_password.py` to generate and update the BaseX admin password hash. - Implemented `setup_basex_password.py` to set the BaseX admin password programmatically. - Introduced `test-summary.sh` for summarizing integration test results. - Developed integration tests for Academic Domains CRUD functionality in `test_academic_domains_crud.py`. - Added Playwright-based UI tests for relations and variants in `test_relations_variants_ui_playwright.py`. - Created unit tests for Academic Domains functionality in `test_academic_domains.py`.
- Introduced entry-level and sense-level academic domain fields in entry_form.html. - Updated validation rules to include checks for academic domain formats. - Enhanced integration tests to cover academic domain functionality, including form submissions and edits. - Implemented JSON Schema validation for validation_rules.json to ensure proper structure and prevent errors. - Added comprehensive tests for academic domains, ensuring correct functionality and data persistence. - Updated requirements.txt to include jsonschema for validation purposes.
- Updated test results in `test_results.txt` and `test_results_updated.txt`. - Notable failures include multiple tests in `test_settings_page_playwright.py`, `test_academic_domains_crud.py`, and `test_morph_type_integration.py`. - Addressed issues related to language selection, academic domain handling, and performance benchmarks. - Improved error handling in various integration tests.
…try model - Implement tests for single and multiple language lexical_unit serialization. - Ensure empty texts are filtered out and at least one language is required. - Validate structure of lexical_unit and raise appropriate errors for invalid formats. - Test integration of serialized lexical_unit with Entry model, ensuring it accepts the correct format and rejects invalid ones.
- Updated BaseX query to use collection-based approach for loading LIFT ranges. - Added mappings for 'academic-domain' to 'domain-type' in API. - Improved API response format to include success status and data. - Enhanced JavaScript to initialize all dynamic-lift-range dropdowns. - Created comprehensive Playwright tests for dropdown population and API accessibility. - Updated test data to reflect new mappings and ensure accurate testing.
- Created comprehensive implementation plan (docs/XML_DIRECT_MANIPULATION_PLAN.md) - Created day-by-day kickoff checklist (IMPLEMENTATION_KICKOFF.md) - Updated TODO.md and specification.md with references - Corrected understanding: No SQLAlchemy models exist - PostgreSQL only used for corpus/worksets, not entries - Entries already in BaseX XML (no migration needed) - Plan focuses on removing WTForms layer for direct XML manipulation
- Created a new development branch for XML direct manipulation. - Developed a client-side library for generating LIFT 0.13 compliant XML from form data. - Added a demo HTML page to showcase the LIFT XML Serializer functionality. - Implemented serialization of various LIFT elements including lexical units, senses, pronunciations, variants, relations, etymologies, and notes. - Included validation functionality for generated XML against basic LIFT schema requirements. - Updated implementation kickoff document with current status and completion of initial setup tasks.
- Created LIFTXMLSerializer class for client-side LIFT 0.13 XML generation - Handles all entry components: lexical units, senses, examples, pronunciations, variants, relations, etymologies, notes, grammatical info, domains - Uses domain-type trait (not academic-domain) throughout - Added comprehensive Jest test suite (38 tests, all passing) - Coverage: 92.43% statements, 79.36% branches, 92.68% functions - Configured Jest with @xmldom/xmldom for XML DOM support in Node.js - Updated package.json with Jest scripts - Added jest.config.js with 90% coverage thresholds - Day 1-2 deliverable complete
✨ Added XQuery Modules (1,110 lines) - entry_operations.xq (370 lines, 9 CRUD functions) - sense_operations.xq (360 lines, 7 sense operations) - validation_queries.xq (380 lines, 10 validation functions) ✅ All modules include: - LIFT 0.13 namespace support - Comprehensive try/catch error handling - Structured XML result format - Automatic order management (senses) - Database integrity checks 🧪 Testing & Verification - Created test_basex_simple.py to verify BaseX connectivity - Confirmed database access (dictionary, 397 entries) - Validated XQuery execution (<50ms for simple queries) - Documented module loading strategy for Python layer 📚 Documentation - XQUERY_TEST_RESULTS.md - Complete test report - DAY_3_4_COMPLETION_SUMMARY.md - Detailed deliverables - Updated IMPLEMENTATION_KICKOFF.md - Marked Day 3-4 complete 🎯 Status: Day 3-4 ✅ COMPLETE - Ready for Day 5-7 (Python XML Service Layer) Technical Details: - BaseX 12.0 compatible XQuery 3.1 - Uses session.query().execute() for XQuery - Module namespace pattern: http://dictionaryapp.local/xquery/* - All functions return structured XML with status/message
- Added XML Preview Panel to entry form for debugging and visualization of generated LIFT XML. - Integrated LIFT XML Serializer in the entry form JavaScript for XML generation and submission. - Created new API endpoints for XML-based CRUD operations on dictionary entries. - Enhanced integration tests to cover XML submission, retrieval, updating, and deletion. - Registered XML entries blueprint in the application and test fixtures. - Improved error handling and logging for XML processing and database interactions. - Updated documentation to reflect new XML features and API endpoints.
…dation engine integration
- Implement tests for direct entry lookup and existence checks in `test_entry_direct.py` and `test_entry_exists.py`. - Create a script to list registered routes in `test_routes.py`. - Develop a validation button in `test_validate_button.html` with error handling modal. - Add integration tests for form field persistence in `test_form_field_persistence.py`, ensuring all fields are saved and retrieved correctly. - Introduce end-to-end tests for grammatical_info field persistence in `test_grammatical_info_e2e.py`. - Implement simple tests for grammatical_info persistence in `test_pos_simple.py`. - Create integration tests for XML entry CRUD operations in `test_xml_entry_crud_integration.py`, verifying data integrity and error handling.
- Implement integration tests for CRUD operations on subsenses, validating creation, reading, updating, and deletion through the API. - Create unit tests for subsense data structures, recursive structures, ordering, and data integrity. - Ensure comprehensive coverage of subsense attributes including glosses, definitions, examples, notes, and relations. - Validate unique IDs for subsenses and test deeply nested structures.
…attributes, reversals, and sense relations - Implement tests for the illustrations attribute in the Sense model, covering single and multiple illustrations, multilingual labels, and integration with other sense data. - Add tests for pronunciation custom fields, including cv-pattern and tone attributes, validation, and serialization to LIFT XML. - Create tests for media attributes in both the Pronunciation and Entry models, ensuring support for various media formats and labels. - Develop comprehensive tests for reversal functionality, including structure, grammatical-info, and XML serialization based on LIFT 0.13 specifications. - Introduce tests for sense-level relations, validating parsing and generation of synonyms, antonyms, and other semantic relationships in LIFT XML.
- Implement integration tests for Entry order and date attributes XML parsing/generation. - Add integration tests for Etymology XML parsing/generation, including comment and custom fields. - Create integration tests for Example enhancements, covering source attribute, note field, and custom fields. - Develop unit tests for Entry order and optional date attributes, ensuring correct handling of order and dateDeleted attributes. - Add unit tests for Etymology enhancements, verifying gloss, comment, and custom fields support. - Implement unit tests for Example enhancements, validating source, note, and custom fields functionality.
…field functionality - Implemented a comprehensive E2E test suite for form submissions covering multilingual data, grammatical information, notes, and special characters. - Added tests to ensure homograph number fields are not visible in the add entry form and that tooltip icons are consistent. - Created tests to verify the behavior of the POS field in both add and edit forms, ensuring proper data handling and submission functionality. - Included logging and timeout handling in the E2E test runner script to improve test execution and reporting.
- Removed outdated FIXES_COMPLETED_NOV29.md and FIXES_COMPLETED_NOV30.md files. - Enhanced application initialization by registering the validation service API. - Improved workset API validation to return detailed error messages without raising HTTP errors. - Cleaned up API routes by removing deprecated query validation endpoint. - Updated integration test setup to ensure PostgreSQL pool is initialized correctly for tests. - Refactored API integration tests to use XML for entry creation, ensuring unique IDs for each test. - Enhanced error handling in entry service to use ValidationError for invalid entries. - Updated test cases to reflect changes in entry structure and ensure compatibility with new data formats. - Improved form field persistence tests to handle LIFT XML namespaces correctly. - Ensured all tests pass with a current pass rate of 94.4%, while addressing remaining Playwright test issues.
- Updated tests in `test_lift_ranges_integration.py` to create entries via XML API instead of using the database directly. - Modified `test_pos_inheritance_scenarios.py` to create entries with multiple senses through XML API. - Changed `test_pos_simple.py` to ensure proper parsing of grammatical info from XML responses. - Enhanced `test_pronunciation_and_dynamic_ranges.py` to validate language codes and variant types through API endpoints. - Improved `test_ui_enhancements.py` to check for multilingual field management in JavaScript files. - Refined `test_web_form_protestantism.py` to simulate form submissions with dynamically generated entry IDs. - Updated `test_xml_form_submission.py` to generate unique IDs for each test case, ensuring isolation and reliability.
…istry, and LIFT to HTML Transformer - Implement comprehensive unit tests for DisplayProfileService covering CRUD operations, profile management, and validation. - Create tests for the Help Page to ensure accessibility, content correctness, and navigation. - Add unit tests for LIFT Element Registry to validate element retrieval, filtering, and configuration. - Develop tests for LIFT to HTML Transformer to verify correct transformation of LIFT XML to HTML with various configurations and visibility rules.
…ration tests for DisplayProfile and CSSMappingService - Created migration script to add `number_senses_if_multiple` boolean field to `DisplayProfile` model. - Implemented integration tests for `CSSMappingService`, including CRUD operations and entry rendering. - Developed integration tests for `DisplayProfile` and `ProfileElement` models, covering creation, relationships, and serialization. - Added integration tests for `DisplayProfileService`, including profile creation, retrieval, updates, deletions, and import/export functionality.
- Implemented the RangesService class for managing LIFT ranges with CRUD operations. - Created API endpoints for range management, including listing, creating, updating, and deleting ranges. - Developed frontend UI for the ranges editor with HTML templates and JavaScript functionality. - Added integration tests for API endpoints and end-to-end tests for UI interactions. - Documented the implementation process, core features, technical architecture, and usage scenarios in RANGES_EDITOR_QUICKSTART.md and RANGES_EDITOR_SUMMARY.md. - Introduced test fixtures for integration tests to ensure a clean state before each test.
- Implement integration tests for ranges editor API covering listing, retrieval, creation, updating, and deletion of ranges. - Add integration tests for range elements CRUD operations including creation, retrieval, updating, and deletion. - Create unit tests for RangesService to validate range and element operations, including uniqueness checks, creation, updating, and deletion logic. - Ensure comprehensive coverage of edge cases such as missing fields, duplicate IDs, and validation errors.
- Implement integration tests for merge/split API endpoints, covering operations like split and merge for entries and senses. - Add tests for handling non-existent entries in XML API, ensuring correct creation and updating behavior. - Introduce unit tests for utility functions related to sense relations and normalization of indexed arrays. - Enhance unit tests for editing non-existent entries, verifying that relations to non-existent targets are handled correctly. - Create unit tests for merge/split operation models and services, validating initialization, status updates, and operation history retrieval. - Implement cache invalidation tests for XML entry updates, ensuring cache is cleared appropriately after updates.
This commit extends the lift ranges editor to support multilingual abbreviations, descriptions, and labels. The backend has been updated to correctly process multilingual data for all three fields, and the frontend has been updated to include multilingual input fields for abbreviations and labels in the element modal. Unit and integration tests have been updated to cover the new functionality.
Contributor
Author
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. New to Jules? Learn more at jules.google/docs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change extends the lift ranges editor to support multilingual abbreviations, descriptions, and labels. The backend has been updated to correctly process multilingual data for all three fields, and the frontend has been updated to include multilingual input fields for abbreviations and labels in the element modal. Unit and integration tests have been updated to cover the new functionality.
PR created automatically by Jules for task 8569543665367553327 started by @milekpl