Repository navigation
Sync main into develop (merge commit — do not squash) - #263
Merged
Merged
Conversation
installs vite for frontend and liblaries like prettier, tailwind, add…
* Main (#194) Co-authored-by: Rafal Korszun <r.korszun@kleder.co> Co-authored-by: github-actions <github-actions@users.noreply.github.com> * chore(deps): bump axios from 1.13.2 to 1.13.5 (#196) Bumps [axios](https://github.com/axios/axios) from 1.13.2 to 1.13.5. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.13.2...v1.13.5) --- updated-dependencies: - dependency-name: axios dependency-version: 1.13.5 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump @langchain/community from 1.1.2 to 1.1.14 (#195) Bumps [@langchain/community](https://github.com/langchain-ai/langchainjs) from 1.1.2 to 1.1.14. - [Release notes](https://github.com/langchain-ai/langchainjs/releases) - [Commits](https://github.com/langchain-ai/langchainjs/compare/@langchain/community@1.1.2...@langchain/community@1.1.14) --- updated-dependencies: - dependency-name: "@langchain/community" dependency-version: 1.1.14 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Feat/plugin loader (#197) * refactor(auth, plugins, schemas): improve code formatting and structure * feat(admin): add Admin Dashboard page and update routing feat(plugin): implement schema registry for core and plugins refactor(plugin-loader): enhance plugin loading with context and order refactor(analytics): update schema registration for analytics models refactor(auth, forms, settings): use core schema registry for Mongoose models docs(README): add schema registry guidelines for plugin development * refactor(docs): standardize architecture document references and remove outdated files - Updated README and plugin instructions to reference `docs/architecture.md` consistently. - Deleted obsolete documents: `TASK_DEPENDENCY_ANALYSIS.md`, `TASK_VALIDATION_REPORT.md`, `architecture_overview.md`, `SAAS_IMPLEMENTATION_PLAN.md`, `SaaS_TODO.md`. - Adjusted links in `docs/ARCHITECTURE.md` for clarity and consistency. * chore(deps): update @opendexcom/plugin-interface to version 1.0.6 and adjust peer dependencies - Bumped @opendexcom/plugin-interface from 1.0.5 to 1.0.6 in package.json and pnpm-lock.yaml. - Updated peer dependencies for compatibility with @nestjs/common and added support for @nestjs/mongoose. - Added deprecation notice for old versions of glob in pnpm-lock.yaml. * refactor(plugin-loader): replace console.log with logger for plugin loading messages - Updated the PluginLoaderService to use a logger instead of console.log for logging plugin loading attempts from npm and local paths, enhancing logging consistency and maintainability. * refactor(ai): enhance AI controller with improved Swagger documentation and code formatting - Added ApiResponse decorator to the generate endpoint for clearer API documentation. - Reformatted code for better readability and consistency, including spacing adjustments. - Improved error handling in the generateStream method to ensure proper response closure. * feat(ai): implement usage tracking in AI service and tests - Added LlmUsage interface to capture AI model usage metrics. - Enhanced AiService methods to return usage metadata alongside generated content. - Updated tests in ai.service.spec.ts to validate usage tracking functionality. - Refactored invokeModel and invokeModelRaw methods to support usage extraction from responses. - Modified analytics components to utilize the new invokeModelRawWithUsage method for improved tracking. * chore: update dependencies * chore: update development scripts for server and client - Modified server and client package.json scripts for consistency in development commands. - Updated server scripts to use `nest start --watch` for development. - Added new start commands for client to facilitate development and preview. - Refactored worker-related scripts in server for clarity and ease of use. * chore: update docker-compose and package scripts for consistency - Changed the command for the worker service in docker-compose.yml to use "pnpm run worker:prod" for clarity. - Updated package.json to use "docker compose" instead of "docker-compose" for improved compatibility. - Enhanced README files for both server and client to reflect updated commands and usage instructions. - Refactored client and server scripts for better organization and clarity. * refactor(ai): update analyzeTextWithUsage method to support JSON format option - Modified analyzeTextWithUsage to include a useJsonFormat parameter for flexibility in output format. - Updated invocations in SummaryGenerator and TopicClusterer to utilize the new method signature. - Adjusted token extraction logic in extractUsageFromResponse for improved clarity and handling of undefined values. * feat(ai): enhance usage tracking and response handling in AI service - Introduced a UsageTrackingRequest type to facilitate optional usage tracking in requests. - Updated extractUsageFromResponse to handle additional metadata formats for improved usage extraction. - Modified ai.service.ts to include raw response handling alongside parsed output. - Enhanced ai.controller.ts to ensure consistent usage tracking during streaming responses. - Added tests in ai.service.spec.ts to validate new usage extraction logic and ensure correct handling of structured output. * fix(AdminDashboard): improve interval management for component mounting - Refactored the interval handling logic to ensure proper cleanup and prevent potential memory leaks. - Added a check to only set the interval if the initial mount attempt fails, enhancing performance and reliability. * feat(ai): update AI service response structure to include usage metadata - Introduced GenerateAIFormResponse interface to encapsulate form and optional usage data. - Modified generateForm and refineForm methods to return structured response with form and usage. - Updated ai.controller.ts documentation to reflect new response format. - Adjusted tests in ai.service.spec.ts to validate the new response structure. * refactor(ai): enhance generateWithSteps method to include detailed usage tracking - Updated the generateWithSteps method to yield usage metadata for each processing step: analyze, questions, and optimize. - Modified the response structure to include usage information in the generated steps. - Adjusted tests in ai.service.spec.ts to validate the new usage tracking implementation and ensure accurate assertions for each step's usage data. * feat(AdminDashboard): enhance admin dashboard loading with improved polling and error handling - Introduced constants for polling intervals and maximum wait time to manage admin dashboard loading. - Added state management for load failure to provide user feedback if the admin dashboard fails to load. - Refactored polling logic to ensure proper cleanup and prevent memory leaks during component unmounting. - Updated the command for the worker service in docker-compose.yml to use "pnpm start:worker:prod" for consistency. - Added new start commands in package.json for better organization of worker scripts. * refactor(AdminDashboard): simplify admin role check in loading logic - Replaced direct role check with a boolean constant for improved readability and maintainability. - Updated dependencies in the useEffect hook to utilize the new isAdmin constant for clarity. * fix(SummaryGenerator): update analyzeTextWithUsage call to validate prompt - Changed the analyzeTextWithUsage method call to set the second parameter to false, ensuring prompt validation while maintaining plain text output format. - Adjusted comments for clarity regarding the purpose of the parameter change. * test(SummaryGenerator): add unit tests for generateAnalyticsSummary method - Introduced a new test suite for the SummaryGenerator class to validate the generateAnalyticsSummary method. - Ensured that the analyzeTextWithUsage method is called with the correct parameters for prompt validation and output format. - Mocked dependencies to isolate the unit tests and verify functionality without external side effects. * Clarify token fallback precedence in AI usage Co-authored-by: Rafal Korszun <rkorszun@users.noreply.github.com> * docs: correct Node.js version requirement to 20+ (#198) * Initial plan * docs: update Node.js version requirement from 18+ to 20+ Co-authored-by: rkorszun <8030590+rkorszun@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rkorszun <8030590+rkorszun@users.noreply.github.com> --------- Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rafal Korszun <rkorszun@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> * chore: Enhance AI service and testing coverage (#199) - Updated `package.json` to exclude specific files from coverage collection, improving test coverage accuracy. - Added new tests in `ai.service.spec.ts` to handle exceptions and validate prompt processing in the `generate` and `analyzeTextWithUsage` methods. - Introduced `guardian.service.spec.ts` to test the prompt validation logic, ensuring security checks are correctly implemented. - Created `correlation.calculator.spec.ts`, `sentiment.calculator.spec.ts`, `statistics.calculator.spec.ts`, `trend.calculator.spec.ts`, `findings.generator.spec.ts`, `recommendations.generator.spec.ts`, and `auth.service.spec.ts` to cover various analytics and authentication functionalities, enhancing overall test coverage and reliability. Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> * feat: Add PasswordStrengthIndicator component and integrate password … (#200) * feat: Add PasswordStrengthIndicator component and integrate password validation in Landing and Reset Password pages - Introduced `PasswordStrengthIndicator` component to provide real-time feedback on password strength. - Integrated password validation logic in the `LandingPage` and `ResetPassword` components to enforce stronger password requirements. - Updated password input fields to require a minimum length of 8 characters and display the strength indicator during user input. * refactor: Simplify PasswordStrengthIndicator imports in Landing and Reset Password pages - Updated imports of `PasswordStrengthIndicator` in `LandingPage` and `ResetPassword` components to streamline the codebase. - Clarified documentation in `passwordValidation.ts` to better explain password strength calculation rules. --------- Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> * Feat/passwords (#201) * feat: Add PasswordStrengthIndicator component and integrate password validation in Landing and Reset Password pages - Introduced `PasswordStrengthIndicator` component to provide real-time feedback on password strength. - Integrated password validation logic in the `LandingPage` and `ResetPassword` components to enforce stronger password requirements. - Updated password input fields to require a minimum length of 8 characters and display the strength indicator during user input. * refactor: Simplify PasswordStrengthIndicator imports in Landing and Reset Password pages - Updated imports of `PasswordStrengthIndicator` in `LandingPage` and `ResetPassword` components to streamline the codebase. - Clarified documentation in `passwordValidation.ts` to better explain password strength calculation rules. * feat: Add User Billing page and conditional rendering in Header - Introduced `UserBillingPage` component and added a new route for billing settings. - Updated `Header` component to conditionally display a Billing button based on the availability of the billing module. - Implemented polling logic to check for the billing module's presence, enhancing user experience by only showing relevant options. --------- Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> * feat: Update error message in AdminDashboardPage for clarity (#202) Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> * feat(plugin-loader): enhance plugin loading mechanism with path resolution (#203) - Updated the PluginLoaderService to utilize `path.resolve` and `pathToFileURL` for constructing plugin paths, improving compatibility and reliability when loading plugins from local directories. - Enhanced logging to provide clearer messages regarding the loading attempts of plugins, ensuring better traceability during the plugin loading process. Co-authored-by: Rafal Korszun <r.korszun@kleder.co> * button fix (#208) * fix (#209) * Fix BullMQ job stalling caused by insufficient lockDuration for LLM workloads (#210) * Initial plan * fix: increase BullMQ lockDuration to prevent job stalling on LLM workloads - Add settings (lockDuration: 5min, stalledInterval: 5min, maxStalledCount: 5) to global BullModule config in bull.module.ts - Add same settings to each analytics queue registration in analytics-queue.module.ts - Include worker in root dev script so it starts alongside server and client - Fix worker:prod entry path (dist/worker -> dist/src/worker) Co-authored-by: rkorszun <8030590+rkorszun@users.noreply.github.com> * chore: simplify development scripts in package.json - Removed the worker script from the main dev command in package.json for clarity and streamlined development process. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rkorszun <8030590+rkorszun@users.noreply.github.com> Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> * Refactor form components to support "other" option functionality - Updated FormCanvas and FormFieldsPanel to rename "Multiple Choice" to "Single Choice" for clarity. - Enhanced FormPreview to handle "other" option inputs for multiple choice, checkbox, and dropdown questions, allowing users to specify custom answers. - Implemented migration logic in QuestionEditor to ensure questions are correctly set up for "other" options, including handling placeholder text. - Added utility functions for managing "other" options, including migration, labeling, and validation. - Updated FormEditor and PublicFormView to ensure forms are loaded with the correct "other" option configurations. This update improves user experience by allowing more flexible responses in forms. * Enhance "other" option handling in AI service - Refactored logic to ensure the "other" option is always positioned at the end of the options list. - Improved handling of the "other" option prefix to prevent duplicates and correctly extract placeholders when necessary. - Updated processed options logic to maintain clarity and consistency in option management. This update improves the flexibility and reliability of form responses involving "other" options. * Chore/dep (#212) * Update dependencies in pnpm-lock and package.json files - Updated versions of several packages in pnpm-lock.yaml, including jspdf (4.2.0), @langchain/community (1.1.20), and @langchain/openai (1.2.11). - Adjusted package.json files for both client and server to reflect the updated jspdf version. - Cleaned up pnpm-workspace.yaml by removing quotes around package names for consistency. These changes ensure that the project uses the latest compatible versions of dependencies, improving stability and functionality. * Update @langchain/core to version 1.1.29 in package.json and pnpm-lock.yaml - Updated the version of @langchain/core to 1.1.29 in both package.json and pnpm-lock.yaml to ensure compatibility with other dependencies. - Adjusted related dependencies in pnpm-lock.yaml to reflect the new versioning, enhancing overall stability and functionality of the project. --------- Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> * Chore/dep (#214) * Update dependencies in pnpm-lock and package.json files - Updated versions of several packages in pnpm-lock.yaml, including jspdf (4.2.0), @langchain/community (1.1.20), and @langchain/openai (1.2.11). - Adjusted package.json files for both client and server to reflect the updated jspdf version. - Cleaned up pnpm-workspace.yaml by removing quotes around package names for consistency. These changes ensure that the project uses the latest compatible versions of dependencies, improving stability and functionality. * Update @langchain/core to version 1.1.29 in package.json and pnpm-lock.yaml - Updated the version of @langchain/core to 1.1.29 in both package.json and pnpm-lock.yaml to ensure compatibility with other dependencies. - Adjusted related dependencies in pnpm-lock.yaml to reflect the new versioning, enhancing overall stability and functionality of the project. * Enhance form generation capabilities with document upload support - Added support for uploading PDF documents to generate forms using AI, allowing users to create forms based on existing documents. - Implemented a new API endpoint for processing document uploads and generating forms, including validation for file types and sizes. - Updated the AI service to handle document input, ensuring seamless integration with existing form generation logic. - Enhanced the frontend to allow users to attach PDF files and receive generated forms based on the document content. This update significantly improves the user experience by enabling form creation from existing documents, streamlining the form-building process. --------- Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> * chore(deps): bump jspdf from 4.2.0 to 4.2.1 (#213) Bumps [jspdf](https://github.com/parallax/jsPDF) from 4.2.0 to 4.2.1. - [Release notes](https://github.com/parallax/jsPDF/releases) - [Changelog](https://github.com/parallax/jsPDF/blob/master/RELEASE.md) - [Commits](parallax/jsPDF@v4.2.0...v4.2.1) --- updated-dependencies: - dependency-name: jspdf dependency-version: 4.2.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Implement timeout handling for AI model invocations (#216) * Implement timeout handling for AI model invocations - Added a timeout feature to the `invokeModelRawWithUsage` method, allowing invocations to abort if they exceed a specified duration (default: 120 seconds). - Refactored invocation logic to utilize an `AbortController` for managing timeouts effectively. - Updated related methods to support the new timeout parameter, ensuring consistent behavior across AI model interactions. This enhancement improves the robustness of AI service calls by preventing indefinite waiting periods during model invocations. * Reduce stalledInterval for analytics queues from 300000ms to 15000ms to improve responsiveness. Enhance error handling in ProgressService's publishProgress method to log non-fatal Redis pub/sub errors, ensuring better monitoring of progress updates. --------- Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> * chore(deps-dev): bump vite from 6.4.1 to 6.4.2 (#218) Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.4.1 to 6.4.2. - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/v6.4.2/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v6.4.2/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 6.4.2 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump @nestjs/core from 11.1.11 to 11.1.18 (#217) Bumps [@nestjs/core](https://github.com/nestjs/nest/tree/HEAD/packages/core) from 11.1.11 to 11.1.18. - [Release notes](https://github.com/nestjs/nest/releases) - [Commits](https://github.com/nestjs/nest/commits/v11.1.18/packages/core) --- updated-dependencies: - dependency-name: "@nestjs/core" dependency-version: 11.1.18 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rafal Korszun <8030590+rkorszun@users.noreply.github.com> * Enhance PDF handling and switch to pdf2json for parsing (#219) * Enhance PDF document handling in AI service - Integrated `pdfjs-dist` for processing PDF uploads, enabling extraction of text and metadata from documents. - Updated `AiController` to handle PDF files, extracting text content and document metadata for form generation. - Added support for new optional dependencies related to PDF processing in `package.json` and `pnpm-lock.yaml`. - Improved error handling and validation for uploaded files, ensuring only PDF documents are processed. This update significantly enhances the AI service's capabilities by allowing users to generate forms based on existing PDF documents. * Replace pdfjs-dist with pdf2json for PDF parsing in AI service * Add security validation check for prompt in generateWithSteps method * Enhance form generation rules and PDF handling in AI service * feat: implement AI controller and service for form generation with PDF document parsing and streaming support * chore(deps): bump nodemailer from 7.0.12 to 8.0.5 (#220) Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 7.0.12 to 8.0.5. - [Release notes](https://github.com/nodemailer/nodemailer/releases) - [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md) - [Commits](nodemailer/nodemailer@v7.0.12...v8.0.5) --- updated-dependencies: - dependency-name: nodemailer dependency-version: 8.0.5 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add documentation for PDF form generation issues and limitations (#222) * Add documentation for PDF form generation issues and limitations - Created a new README file outlining the PDF to form generation workflow and known limitations. - Added individual documentation files for specific issues related to PDF form generation, including: - Issue 1: Repeated fields truncation - Issue 2: Expensive verification and build process - Issue 3: Retry mechanism losing PDF context - Issue 4: Challenges with large forms (~130 questions) - Issue 5: Need for smaller, focused prompts These documents provide insights into the current behavior and pain points in the form generation process, aiding in future improvements. * feat: integrate OpenDataLoader for PDF parsing and enhance AI form generation - Added @opendataloader/pdf dependency for improved PDF parsing capabilities. - Replaced pdf2json with OpenDataLoader in ai.controller.ts for parsing PDF files. - Updated parsePdfWithOpenDataLoader function to handle PDF extraction and cleanup. - Enhanced buildPageChunks function to optimize chunking of extracted text. - Implemented concurrency in AI processing to improve performance during document analysis. - Added coverage check for missing fields in generated forms based on PDF content. - Updated AI service methods to support maxTokens parameter for better control over response length. - Improved validation and error handling in AI form assembly process. Co-authored-by: Copilot <copilot@github.com> --------- Co-authored-by: Rafał Korszuń <r.korszun@kleder.co> Co-authored-by: Veresek <filipowskimedia@gmail.com> Co-authored-by: Copilot <copilot@github.com> * chore(deps): bump axios from 1.13.5 to 1.15.2 (#224) Bumps [axios](https://github.com/axios/axios) from 1.13.5 to 1.15.2. - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.13.5...v1.15.2) --- updated-dependencies: - dependency-name: axios dependency-version: 1.15.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(server): enhance bootstrap process with error handling and Stripe webhook support - Updated the Nest application creation to include `rawBody` for Stripe webhook signature verification. - Added error handling to the bootstrap function to log failures and exit the process if initialization fails. chore(ci): add security audit step to CI workflow - Included a `pnpm audit --audit-level=high` step in the CI workflow to ensure high-severity vulnerabilities are identified during the build process. * chore(deps): update dependencies and add overrides for security patches - Updated various dependencies in `client/package.json`, `server/package.json`, and `pnpm-lock.yaml` to their latest versions, including axios, react, and @nestjs packages. - Added overrides for critical packages in `pnpm-lock.yaml` and `pnpm-workspace.yaml` to ensure compatibility and security. - Adjusted versions for @langchain packages to improve functionality and maintain compatibility across the project. * refactor(docker): streamline Docker configuration and enhance plugin support - Updated `.dockerignore` to include build artifacts and improve dependency management. - Refactored `docker-compose.yml` to remove unnecessary services and clarify instructions for OSS-only app services. - Adjusted `package.json` scripts for better Docker integration with external compose files. - Enhanced client-side code by integrating plugin architecture, allowing for dynamic route and slot management. - Improved AI form chat component to utilize plugin slots for document attachment handling. - Updated authentication context and registration process to support optional terms acceptance via plugins. This commit enhances the overall structure and functionality of the application, paving the way for better extensibility and maintainability. * chore(deps): bump concurrently to fix lodash audit chain Co-authored-by: Cursor <cursoragent@cursor.com> * chore(deps): require @opendexcom/plugin-interface ^1.0.7 Aligns with hostModules and registration extension APIs. Run npm publish for plugin-interface 1.0.7 before refreshing pnpm-lock.yaml. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: update dependency * feat: open-core host seams, absorb plugin-interface, OSS/EE boundary cleanup - Move plugin-interface into packages/ as link-only workspace package - Add capability registry, GET /api/capabilities, schemaRegistry.extend() - Contribution-based UI manifest (routes, nav, slots) with event-based readiness - Remove EE leakage: hardcoded admin/billing routes, legacy module globals - Commit genuine OSS work: AI/MLflow, embeddings, semantic cache, auth session Co-authored-by: Cursor <cursoragent@cursor.com> * fix(server): keep dist output during nest watch to avoid missing modules Disable deleteOutDir in nest-cli.json so incremental rebuilds do not delete unchanged outputs like app.module.js while main.js is restarted. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(mlflow): add LangGraph tracing, embeddings, tokens, orchestration, and DLQ spans Introduce MLflow trace context helpers (LLM, EMBEDDING, graph node, orchestration stage, DLQ failure traces), LangChain bootstrap, LangGraph parent/node spans, token usage on spans, embedding spans for semantic cache, orchestration parent span, and worker trace contexts on analytics queue consumers. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(analytics): add vector topic clustering and pipeline improvements Topic vector store, clusterer refactor, question/sentiment filters, combined batch prompts integration, aggregation utilities, and queue orchestration updates for scalable analytics processing. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(client): improve analytics topic filtering and report cards Add topic filter utilities and update analytics cards and FormAnalytics for filtered topic views and richer report presentation. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(plugins): extend schema registry and host module wiring Extend plugin-interface schema registry, update core schemas and app/worker module wiring for plugin host integration. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(client): enhance AIFormChat with document attachment slot API integration Integrate a new document attachment slot API into the AIFormChat component, allowing for dynamic handling of document uploads. Implement event listeners for slot API updates and improve file management by utilizing the updated plugin slot system. This enhances the user experience by ensuring real-time updates and better attachment handling. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(client): add settings layout, user preferences, and plugin nav hook * feat(client): surface quota limit errors and upgrade banner in forms and AI * feat(server): track analytics LLM usage and add users profile module * fix(server): stabilize analytics worker boot and RediSearch detection Use AiCoreModule in the worker dependency graph, break FormsModule circular imports with forwardRef, and parse nested Redis 7 MODULE LIST responses so topic vector clustering works with Redis Stack. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(server): add projects domain with variants and study analysis Introduce the projects module, schema registry entries, form-to-project migration tooling, and form service updates for multi-variant studies. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(server): add cross-variant comparative analytics pipeline Add cross-variant orchestration, topic alignment, metrics, and MLflow prompts with supporting prompt builder and topic normalization utilities. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(server): add integrations schema and inbound notification hooks Register integration connection and notification schemas, wire plugin loader updates, and expose inbound webhook handling for external integrations. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(server): improve analytics queue processing and aggregation Extend response processing, topic clustering, aggregation consumers, and summary generation for project-aware analytics workloads. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(client): replace dashboard with projects-first app shell Add projects dashboard, overview routing, settings/help integrations entry points, capability hooks, and updated plugin navigation for the new shell. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(client): add project detail, variants, and comparative analytics pages Introduce project tabs for variants, responses, analytics, and cross-variant compare flows with supporting form editor and analytics UI updates. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: upgrade to Node 24 and remove VPS deploy workflow Deployment is handled by the EE repo; drop the legacy OSS deploy workflow and align Docker/CI on Node 24. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: bump GitHub Actions to checkout@v7 and setup-node@v6 Enable pnpm dependency caching in CI for faster installs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: support optional SMTP auth and TLS env configuration Allow auth-free internal relays and self-signed certificates via SMTP_SECURE and SMTP_TLS_REJECT_UNAUTHORIZED. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(server): delete a study instead of only archiving it Archive left the project in place, so users could not free the study or remove it from the list. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(client): redesign overview, editor, and study variants Match the UX audit frames so study chrome, the form canvas, and variant cards share one layout. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auth): keep login errors visible and match emails case-insensitively Failed sign-in remounted the landing form because AuthContext toggled global loading, wiping the error. Do not treat credential 401s as a session reset, and look up emails without case sensitivity. Co-authored-by: Rafal Korszun <rkorszun@users.noreply.github.com> * ci: install pnpm before setup-node cache setup-node@v6 with cache: pnpm looks up the pnpm binary during setup, before corepack had installed it, so CI failed before tests. Co-authored-by: Rafal Korszun <rkorszun@users.noreply.github.com> * chore: patch high-severity dependencies for CI audit Bump axios, nodemailer, postcss, vite, vitest, react-router, and js-yaml, and add pnpm overrides so `pnpm audit --audit-level=high` passes. Also align outdated service specs and getErrorMessage so build and tests stay green. Co-authored-by: Rafal Korszun <rkorszun@users.noreply.github.com> * chore: patch remaining high/critical npm advisories (#245) Bump nodemailer to 10.x (Node 20+; project is on Node 24), mongoose 8.24.5, DOMPurify 3.4.16, libreoffice-convert 1.8.2, and postcss 8.5.23. Add pnpm overrides for handlebars, proxy-addr, @grpc/grpc-js, brace-expansion, sharp, source-map-js, uuid 11, and other transitive patches so `pnpm audit --audit-level=high` passes. Ignore unpatched braces GHSA (Tailwind/chokidar build-time only; no 3.0.4+ release). Co-authored-by: Cursor Agent <cursoragent@cursor.com> * Cursor/multilingual e5 embedding 34bc (#244) * feat(ai): default local embeddings to multilingual-e5-small Switch the in-process default from English MiniLM-L6-v2 to Xenova/multilingual-e5-small. Both models are 384-d, so Redis vector indexes keep DIM 384. E5 query/passage prefixes are applied for local E5 models. Topic clustering still uses embedQuery (query prefix) for symmetric similarity. Add an opt-in Polish survey clustering comparison (RUN_LOCAL_EMBEDDING_CLUSTER_TEST=1) and document re-embedding existing Redis topic/cache vectors after the model change. * fix(ai): run Polish embedding comparison outside Jest @huggingface/transformers is ESM and cannot be dynamic-imported from Jest's VM. Keep the opt-in Jest wrapper and run the real comparison via ts-node (pnpm test:polish-cluster). * test(ai): score Polish clustering with k-means and E5 cosine band E5 embeddings sit in a tighter, higher cosine band than MiniLM, so the MiniLM-tuned 0.82 graph threshold over-merges. Assert embedding quality with spherical k-means (k=topics) and same-vs-diff gap, and document raising ANALYTICS_TOPIC_SIMILARITY_THRESHOLD toward 0.88-0.90. * fix(analytics): per-model topic similarity threshold for E5 0.82 collapses multilingual-e5-small into one cluster. Use 0.90 for E5-family models (tuned on the Polish survey fixture) and keep 0.82 for MiniLM. ANALYTICS_TOPIC_SIMILARITY_THRESHOLD remains an explicit override when set. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> * fix(integrations): parse inbound commands in linear time (#247) Replace the polynomial CREATE_STUDY_PATTERN regex with token-boundary scanning so inbound notification text cannot trigger ReDoS. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rafal Korszun <rkorszun@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions <github-actions@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Rafal Korszun <rkorszun@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Dreamiee <118623215+Dreamyy313@users.noreply.github.com> Co-authored-by: relosigma <167084104+relosigma@users.noreply.github.com> Co-authored-by: Kacper Filipowski <145197055+Veresek@users.noreply.github.com> Co-authored-by: Veresek <filipowskimedia@gmail.com> Co-authored-by: Copilot <copilot@github.com>
Record main's history on develop so #253 (develop → main) can merge. #262 was squash-merged, which dropped main as a parent and left the merge-base at f558f27. This is a regular merge commit (two parents). Conflicts resolved in favor of develop. git log develop..main is only prior develop-into-main merges/squashes (latest #246); main has no hotfix that develop lacks. Co-authored-by: Rafal Korszun <rkorszun@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MUST be merged with Create a merge commit. Do not squash. Do not rebase. Squash-merging this PR will leave #253 conflicted.
Why #253 is still conflicted
#261 and #262 both landed on
develop, but #253 (develop→main, head1a9c10e) is stillCONFLICTING.Order
global-agent/ sprintf-js). Squash is fine here — it is a normal feature commit.origin/mainintodevelopso #253’s merge-base would move tomain.Order was correct (#261 first, then the sync). Order is not the bug.
Exact cause: #262 was squash-merged
The #262 branch head was
642dcf3with two parents:e1ec761(developafter fix(security): drop sprintf-js via global-agent 4; accept unpatched braces #261)95cb9b6(origin/maintip, “Merge develop into main (Merge develop into main #246)”)What GitHub wrote onto
developis1a9c10ewith one parent (e1ec761only). Trees are identical (941f555), butmainis not an ancestor ofdevelop.Because squash drops the second parent,
git merge-base origin/develop origin/mainis stillf558f27(chore(deps): bump jspdf … #189), not95cb9b6. Git therefore replays every stalemainsnapshot against currentdevelop, which is why #253 stays dirty.git log develop..mainis only prior develop-into-main merges/squashes (latest #246).mainhas no hotfix thatdeveloplacks.What this PR does
sync/main-into-develop-2from currentdevelop(1a9c10e)git merge origin/mainas a regular merge commit (parents:1a9c10e+95cb9b6)develop)develop(git diff developis empty)git merge-tree origin/main+ this branch is clean (exit 0). After this is merge-committed ontodevelop, Merge develop into main #253 can merge.Never force-push
developormain.Conflicts (all: keep develop)
README.mdexample.comFROM_EMAIL (#251, #257)client/src/pages/FormEditor.tsxclient/src/pages/PublicFormView.tsxdocker-compose.ymlpnpm-lock.yamlpnpm-workspace.yamlglobal-agentoverride / sprintf-js (#261)server/src/analytics/processors/response.processor.tsserver/src/auth/admin-bootstrap.service.tsserver/src/auth/admin-bootstrap.service.spec.tsserver/src/auth/auth.module.tsserver/src/auth/jwt.strategy.tsgetJwtSecret()— notyour-secret-keyfallback (#257)server/src/forms/dto/form.dto.tsserver/src/forms/email.service.spec.tsserver/src/forms/forms.controller.tsserver/src/forms/forms.service.tsserver/src/forms/forms.service.spec.tsserver/src/main.tsgetJwtSecret()at boot (#257)server/src/notifications/notifications.controller.tsserver/src/projects/projects.controller.tsserver/src/projects/projects.module.tsserver/src/projects/projects.service.tsserver/src/schemas/form.schema.tsserver/src/worker.tsgetJwtSecret()at boot (#257)Hotfixes kept from main
None.
main’s conflict sides were older insecure/PII/docs snapshots.How to merge this PR
In the GitHub merge dropdown choose Create a merge commit.
mainis not a parent, merge-base staysf558f27, Merge develop into main #253 stays conflicted.Do not merge until CI is green. Do not merge #253 until this PR is on
developas a merge commit.