Repository navigation
Make the engine hostable in-process: context factory seam, run ids and progress events - #33
Merged
Merged
Conversation
MigrationService took IOptions<RayMigratorOptions> from the container and read the products and the CLI tools from it, while the repository settings came from the MigrationContext. A host that builds one context per run from pre-built options (RayMigrator Studio) therefore failed with "Sequence contains no matching element" as soon as the container's options did not list the product. The service now reads everything from the context of the current run, and the IOptions parameter is gone from its constructor. TemplateCache no longer depends on the options either: a host builds the cache once per process and validates each configuration against it with ValidateConfigurationAgainstTemplateCache. DirectModePipeline and the engine test host do that right after building the cache, so the CLI behaves as before. Breaking change for hosts that construct MigrationService or TemplateCache themselves (allowed before 1.0).
The factory existed but no production code used it: DirectModePipeline and
the engine test host built the MigrationContext themselves and then repeated
the same post-build steps (DAL properties, schema names, logging context),
and a host such as RayMigrator Studio that used the factory got none of
them. Pre-built options also skipped everything the options framework does
for options bound from files, so null timeouts failed later with an
InvalidOperationException and every engine used the SQL Server block
delimiter.
RayMigratorOptionsPreparer now gives options a host built in memory the
same treatment: {ENV:VARIABLE} placeholders are resolved in every string of
the options graph (a missing variable names the variable and the path), the
product defaults are merged, the enum values are probed, the data
annotations run and write their defaults back, and the rule catalog is
applied. The factory calls it in Create, skips it in
CreateFromValidatedOptions for the CLI, fills the DAL properties, checks the
schema names, and makes the context current for the calling async flow.
ProductAliasGuard carries the explained product lookup that the pipeline
had inline, so hosts get the same message.
The factory moves to Raycoon.RayMigrator.Services because it needs
DalFactory, takes the accessor and a logger, and loses the version
argument. The accessor is a singleton in both host modes. The pipeline and
the engine test host resolve the context through the factory and drop their
duplicated steps.
Breaking change for hosts that used the old factory signature (allowed
before 1.0).
The parameter was bound to its own name instead of a value, so every run row stored the literal text "FromReleaseVersion". The run now records the highest Migrated release of the product and environment at the time it starts (null for a first run). The records are read in a dedicated query right before the run row is inserted, because the run row is also the run lock and the reads that drive the run must stay behind it. GetHighestMigratedRelease is now the one definition of that "current release": Info reports it, the out-of-order check quotes it, and the run insert stamps it.
MigrationOperationResult, BaselineResult and MigrationRunInfo carry MigrationRunId, the id of the run row (0 when the run mode writes no row). The random RunId Guid was never stored anywhere, and the InitiatedBy placeholder was never filled; both are gone. MigrationFileResult carries TargetAlias and MigrationRecordId wherever an entry belongs to one target: TargetByTarget order, failures and rollbacks. A file that succeeded in FileByFile order keeps its one entry per file. MigrationRunInfo carries Environment and FromRelease; MigrationHistory, MigrationStatusInfo and BaselineResult carry Environment. GetHistoryAsync and GetStatusAsync read the runs of the requested environment only. Repository_MigrationRun_Select filtered by product alone, so the history listed the runs of every environment of the product with zero counts, and the status reported the last run of whichever environment ran last. The template takes @EnvironmentId in all five engines. MigrateUpAsync, MigrateDownAsync and BaselineAsync refuse a request whose TargetReleaseVersion differs from the MigrationContext, like the run-mode check does: the run row stamps the target release from the context, so a host that let them diverge recorded a release the run never aimed at. The engine test scenario rebuilds its host when the target release changes. Breaking change for consumers of the result DTOs (allowed before 1.0).
A host that shows the progress of a run had to parse log text: the only signal per file was the "Migration successful | ..." message, which has no event id and changes with the wording. MigrationContext.ProgressObserver now takes an IMigrationProgressObserver that receives RunStartedEvent after the run row exists (with its id), FileStartedEvent and FileCompletedEvent for every file on every target with the record id and a FileOutcome (migrated, recovered, simulated, failed, failed-ignored, rolled back, rollback failed, baselined), BlockCompletedEvent for every executed SQL block on the block-by-block, atomic and simulate paths, and RunCompletedEvent with the result. Rollbacks report MigrateDown or Rollback (error recovery) as their operation. The public operations are now thin wrappers around the existing bodies: they reset the run-started flag and raise the completion from the result, so an operation that ends before its run row exists still reports a matching start. An observer that throws is logged and the run continues; the default observer ignores every event, and the CLI sets none. The engine test scenario keeps an observer across host rebuilds, and the SQLite collection asserts the sequences for migrate-up, simulate, migrate-down, baseline, a failing file and a throwing observer.
The MigrationContext chapter gains a "Hosting the engine in-process" section with the per-run sequence a host follows, the per-process pieces, the process-global state and the progress observer; the factory section shows the new contract. The service chapter shows the constructor without IOptions, the progress events and the registration with the singleton accessor; the configuration chapter describes RayMigratorOptionsPreparer in place of the "pre-built options are trusted" paragraph; the DI chapter, the template-executor chapter and the coding conventions follow. ADR-023 records the decision, and CLAUDE.md names the seam.
The architecture pattern and component tables still showed the Guid RunId and the InitiatedBy placeholder, and the external-DAL table and the example plugin's template skeleton described Repository_MigrationRun_Select as a query by product; it takes @EnvironmentId as well now.
The migrate-up catch blocks read the target and the record id from MigrationState, which is assigned after the record insert, so a failure before the insert was attributed to the previous file and target, and the Failed update went to the previous record. The loops now keep the pair they work on in locals; a failure before the insert marks no record. Every FileCompletedEvent now follows a FileStartedEvent for the same file and target: a record a previous run completed but never finalized reports a start before Recovered; a rollback reports its start once per record, so the paths that end before the rollback file is parsed (missing file, lookup failure, the outer catch) pair as well, and a record without a rollback file reports the new outcome Skipped instead of nothing; a baseline reports the start after the record exists, and a failure completes the file it hit. The rollback start carries the resume offset, the start of a CLI-tool file one block, and a file-by-file run stamps the start time per target. The atomic paths raise their block events after the commit, so a transient retry of the batch repeats no event. Simulate and Validate set FromReleaseVersion from the records they read, so RunStartedEvent reports where the product stands without a run row. BaselineResult.BaselinedRecords counts the records written, one per file and target, also when the baseline failed midway; the baseline run events use it. BaselinedFiles keeps its meaning.
The repository ids, the run row and the settings snapshot take the environment from the MigrationContext, while file discovery and the environment-specific migsettings take it from the request, so a host that let them diverge ran one environment's files against another environment's rows. MigrateUpAsync, MigrateDownAsync and BaselineAsync now check the environment like the run mode and the target release. AllowOutOfOrder is decided by the request alone; the settings snapshot of a migrate-up run now records the request value instead of the console options, which a host may not have set.
MigrationContextFactory replaced the pipeline's post-build steps but dropped the report of each DAL's transient error codes (ADR-022), so a TransientErrorCodes.txt that replaced the built-in list, and an empty list that disables every retry, went unlogged; the pipeline's helper had become dead code. The factory now logs the source when it adds the DAL properties, and makes the context current before its first log line, so the enricher stamps the schema-name warning with the run properties. The interface documentation names the actual schema-name contract: a missing name is an error only where the database requires one, a name on a database without schemas is a warning.
RayMigratorOptionsPreparer resolved the {ENV:...} placeholders in place
and then ran the rule catalog without a section as written, so a host
whose connection string supplied its password through a placeholder
got the RULE_7_3 hint that the CLI stopped giving in #30. The preparer
now builds the as-written section from the values it replaced, keyed
by their configuration path, and the validator checks those as written
and everything else as bound.
The tests that create a context through the factory registered the options' secrets with the global SensitiveDataMasker, which runs in parallel with the masker tests and made them order-dependent. Each test now opens its own masker scope, as the preparer tests already did.
The per-environment history, the release a run starts from and the progress events were tested on SQLite only. The tests now run against PostgreSQL, SQL Server, MariaDB and MySQL as well, as the testing convention requires for engine behaviour.
Six documents still showed the constructor that took the options and a validateConfiguration flag. They now describe the constructor without options and the explicit ValidateConfigurationAgainstTemplateCache call per configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the engine cleanly hostable in-process by a long-running host that runs one migration per request with options it builds in memory (RayMigrator Studio). This is work package 1 of the Studio target architecture (
Docs/00-target-architecture.mdin the Studio repository). Six commits, one per change, plus seven review-fix commits (see below):MigrationServicereads the configuration of the current run from theMigrationContextinstead ofIOptions<RayMigratorOptions>;TemplateCacheno longer takes the options and is validated per configuration. Before, a context built from other options than the container's failed with "Sequence contains no matching element".IMigrationContextFactoryis the entry point of a host for one run (the CLI pipeline included).Createprepares in-memory options with the newRayMigratorOptionsPreparer({ENV:VARIABLE}placeholders in the whole options graph, defaults merge, enum probe, data annotations with their defaults written back, rule catalog);CreateFromValidatedOptionsskips that for the CLI. Both fill the DAL properties (a host that skipped this used the SQL Server block delimiter for every engine), check the schema names, register the secrets with the masker and make the context current.ProductAliasGuardexplains an unknown product alias to hosts the way the CLI did. The factory moves toServices, takes the accessor and a logger, and loses the version argument; the accessor is a singleton in both host modes.MigrationRun.FromReleaseVersionholds the release the run started from instead of the literal textFromReleaseVersion(the parameter was bound to its own name).MigrationRunId, replacing a Guid that was never stored) and the target (TargetAlias,MigrationRecordId);MigrationRunInfocarriesEnvironmentandFromRelease, the never-filledInitiatedByis gone. History and status read the runs of the requested environment only (Repository_MigrationRun_Selecttakes@EnvironmentIdin all five engines; before, the history listed every environment's runs with zero counts). The three run operations refuse a request whose target release differs from the context, like the run-mode check does.MigrationContext.ProgressObserverreceives run, file-on-target and block events with ids and outcomes (Raycoon.RayMigrator.Core.Progress). An observer that throws is logged and the run continues; the CLI sets none.Review fixes
A review of the branch produced 15 findings; seven further commits address them, one per group:
MigrationState, which is assigned after the record insert, so a failure before the insert was attributed to the previous file and target and marked the previous record Failed. EveryFileCompletedEventnow follows aFileStartedEventfor the same file and target (recovered records, rollbacks that never reach their file, baselines), a rollback without a rollback file reports the new outcomeSkipped, the atomic paths raise their block events after the commit, Simulate and Validate reportFromRelease, the start of a CLI-tool file carries one block, andBaselineResult.BaselinedRecordscounts the records written (also when the baseline failed midway).Environmentdiffers from the context; the settings snapshot records theAllowOutOfOrdervalue of the request.RayMigratorOptionsPreparerchecks connection strings that contain placeholders as written, so a host does not get the RULE_7_3 hint that RULE_7_3 reports a hardcoded credential for connection strings that use an {ENV:} placeholder #30 removed for the CLI.FromReleaseVersionand the progress events are tested on all five engines, not on SQLite only.TemplateCacheconstructor that took the options.Not changed, deliberately: the file-by-file result list keeps one entry per file (the multi-target tests pin the counts; the events are per target),
GetHighestMigratedReleasekeeps its ordinal comparison (pre-existing; a shared release comparer is a separate change), and the pipeline still resolves the context before the database sink is wired, becauseTemplateExecutorlogs the context while the sink initializes.Related Issue
No issue; see the Studio target architecture (Studio repository,
Docs/00-target-architecture.md, E1-E5 and E11).Changes
MigrationServiceandTemplateCacheconstructors, theIMigrationContextFactorycontract and its implementation's assembly, the result DTOs ofIMigrationService, and theRepository_MigrationRun_Selecttemplate parameters.DirectModePipelineand the engine test host resolve the context through the factory and lose their duplicated post-build steps.CHANGELOG.md: entries under Unreleased (Added, Changed, Fixed).Testing
RayMigratorOptionsPreparer,MigrationContextFactory,GetHighestMigratedRelease, run-insert parameters, the as-written section of the preparer; 2185 tests green)dotnet test Raycoon.RayMigrator.Tests.Unit/)Tests.Enginegreen on SQLite (232), PostgreSQL (217), SQL Server (229), MariaDB (219), MySQL (219); new engine tests on all five engines forFromReleaseVersion, the per-environment history and the progress event sequences (migrate-up, simulate withFromRelease, migrate-down, migrate-down without a rollback file, baseline, failing file, throwing observer);dotnet build -c Releasefor net8.0/net9.0/net10.0 without warnings;inforun of the CLI against the Docker SQL Server configuration.CLA