Skip to content

Restrict import resume and abort to the unfinished lifecycle owner #430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adamziel wants to merge 8 commits into
base: trunk
Choose a base branch
from
Open

Restrict import resume and abort to the unfinished lifecycle owner #430

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
fd1bb00
Keep unrelated import state across command aborts
adamziel Jul 30, 2026
70ad59b
Remove stale SQL artifacts on db-pull abort
adamziel Jul 30, 2026
5dd9e56
Enforce import lifecycle ownership before aborting
adamziel Jul 31, 2026
4096ecb
Complete database index resumes after interrupted responses
adamziel Jul 31, 2026
08d88be
Merge trunk into preserve-command-abort-state
adamziel Jul 31, 2026
422ed6d
Persist import checkpoint provenance and cleanup phase
adamziel Jul 31, 2026
3f23d31
Protect starter scopes and fresh command initialization
adamziel Jul 31, 2026
ee39825
Merge trunk into preserve-command-abort-state
adamziel Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -720,4 +720,13 @@ php reprint.phar <command> <URL> --state-dir=DIR --fs-root=DIR [options]
* `flat-docroot` — Reassemble pulled files into a standard WordPress directory layout using symlinks. Useful when the source site has a non-standard layout (e.g. WP Cloud with ABSPATH separate from wp-content).
* `apply-runtime` — Generates server configuration files (`runtime.php`, `start.sh` or `nginx.conf`) from preflight data. See [Step 6](#step-6--generate-runtime-configuration).

All commands except `preflight-assert` support `--abort` to abort the current sync and exit. For `files-pull`, this clears sync progress but keeps the remote index and downloaded files — the next run performs a delta sync. For `db-pull` and `db-index`, it clears the output file so the next run starts from scratch. Interrupted commands automatically resume from the last saved cursor.
The resumable commands `pull`, `pull-files`, `pull-db`, `files-pull`,
`files-index`, `db-pull`, `db-index`, and `db-apply` support `--abort`. An
unfinished import belongs to the command that started it; only that exact
command may resume or abort it. If another command is requested, the error
names the owning command and gives its exact resume and abort invocations.
Abort ignores options used only to perform normal work. File aborts retain
downloaded files and `pull/remote-index.jsonl`, so the next file pull computes a
delta. Database download aborts discard their SQL, table-index, domain, buffer,
and statistics files, while `db-apply` aborts retain its SQL/domain inputs and
do not alter the external database target.
873 changes: 643 additions & 230 deletions packages/reprint-importer/src/import.php
View file
Open in desktop

Large diffs are not rendered by default.

269 changes: 28 additions & 241 deletions packages/reprint-importer/src/lib/pull/class-pull.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,10 @@ public function stage_label(string $stage): string
/**
* Run the pull pipeline.
*/
public function run(array $options): void
public function run(
array $options,
bool $resume_pipeline = false
): void
{
$this->normalize_url();
$this->progress->set_mode('pipeline');
Expand All @@ -109,48 +112,18 @@ public function run(array $options): void
'pull',
$this->stages($options),
$options,
'Pulling'
'Pulling',
$resume_pipeline
);
}

/**
* Handle --abort for high-level pull commands.
*
* File pipelines keep downloaded site files in place. The database
* pipeline removes stale database artifacts so the next pull-db fetches
* and applies a fresh dump.
*/
public function abort(string $command = 'pull'): void
{
$state = $this->client->get_state();
if (
$command === 'pull-files'
|| (
$command === 'pull'
&& $state->active_resumable_command->command_name === 'files-pull'
)
) {
$this->client->clear_files_pull_progress();
}
$this->prepare_repull($command);
$label = $command === 'pull' ? 'Pull' : $command;
$message = "{$label} state cleared.";
$message .= $command === 'pull-db'
? " Database artifacts will be downloaded again."
: " Downloaded files are preserved.";
$this->progress->show_lifecycle_line("{$message}\n");
$this->client->output_progress([
"type" => "lifecycle",
"event" => "aborted",
"command" => $command,
"message" => $message,
], true);
}

/**
* Run only the file stages from the pull pipeline.
*/
public function run_pull_files(array $options): void
public function run_pull_files(
array $options,
bool $resume_pipeline = false
): void
{
$this->normalize_url();
$this->progress->set_mode('pipeline');
Expand All @@ -169,14 +142,18 @@ public function run_pull_files(array $options): void
'pull-files',
['preflight', 'files-pull'],
$options,
'Pulling files from'
'Pulling files from',
$resume_pipeline
);
}

/**
* Run only the database stages from the pull pipeline.
*/
public function run_pull_db(array $options): void
public function run_pull_db(
array $options,
bool $resume_pipeline = false
): void
{
$this->normalize_url();
$this->progress->set_mode('pipeline');
Expand All @@ -189,7 +166,8 @@ public function run_pull_db(array $options): void
'pull-db',
['preflight', 'db-pull', 'db-apply'],
$options,
'Pulling database from'
'Pulling database from',
$resume_pipeline
);
}

Expand All @@ -211,122 +189,19 @@ private function run_pipeline(
string $command,
array $stages,
array $options,
string $title
string $title,
bool $resume_pipeline
): void {
$state = $this->client->get_state();
$pull_pipeline = $state->pull_pipeline->started_by_command;
$pull_stage = $state->pull_pipeline->last_completed_stage;
$stage_sequence = $state->pull_pipeline->stage_sequence;
if (!is_array($stage_sequence) || $stage_sequence === []) {
$stage_sequence = $stages;
}
$pipeline_final_stage = $stage_sequence[count($stage_sequence) - 1] ?? null;
$state_command = $state->active_resumable_command->command_name;
$state_status = $state->active_resumable_command->completion_state;
$completed_stage = $pull_pipeline === $command ? $pull_stage : null;
$completed_pipeline =
$pull_pipeline !== null &&
$pull_stage !== null &&
$pipeline_final_stage !== null &&
$pull_stage === $pipeline_final_stage;

if ($completed_pipeline) {
$this->prepare_repull($command);
if ($resume_pipeline) {
$stages = $state->pull_pipeline->stage_sequence;
$completed_stage = $state->pull_pipeline->last_completed_stage;
} else {
// A completed standalone command is not this pipeline's
// checkpoint. Start every selected stage fresh rather than
// allowing that historical checkpoint to skip its matching stage.
$this->client->prepare_fresh_pull_pipeline($command, $stages);
$completed_stage = null;
$pull_pipeline = null;
$pull_stage = null;
$state_command = null;
$state_status = null;
}

$pipeline_has_resume_state =
$pull_pipeline !== null &&
(
$pull_stage !== null ||
$state_command !== null ||
$state_status !== null
);

if ($pipeline_has_resume_state && $pull_pipeline !== $command) {
throw new RuntimeException(
"Another command is already in progress: {$pull_pipeline}. " .
"Rerun {$pull_pipeline} to resume it. Only use --abort if you want to discard " .
"that pipeline's resume state before running {$command}."
);
}

$has_direct_command_state = !$pipeline_has_resume_state && $state_status !== null;
if (
$has_direct_command_state &&
$state_status !== 'complete' &&
in_array($state_command, ['files-pull', 'db-pull', 'db-apply'], true) &&
!in_array($state_command, $stages, true)
) {
throw new RuntimeException(
"Another command is already in progress: {$state_command}. " .
"Rerun {$state_command} to resume it. Only use --abort if you want to discard " .
"that command's resume state before running {$command}."
);
}

if ($has_direct_command_state && $state_status === 'complete') {
// Users can run lower-level commands directly, e.g.
// `reprint files-pull` or `reprint db-pull`, without going through
// this pull pipeline. Those commands save their own completion
// state in active_resumable_command. That state must not make a
// high-level command skip its matching stage: the pipeline has not
// recorded that stage as complete. Clear the direct command
// checkpoint first so the stage computes a fresh delta.
$state_dir = $this->client->state_dir;
if ($state_command === 'files-pull' && in_array('files-pull', $stages, true)) {
// Keep the remote index, but clear transient files-pull state
// so this pipeline downloads a next remote index and compares
// it with the remote index.
$state = $this->client->get_state();
$state->active_resumable_command->command_name = null;
$state->active_resumable_command->completion_state = null;
$state->active_resumable_command->remote_cursor = null;
$state->active_resumable_command->current_stage = null;
$state->consecutive_interrupted_responses = 0;
$state->current_file = null;
$state->current_file_bytes = null;
$state->diff = new FileDiffProgressState();
$state->index = new RemoteFileIndexCursorState();
$state->fetch = new FetchListProgressState();
$state->fetch_skipped = new FetchListProgressState();
$state->files_pull_summary = new FilesPullSummaryState();
$state->files_pull_only_fingerprint = null;
$this->client->save_state();
foreach ([
"{$state_dir}/pull/remote-index.next.jsonl",
"{$state_dir}/pull/fetch-list.jsonl",
"{$state_dir}/pull/skipped-fetch-list.jsonl",
] as $path) {
if (file_exists($path)) {
@unlink($path);
}
}
} elseif ($state_command === 'db-pull' && in_array('db-pull', $stages, true)) {
// Discard database dump artifacts from any previous runs.
$state = $this->client->get_state();
$state->active_resumable_command->command_name = null;
$state->active_resumable_command->completion_state = null;
$state->active_resumable_command->remote_cursor = null;
$state->active_resumable_command->current_stage = null;
$state->consecutive_interrupted_responses = 0;
$state->sql_bytes = null;
$state->db_index = new DatabaseTableIndexState();
$this->client->save_state();
foreach ([
"{$state_dir}/db.sql",
"{$state_dir}/db-tables.jsonl",
"{$state_dir}/pull/domains.json",
] as $path) {
if (file_exists($path)) {
@unlink($path);
}
}
}
}

$total = count($stages);
Expand Down Expand Up @@ -757,94 +632,6 @@ private function normalize_url(): void
}
}

/**
* Reset sub-command state for a delta re-pull.
*
* Keeps preflight data in place, then clears only the checkpoint groups
* owned by the high-level command being restarted. Keeping that ownership
* map here prevents callers from having to know which file/database state
* belongs to which pipeline.
*/
private function prepare_repull(string $command): void
{
$state_dir = $this->client->state_dir;
switch ($command) {
case 'pull':
$reset_file_transfer_state = true;
$reset_file_selection_state = false;
$reset_db_state = true;
break;

case 'pull-files':
$reset_file_transfer_state = true;
$reset_file_selection_state = true;
$reset_db_state = false;
break;

case 'pull-db':
$reset_file_transfer_state = false;
$reset_file_selection_state = false;
$reset_db_state = true;
break;

default:
throw new InvalidArgumentException("Unknown pull command: {$command}");
}


$state = $this->client->get_state();
$state->pull_pipeline->started_by_command = $command;
$state->pull_pipeline->stage_sequence = [];
$state->pull_pipeline->last_completed_stage = null;
$state->pull_pipeline->files_filter = null;
$state->pull_pipeline->skipped_pending = false;
$state->pull_pipeline->has_completed_once = true;
$state->active_resumable_command->command_name = null;
$state->active_resumable_command->completion_state = null;
$state->active_resumable_command->remote_cursor = null;
$state->active_resumable_command->current_stage = null;
$state->consecutive_interrupted_responses = 0;
if ($reset_file_transfer_state) {
$state->current_file = null;
$state->current_file_bytes = null;
$state->diff = new FileDiffProgressState();
$state->fetch = new FetchListProgressState();
$state->fetch_skipped = new FetchListProgressState();
$state->files_pull_summary = new FilesPullSummaryState();
}
if ($reset_file_selection_state) {
$state->index = new RemoteFileIndexCursorState();
$state->files_pull_only_fingerprint = null;
}
if ($reset_db_state) {
$state->sql_bytes = null;
$state->db_index = new DatabaseTableIndexState();
$state->apply = new DatabaseApplyCommandState();
$state->sql_output = null;
}
$this->client->save_state();

$paths = [];
if ($reset_file_transfer_state) {
$paths[] = $state_dir . "/pull/remote-index.next.jsonl";
$paths[] = $state_dir . "/pull/fetch-list.jsonl";
$paths[] = $state_dir . "/pull/skipped-fetch-list.jsonl";
}
if ($reset_db_state) {
$paths[] = $state_dir . "/db.sql";
$paths[] = $state_dir . "/db-tables.jsonl";
$paths[] = $state_dir . "/pull/domains.json";
}

foreach ($paths as $path) {
if (file_exists($path)) {
@unlink($path);
}
}

$this->client->audit_log(strtoupper($command) . " | prepared for delta re-pull", true);
}

/**
* Lower-level commands return with completion_state="partial" when a
* server timeout drops the connection. This loop retries automatically,
Expand Down
15 changes: 13 additions & 2 deletions packages/reprint-importer/src/lib/state/class-pull-state.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ class ResumableCommandCheckpointState
/** @var string|null Lower-level command name, e.g. files-pull/db-pull/db-apply. */
public ?string $command_name = null;

/** @var string|null User-facing command which started this checkpoint. */
public ?string $started_by_command = null;

/** @var string|null Completion state: in_progress, partial, complete, or null before start. */
public ?string $completion_state = null;

Expand All @@ -26,8 +29,14 @@ class ResumableCommandCheckpointState
public static function from_array(array $data): self
{
$state = new self();
// State written before checkpoint provenance was persisted has no
// reliable way to distinguish pipeline work from a later direct run.
if (!array_key_exists('started_by_command', $data)) {
$data['started_by_command'] = null;
}
reprint_assert_state_keys($data, array_keys($state->to_array()), self::class);
$state->command_name = $data['command_name'];
$state->started_by_command = $data['started_by_command'];
$state->completion_state = $data['completion_state'];
$state->current_stage = $data['current_stage'];
$state->remote_cursor = $data['remote_cursor'];
Expand All @@ -38,6 +47,7 @@ public function to_array(): array
{
return [
'command_name' => $this->command_name,
'started_by_command' => $this->started_by_command,
'completion_state' => $this->completion_state,
'current_stage' => $this->current_stage,
'remote_cursor' => $this->remote_cursor,
Expand Down Expand Up @@ -298,7 +308,7 @@ public function to_array(): array

class PullPipelineCheckpointState
{
/** @var string|null User-facing pipeline command that owns the checkpoint. */
/** @var string|null User-facing pipeline command that started the checkpoint. */
public ?string $started_by_command = null;

/** @var string[] Ordered stage names for the pipeline currently being resumed. */
Expand Down Expand Up @@ -346,7 +356,8 @@ public function to_array(): array
* In-process pull state with typed properties for each persisted field.
*
* This object mirrors pull/state.json. Add new persistent state here first;
* from_array() requires the complete current schema.
* from_array() validates the on-disk schema, with explicit nested defaults
* for fields added after state files entered use.
*/
class PullState
{
Expand Down
Loading
Loading