Skip to content

Say when a skip is probing, cross damage in doubling steps, and stop … #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
olavgg merged 1 commit into from
Sep 27, 2026
Merged

Say when a skip is probing, cross damage in doubling steps, and stop … #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -352,8 +352,18 @@ cannot: a skipped region was never read, so it has no sectors to count. A probe
of its chunks are over budget, or if drilling the ones that are would bring it
under the floor. That cost is estimated from what drilling has cost on this
drive so far; it is a cost, not a budget, and decides nothing about what counts
as over. The steps are fixed rather than doubling, so a healthy island more
than a gigabyte wide is still found.
as over. The steps start at 1 GiB and double after each slow probe, up to 64 GiB.
At a fixed gigabyte, a drive damaged to the end took thirteen thousand probes
to cross; a real one spent minutes at it with nothing on the screen. Doubling
crosses it in a couple of hundred. The price is that a healthy stretch shorter
than the step, lying between two slow probes, is condemned with them.

While a skip is probing, the drive's STATE says `probing` and the line under
it says where ("skip: probing ahead at 1.00 TiB for where the slow region
ends"); so does the status line of a single-drive scan. The percentage follows
the last point known to be slow. Before this, a skip crossing a terabyte of
damage looked like a read that never came back. The checkpoint keeps the scan's real position, because
nothing is condemned until the skip has finished.

And whatever the window says, one chunk may not eat the scan. A drill-down
that has spent 10 s of I/O on one chunk, about three times what drilling a
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ every chunk there misses its budget, nearly every sector inside comes back in
time, and the scan crawls while its WEAK count stands still. Or only a few
chunks miss it, each so full of weak sectors that retrying them takes minutes.
`--skip-slow` (on under repair) stops reading a region that is too slow to
scan either way: it probes 1 GiB on, and on, until the drive is fast again,
scan either way: it probes 1 GiB on, then doubling steps, until the drive is fast again,
bisects back to the edge, and carries on there. What it jumped over is
condemned unread, and the report says so plainly. The dashboard's **SKIP**
column shows how much that is.
Expand Down
2 changes: 1 addition & 1 deletion docs/guide.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,7 @@ <h3>Test mode</h3>
<dt id="opt-second-pass"><code>--second-pass</code></dt>
<dd>after <a href="#opt-mode"><code>--mode</code></a> write, re-read the whole device and re-verify the pattern (catches slow data decay)</dd>
<dt id="opt-skip-slow"><code>--skip-slow</code></dt>
<dd>when a region is too slow to scan -- most chunks over the last 64 MiB missed their budget, or the scan there, drill-downs and retries included, ran under the <a href="#opt-min-rate"><code>--min-rate</code></a> floor -- stop reading: probe 1 GiB on, and on, until a probe is fast, bisect back to the edge, carry on there. A chunk whose drill-down costs over 10 s is condemned whole. What was jumped over is condemned unread; the report says so and <a href="#opt-hide-bad"><code>--hide-bad</code></a> cuts it out. Sequential or reverse order only. On under repair</dd>
<dd>when a region is too slow to scan -- most chunks over the last 64 MiB missed their budget, or the scan there, drill-downs and retries included, ran under the <a href="#opt-min-rate"><code>--min-rate</code></a> floor -- stop reading: probe 1 GiB on, doubling to 64 GiB, until one is fast, bisect back to the edge, carry on there. A chunk whose drill-down costs over 10 s is condemned whole. What was jumped over is condemned unread; the report says so and <a href="#opt-hide-bad"><code>--hide-bad</code></a> cuts it out. Sequential or reverse order only. On under repair</dd>
<dt id="opt-no-skip-slow"><code>--no-skip-slow</code></dt>
<dd>read every region, however slow, under repair too</dd>
<dt id="opt-rewrite-weak"><code>--rewrite-weak</code></dt>
Expand Down
8 changes: 6 additions & 2 deletions hddscan.8
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -382,8 +382,9 @@ apiece. A region counts as too slow once three quarters of the chunks over
the last 64 MiB have missed their budget, or once scanning it \(em every
drill-down, retry and seek included \(em has run below the
.B \-\-min\-rate
floor for 20 s of I/O over at least three chunks. Then stop reading: probe 64 MiB 1 GiB further on,
and again, until a probe comes back fast, then bisect between the last slow
floor for 20 s of I/O over at least three chunks. Then stop reading: probe
64 MiB 1 GiB further on, and again, doubling the step after each slow probe up
to 64 GiB, until a probe comes back fast, then bisect between the last slow
probe and that one down to 16 MiB, with 8 MiB probes once the gap is narrower
than a wide one, and carry on scanning there. Every
probe is judged against the same calibrated budget as the scan; what drilling
Expand All @@ -394,6 +395,9 @@ I/O stops being drilled and is condemned whole. What was jumped over is
the report says so on its Coverage line and under the verdict, draws it as
.B ~
on the surface map, and lists each region;
While it is probing, the drive's state on the dashboard is
.B probing
and the line under it says where it has got to.
.B \-\-hide\-bad
cuts the regions out with the rest of the damage, and a scan with skips still
counts as covering the whole drive for it. Any skip makes the verdict at least
Expand Down
144 changes: 120 additions & 24 deletions hddscan.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,6 +448,13 @@ typedef struct {
* sectors to count.
*/
uint64_t slow, chunks, skipped;
/*
* --skip-slow is probing ahead for where a slow region ends, and has
* got to probe_off. Its reads are not the scan's, and without saying
* so the drive looked hung for as long as it took.
*/
int probing;
uint64_t probe_off;
int too_slow; /* under the throughput floor, see rate_judge() */
double lat_med, lat_avg, lat_min, lat_max;
double wlat_med; /* 0 when nothing has been written */
Expand Down Expand Up @@ -658,6 +665,8 @@ static void jrec_put(const jrec_t *j)
fprintf(f, "over %" PRIu64 " %" PRIu64 "\n", j->slow, j->chunks);
if (j->skipped)
fprintf(f, "skipped %" PRIu64 "\n", j->skipped);
if (j->probing)
fprintf(f, "probing %" PRIu64 "\n", j->probe_off);
if (j->too_slow)
fprintf(f, "tooslow 1\n");
if (j->wlat_med > 0)
Expand Down Expand Up @@ -727,6 +736,10 @@ static int jrec_get(const char *path, jrec_t *j)
sscanf(v, "%" SCNu64 " %" SCNu64, &j->slow, &j->chunks);
else if (!strcmp(k, "skipped"))
j->skipped = strtoull(v, NULL, 10);
else if (!strcmp(k, "probing")) {
j->probing = 1;
j->probe_off = strtoull(v, NULL, 10);
}
else if (!strcmp(k, "wlat"))
j->wlat_med = atof(v);
else if (!strcmp(k, "blat"))
Expand Down Expand Up @@ -2825,6 +2838,9 @@ typedef struct {
uint64_t sw_io[SKIP_WIN_MAX]; /* g_io_us when that chunk was done */
unsigned char sw_over[SKIP_WIN_MAX];
int sw_len, sw_n, sw_head, sw_cnt;
uint64_t probe_at; /* how far skip_ahead() has probed, 0 when not */
int probing; /* skip_ahead() is running */
uint64_t probe_off; /* and the byte it is probing at */
uint64_t drill_io_us; /* I/O the drill-downs have cost, and over */
uint64_t drill_chunks; /* how many chunks: what drilling one costs */
skip_t *skips;
Expand Down Expand Up @@ -4529,15 +4545,23 @@ static void progress(ctx_t *c, int final)
rate = avg_rate;
rate_judge(c, now);
{
uint64_t left = c->nchunks > c->step ? c->nchunks - c->step : 0;
uint64_t at = c->probe_at > c->step ? c->probe_at : c->step;
uint64_t left = c->nchunks > at ? c->nchunks - at : 0;

if (c->sample > 1)
left /= c->sample;
eta = rate > 0 ? (double)left * (double)c->chunk / rate : -1;
eta_avg = avg_rate > 0 ?
(double)left * (double)c->chunk / avg_rate : -1;
}
pct = c->nchunks ? 100.0 * (double)c->step / (double)c->nchunks : 100.0;
/*
* While a skip is probing ahead, how far it has got is the progress
* worth showing. Only for the screen: the checkpoint keeps the step,
* because nothing probed is condemned until the skip has finished.
*/
pct = c->nchunks ? 100.0 * (double)(c->probe_at > c->step ?
c->probe_at : c->step) /
(double)c->nchunks : 100.0;

/*
* A scan that will not finish this decade is not a scan. It happens
Expand All @@ -4549,23 +4573,42 @@ static void progress(ctx_t *c, int final)
* only symptom is a percentage that does not move, and by the time
* anyone works out why, days are gone.
*/
/*
* Only the levers not already pulled: this once told a run that had
* been given '--retries 3 --recovery-time 300' to re-run with exactly
* that. And not under --skip-slow, which is the lever of last resort
* and is already pulling: what it needs is time to probe, not advice.
*/
if (!c->stall_warned && el > 900 && eta_avg > 30.0 * 24 * 3600 &&
c->retry_ios > c->chunks_read) {
c->retry_ios > c->chunks_read && !c->skip_slow) {
char fix[128] = "";
int scsi = c->dev->transport == TR_SCSI;

c->stall_warned = 1;
if (c->retries > 3)
snprintf(fix + strlen(fix), sizeof(fix) - strlen(fix),
" --retries 3");
if (scsi && (c->rtl_state <= 0 || c->rtl_state > 300))
snprintf(fix + strlen(fix), sizeof(fix) - strlen(fix),
" --recovery-time 300");
if (scsi && c->rcd_state == 0)
snprintf(fix + strlen(fix), sizeof(fix) - strlen(fix),
" --read-cache off");
if (c->order != ORD_RANDOM && c->sample <= 1)
snprintf(fix + strlen(fix), sizeof(fix) - strlen(fix),
" --skip-slow");
msg(PROG ": %s: %sat this rate this scan needs %s%s, and %"
PRIu64 " retry reads\n"
" against %" PRIu64 " chunks says the time is going "
"into re-reading marginal\n"
" sectors rather than covering the surface. Stop it "
"and re-run with\n"
" '--retries 3 --recovery-time 300 --read-cache off' "
"to make the drive\n"
" give up quickly; sectors it cannot deliver in "
"300 ms will then be\n"
" reported as unreadable, which is the honest "
"answer for them.\n",
" '%s' so the drive gives up quickly and\n"
" regions it cannot deliver are condemned rather "
"than ground through.\n",
c->dev->name, c_yel(), human_time(eta_avg, b3, sizeof(b3)),
c_off(), c->retry_ios, c->chunks_read);
c_off(), c->retry_ios, c->chunks_read,
fix[0] ? fix + 1 : "--skip-slow");
}
/*
* The other way a scan stops covering the surface: nearly every chunk
Expand Down Expand Up @@ -4617,6 +4660,8 @@ static void progress(ctx_t *c, int final)
g_job.slow = c->chunks_slow;
g_job.chunks = c->chunks_read;
g_job.skipped = c->skip_bytes;
g_job.probing = c->probing;
g_job.probe_off = c->probe_off;
g_job.too_slow = c->too_slow;
g_job.bytes = c->bytes_done;
g_job.lat_med = med;
Expand Down Expand Up @@ -4647,6 +4692,9 @@ static void progress(ctx_t *c, int final)
if (!lw[0] && c->chunks_read)
snprintf(lw, sizeof(lw), " no read finished in the "
"last %ds", LATWIN_SECS);
if (c->probing)
snprintf(lw, sizeof(lw), " skip: probing ahead at %s",
human_size(c->probe_off, b3, sizeof(b3)));
}
fprintf(stderr, "\r%s %5.1f%% %s/s%s bad:%" PRIu64 " weak:%" PRIu64
"%s elapsed %s eta %s ",
Expand Down Expand Up @@ -5386,7 +5434,8 @@ static void check_temperature(ctx_t *c)
#define SKIP_WIN_MIN 32
#define SKIP_OVER_NUM 3 /* ... of which three quarters over */
#define SKIP_OVER_DEN 4
#define SKIP_JUMP (1ull << 30)
#define SKIP_JUMP (1ull << 30) /* the first step, doubling ... */
#define SKIP_JUMP_MAX (64ull << 30) /* ... to this */
#define SKIP_RES (16ull << 20)
#define SKIP_PROBE_BYTES (8ull << 20) /* to find an edge */
#define SKIP_WIDE_BYTES (64ull << 20) /* to find damage: 4% of it, 94% */
Expand Down Expand Up @@ -5602,8 +5651,17 @@ static int skip_probe(ctx_t *c, uint64_t s, uint64_t n, void *buf)
continue;
len = (size_t)(c->end - pos < (uint64_t)c->chunk ?
c->end - pos : (uint64_t)c->chunk);
c->probe_off = pos;
r = timed_pread(c->fd, buf, len, pos, &us, &err);
c->skip_probes++;
/*
* A probe is reading the drive, and the screen should say so:
* left out of the window, a skip crossing a terabyte of damage
* looked for minutes like a read that never came back.
*/
if (r == (ssize_t)len)
latwin_add(&c->latwin, us);
progress(c, 0);
if (r < 0 && (err == EINVAL || err == EILSEQ))
continue;
if (dev_gone(c, r, err, len))
Expand Down Expand Up @@ -5643,6 +5701,7 @@ static uint64_t skip_ahead(ctx_t *c, void *buf)
uint64_t pn = skip_probe_len(c, SKIP_PROBE_BYTES);
uint64_t wn = skip_probe_len(c, SKIP_WIDE_BYTES);
uint64_t jump = SKIP_JUMP / (uint64_t)c->chunk;
uint64_t jmax = SKIP_JUMP_MAX / (uint64_t)c->chunk;
uint64_t res = SKIP_RES / (uint64_t)c->chunk;
int to_end = 0, i;
skip_t *k;
Expand All @@ -5651,32 +5710,55 @@ static uint64_t skip_ahead(ctx_t *c, void *buf)
c->sw_n = c->sw_cnt = c->sw_head = 0;
if (c->nskip >= MAX_SKIPS)
return t + 1;
c->probing = 1;
if (jump < 1)
jump = 1;
if (res < pn)
res = pn;
/*
* Forward a gigabyte at a time until a probe comes back fast. These
* probes are wide: damage in one chunk of twenty-five slips past an
* eight-chunk probe more often than not, and the drive would be called
* clean a gigabyte on when it is not.
* Forward until a probe comes back fast: a gigabyte, then twice that
* each time a probe is still slow, up to 64 GiB. At a fixed gigabyte a
* drive damaged to the end took thirteen thousand probes to cross --
* hours of it, with nothing on the screen; doubling crosses it in a
* couple of hundred. The price is that a healthy stretch shorter than
* the step, between two slow probes, is condemned with them.
*
* These probes are wide: damage in one chunk of twenty-five slips past
* an eight-chunk probe more often than not, and the drive would be
* called clean a gigabyte on when it is not.
*/
while (!F && !to_end) {
if (g_stop || c->dev_gone)
return t + 1;
break;
p = L + jump;
if (p + wn > c->nchunks)
p = c->nchunks > wn ? c->nchunks - wn : 0;
if (p <= L) {
F = L + 1; /* too close to the end to bother */
break;
}
if (!skip_probe(c, p, wn, buf))
if (!skip_probe(c, p, wn, buf)) {
F = p;
else if (p + wn >= c->nchunks)
} else if (p + wn >= c->nchunks) {
to_end = 1;
else
} else {
if (L == t) {
uint64_t o0, l0;

steps_extent(c, t, t + 1, &o0, &l0);
msg(PROG ": %s: still slow %s on from %s; probing "
"ahead for where it ends (--skip-slow)\n",
c->dev->name,
human_size(jump * (uint64_t)c->chunk, b1,
sizeof(b1)),
human_size(o0, b2, sizeof(b2)));
}
L = p;
/* only what is known slow: the screen never goes back */
c->probe_at = L;
if (jump < jmax)
jump = jump * 2 > jmax ? jmax : jump * 2;
}
}
/*
* And back: halve the gap until the edge is known to 16 MiB. Wide
Expand All @@ -5692,6 +5774,9 @@ static uint64_t skip_ahead(ctx_t *c, void *buf)
else
F = m;
}
c->probe_at = 0;
c->probing = 0;
progress(c, 0);
if (g_stop || c->dev_gone)
return t + 1;
s1 = to_end ? c->nchunks : L == t ? t + 1 : L + 1;
Expand Down Expand Up @@ -8006,11 +8091,11 @@ static void usage(void)
" over the last 64 MiB missed their budget, or the\n"
" scan there, drill-downs and retries included, ran\n"
" under the --min-rate floor -- stop reading: probe\n"
" 1 GiB on, and on, until a probe is fast, bisect\n"
" back to the edge, carry on there. A chunk whose\n"
" drill-down costs over 10 s is condemned whole.\n"
" What was jumped over is condemned unread; the\n"
" report says so and --hide-bad cuts it out.\n"
" 1 GiB on, doubling to 64 GiB, until one is fast,\n"
" bisect back to the edge, carry on there. A chunk\n"
" whose drill-down costs over 10 s is condemned\n"
" whole. What was jumped over is condemned unread;\n"
" the report says so and --hide-bad cuts it out.\n"
" Sequential or reverse order only. On under repair\n"
" --no-skip-slow read every region, however slow, under repair too\n"
" --rewrite-weak rewrite readable-but-slow sectors in place to make\n"
Expand Down Expand Up @@ -8851,6 +8936,8 @@ static const char *jrec_word(const jrec_t *j, int format)
return "queued";
if (format)
return "formatting";
if (j->probing)
return "probing";
return j->too_slow ? "too slow" : "scanning";
}

Expand Down Expand Up @@ -9355,6 +9442,15 @@ static int tui_drive_tail(const jrec_t *x, int pw, int ident, int room,
char b[32];
int used;

if (x->state == JS_RUNNING && x->probing) {
used = tui_put(draw, " %*s ", pw, "");
tui_put(draw, "%s", c_yel());
used += tui_put(draw, "skip: probing ahead at %s for where the "
"slow region ends",
human_size(x->probe_off, b, sizeof(b)));
tui_put(draw, "%s", striped ? c_plain() : c_off());
return used;
}
if (x->state == JS_RUNNING && (x->lat_max > 0 || x->blat_max > 0)) {
static const char *const lbl[4] = {
" med ", " avg ", " min ", " max "
Expand Down
18 changes: 18 additions & 0 deletions tests/cli.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -594,6 +594,19 @@ assert_has "a region slow to the end of the drive is condemned to it" \
read -r roff rend <<< "$(json_region "$TMP/skipend.json")"
assert_eq "and the region ends where the drive does" "$rend" $((3 * G))

# REGRESSION: the forward steps were a fixed gigabyte, so a drive damaged
# from early on to the end took a probe per gigabyte to cross -- thirteen
# thousand on a 12.7 TiB drive, minutes of it with nothing on the screen.
# The steps double now; this image is 300 GiB of damage after the first.
f=$(slowimg skipfar.bin 301G)
out=$(slowrun 1G:300G --skip-slow --json "$TMP/skipfar.json" "$f")
read -r roff rend <<< "$(json_region "$TMP/skipfar.json")"
assert_eq "a drive damaged to the end is condemned to the end in doubling steps" \
"$rend" "$((301 * G))"
assert_has "and says it is probing ahead while it does" "$out" \
"probing ahead for where it ends"
rm -f "$f"

f=$(slowimg skiprev.bin 3G)
out=$(slowrun 1G:1G --skip-slow --order reverse --json "$TMP/skiprev.json" "$f")
read -r roff rend <<< "$(json_region "$TMP/skiprev.json")"
Expand Down Expand Up @@ -1070,6 +1083,11 @@ tooslow 1
EOF
assert_has "a running drive under the floor says too slow" \
"$(run --status 20200303-000000)" "too slow"
# REGRESSION: while --skip-slow probed ahead, nothing said so, and a drive
# crossing a terabyte of damage looked hung for as long as it took
echo "probing 1073741824" >> "$d/sdy.job"
assert_has "a drive whose skip is probing ahead says probing" \
"$(run --status 20200303-000000)" "probing"
kill $sp 2>/dev/null; wait $sp 2>/dev/null
run --forget 20200303-000000 >/dev/null

Expand Down
Loading
Loading