feat(document): adaptive contour-Canny for low-contrast backgrounds#673
Merged
Barnabas A Nsoh (ayinloya) merged 193 commits intoJun 26, 2026
Conversation
…deploy-preview (#586) * feat: add manual workflow_dispatch trigger with skip_tests option to deploy-preview * feat: add manual workflow_dispatch trigger with skip_tests option to deploy-preview * fix: remove unnecessary if condition from share-preview-url step * refactor: rename step id set_dest_dir_hosted_web to set_dest_dir_embed for clarity * feat: add manual workflow_dispatch trigger to destroy-preview with safe branch handling * chore: limit preview comment step to pull_request events * fix: use safe inputs expression for skip_tests across all trigger types
…oup across 1 directory (#582) chore(deps-dev): bump vite in the npm_and_yarn group across 1 directory Bumps the npm_and_yarn group with 1 update in the / directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite). Updates `vite` from 7.2.2 to 7.3.2 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/v7.3.2/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v7.3.2/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 7.3.2 dependency-type: direct:development dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Barnabas A Nsoh <banasco@gmail.com>
The new Preact-based component was missing shadow DOM, which caused
Cypress tests to fail when looking for .back-button in the shadow root.
The original JS component used attachShadow() but preact-custom-element
defaults to no shadow DOM. Adding { shadow: true } option fixes the test.
The embed tests expect an element with id="take-photo" in the document capture instructions component. The new Preact component was missing this ID, causing test failures.
* feat(web-components): update Navigation to match new design - Replace circular colored icon buttons with minimal 40x40px semi-transparent buttons - Use simple white arrow icon for back button (no text label) - Use simple white X icon for close button - Add hover and focus-visible states - Move button labels to aria-label for accessibility - Update stories with dark background to visualize white icons * Update navigation to Figma tokens and parent-controlled padding * Fix dependency * fix(web-components): address PR review comments on Navigation * fix(web-components): improve Navigation hover performance and story visibility * fix(web-components): add appearance resets and decouple theme-color to icon
…ated Side-mounted capture/gallery buttons were keyed on useLandscapeUi, which stays true for landscape doc types (id-card, passport) even on desktop where rotation is suppressed. Switch to shouldRotateUi so the bottom row renders the buttons whenever the UI isn't actually rotated.
Add https local host for mobile testing Show guide throughout capture
Add focusMode continuous as constraints Comment out 4k resolution
* feat(web-components): new document capture instructions screen
* fix(web-components): enable shadow DOM for DocumentCaptureInstructions
The new Preact-based component was missing shadow DOM, which caused
Cypress tests to fail when looking for .back-button in the shadow root.
The original JS component used attachShadow() but preact-custom-element
defaults to no shadow DOM. Adding { shadow: true } option fixes the test.
* fix(web-components): add take-photo id to start button for test compat
The embed tests expect an element with id="take-photo" in the document
capture instructions component. The new Preact component was missing
this ID, causing test failures.
Reject card-shaped quads framed by straight background lines (parquet floors, slatted tables) rather than a real document border. HoughLinesP runs lazily on the closed contour edge map; a candidate is rejected when >= 2 of its edges sit on "through-lines" that overshoot its corners. - New pure helper detection/seamRejection.ts (classifyEdgesOnThroughLines / isSeamFalseQuad) + unit spec — no OpenCV dependency, mirrors qualityScoring.ts. - Wired into the candidate-acceptance gate in useCardDetection.ts; gated on settings, additive (off => byte-identical behaviour). - Tunable via TuningPanel (seamRejectEnabled / houghThreshold / houghMinLengthRatio / houghMaxLineGap) with Hough Lines / Seam Rejected debug metrics. Verified: opencv.js 4.8.0 build ships HoughLinesP (runtime-confirmed callable). Unit + routing + detection specs green; lint + type-check clean.
Collaborator
Author
|
…rash Rounding the downscaled ROI x and width independently could push x+width one pixel past the dsCanvas when the guide box sits flush against the frame edge, tripping the OpenCV matrix.cpp assertion (-215: 0 <= roi.x && roi.x + roi.width <= m.cols) and surfacing as a recurring "CV Errors: retrying". Clamp x/y to the canvas first, then size width/height to the remaining span so x+w <= cols and y+h <= rows always hold. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…early-out Three regression-safe detection improvements for hard backgrounds, all gated and one-directional (cannot newly reject a previously-working capture): - Seam-rejection clutter guard (seamMaxHoughLines, default 60): skip the through-line seam test when HoughLinesP floods the map (>60 lines). A woven fabric/carpet produces 400+ lines so every real card has edges on overshooting lines and was wrongly rejected; a parquet shows only a handful, so the guard leaves the seam test active where it works. - Clutter-adaptive Canny high-threshold floor (lowClutterEdgeDensity 2, cannyHighMinLowClutter 40): on a near-empty scene drop the 60 floor to 40 so a faint pale-card-on-pale-wood border can still form a quad. Busy/high-contrast scenes keep the proven 60 floor, so the working path is byte-identical. - Gate-0 grid coverage relaxed to an early-out (captureGridMinCells, default 4): the capture-phase 7/9 bar false-rejected low-contrast cards on plain backgrounds (only the printed center cells carry edges) before the contour pass ran. Distance is already enforced downstream by docFillPercent >= minFillPercent (65%); synthetic-fallback eligibility keeps its own 7/9 signal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reduce per-frame capture flicker (feedback ping-ponging between "Align" and "Hold Still" and the progress ring draining) on the slightest movement, and surface which gate is blocking capture in the tuning panel. Anti-flicker (mobile only, gated by gateDecayEnabled; desktop unchanged via the hook's ?? fallbacks): - softFailStability(): generalize the blur/glare decay-within-tolerance pattern to the distance / chroma-content / desktop-overflow gates and the soft Gate-0 (coverage-flicker) case. On a transient miss the stability count decays by 1 (within BEST_FRAME_MISS_TOLERANCE) and the displayed state holds, instead of zeroing the count and downgrading the state. It can only ever decrement or hard-reset the counter, never increment, so capture cannot fire on a non-compliant frame. Genuine "document gone" exits (off-guide, no-contour past tolerance, Gate-0 absent, lifecycle reset) still hard-reset. - EMA on docFillPercent (docFillEmaAlpha 0.3) + asymmetric hysteresis deadband (fillHysteresis 3 pts) so a hand hovering on the 65%/95% fill thresholds doesn't toggle the distance gate. EMA reset on every "document gone" path. - Route the desktop video-border color and the two desktop progress feeds through the already-debounced visibleComplianceState (mobile already did). Diagnostics (debug-only, gated by IS_DEBUG_MODE): - Every detection terminal now emits a rejectReason; the tuning panel shows a "Blocking check" readout (with [held] when a transient miss was absorbed) plus sliders for the seam/clutter/grid and anti-flicker knobs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ground When no 4-corner quad forms in luminance, segment a strongly coloured card (e.g. a green/yellow ID) from a near-neutral background (grey fabric) by chroma magnitude: threshold |a-128|+|b-128| on the Lab a/b channels, take the largest blob, and accept it as a real contour only if it passes the same coverage-band / 4-vertex / fill / aspect / wall-hug gates. Gated to mobile (needs the chroma fusion path) behind chromaMaskFallback; tunable via chromaMaskThreshold / chromaMaskMinFrac / chromaMaskMaxFrac and exposed in the tuning panel with a live metric. KNOWN LIMITATION: a colourful rug/carpet patch is classically indistinguishable from a card (geometry, fill, aspect and internal-edge density all overlap) and can pass this path, so it can auto-capture a rug. Toggle off via the panel or chromaMaskFallback if it false-captures; the long-term fix for this class is ML segmentation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Throttle the heavy CV pipeline to a configurable processing rate (default 30fps) to cut ~2x CPU/battery/thermal on mobile and steady the cadence: - Time-based gate in processFrame (performance.now delta), placed before any Mat/canvas/state work, so skipped ticks cost nothing and the UI holds. Chosen over frameCount % N because rAF runs at the display refresh (60/90/120Hz/ adaptive), so a modulo would make the real rate — and every frame-count constant — device-dependent; the time gate self-corrects under thermal load. - targetProcessingFps tunable (SHARED_DEFAULTS 30) + panel slider (60 = off) + a live "Proc FPS" debug metric. - Light re-tune for the 2x cadence drift: docFillEmaAlpha 0.3 -> 0.45 (EMA now updates half as often) and DISCOVERY_TIMEOUT_FRAMES 60 -> 30 (~1s at 30fps). Also fix the guide border not turning green during the Hold-Still (STABLE) phase: that phase is only ~5 frames, shorter than the 150ms compliance debounce, so STABLE never surfaced before CAPTURING preempted it. Make the debounce asymmetric — snap immediately into the green states (STABLE/CAPTURING/SUCCESS), keep the 150ms trailing debounce only when downgrading to DETECTING/IDLE so a transient miss doesn't flash the border back to amber. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Barnabas A Nsoh (ayinloya)
requested review from
Kwame Yeboah (Yeboahmedia),
Ed Fricker (beastawakens) and
Solomon Nsubuga (solnsubuga)
Solomon Nsubuga (solnsubuga)
approved these changes
Jun 26, 2026
Barnabas A Nsoh (ayinloya)
deleted the
feat/mobile-capture-adaptive-canny
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Why
Document auto-capture on mobile only fires reliably when the document sits on a high-contrast surface (e.g. a card on metal) and stalls on general backgrounds (wood, matte desk, similar-toned paper).
Root cause: the contour gate — the stage that actually drives capture, since the distance/shape/fill checks all depend on finding the 4-corner outline — used a fixed
cv.Canny(blurred, edges, 50, 150). That fixed pair needs a strong brightness gradient at the document border; on low-contrast backgrounds the gradient is too weak, the contour is never found, and capture never triggers.What
Make the contour-stage Canny thresholds adaptive per frame, derived from the frame's own gradient-magnitude statistics:
blurredimage →mean,stddev.high = clamp(mean + autoCannySigma·stddev, 60, 150),low = max(15, 0.4·high).cv.Cannykeeps its default gradient norm, so behaviour at the cap is unchanged.The 150 ceiling is the previously-fixed value, so the high-contrast/metallic path that already works is byte-identical — this change only relaxes detection for low contrast, never tightens it. The 60 floor lets faint borders on plain backgrounds produce a contour so distance/shape gating can succeed.
Why gradient-stats, not pure-median auto-Canny
A raw-median anchor tracks brightness, so a bright-but-plain desk would push thresholds up and worsen low-contrast detection. Canny thresholds operate on gradient magnitude, so anchoring on the gradient distribution is what actually relaxes detection when the border is faint.
Tuning
autoCannySigmasetting (default1.0, mobile + desktop).mergeDebugInfohelper) so the resolved thresholds are visible per surface.Files
hooks/useCardDetection.ts—CANNY_HIGH_MAX/MINconstants; adaptive Canny in the contour pass.DocumentAutoCapture.tsx—autoCannySigmadefault.components/TuningPanel.tsx— slider + metric.Testing
type-check, ESLint, Prettier all clean.[60,150]band andσ=1.0are reasoned defaults, not measured. Open ⚙️ Settings on a phone, watch Canny (lo/hi) on plain vs. metallic surfaces, and tune σ so plain backgrounds capture without false-firing on busy ones. The winning σ should be locked intogetOptimalDefaultsbefore this leaves draft.🤖 Generated with Claude Code
PR Type
Enhancement, Bug fix
Description
Adaptive per-frame Canny thresholds from gradient statistics for low-contrast backgrounds
Chroma (Lab a/b) edge fusion to detect luminance-invisible card borders
Tilt-robust shape gates using
cv.minAreaRectinstead of axis-aligned bboxComposite quality scoring, chroma-content gate, and transient-miss tolerance
Diagram Walkthrough
File Walkthrough
DocumentAutoCapture.tsx
Refactor settings into shared defaults with device overridespackages/web-components/lib/components/document/src/document-auto-capture/DocumentAutoCapture.tsx
SHARED_DEFAULTS) and per-device overrides(
MOBILE_OVERRIDES,DESKTOP_OVERRIDES) into named constantsautoCannySigma,chromaEdgeFusion,chromaCannyLow/High,mobileRegionFallback,idAspectTolerance,bookDocAspectTolerance,minFillRatio,chromaContentGate,minChromaContentgetOptimalDefaults()to spread shared + device-specificoverrides
TuningPanel.tsx
Extend tuning panel with adaptive Canny and chroma controlspackages/web-components/lib/components/document/src/document-auto-capture/components/TuningPanel.tsx
aspect tolerance, fill ratio, and content gate
chroma, and quality score
ratio, chroma Canny thresholds, and min chroma content
fallback, and chroma content gate
useCardDetection.ts
Adaptive Canny, chroma fusion, tilt-robust gates, and quality scoringpackages/web-components/lib/components/document/src/document-auto-capture/hooks/useCardDetection.ts
gradient magnitude statistics (mean + σ·stddev, clamped to [60, 150])
luminance edge map for low-contrast borders
boundingRectwithcv.minAreaRectfortilt-invariant aspect ratio and fill ratio measurement
near-monochrome objects (keyboards, blank paper)
aspect, contour, chroma) replacing sharpness-only best-frame selection
BEST_FRAME_MISS_TOLERANCE)to avoid discarding good captures on momentary bad frames
tunable settings
DocumentCaptureInstructions.tsx
Remove unused Ref importpackages/web-components/lib/components/document/src/document-capture-instructions/DocumentCaptureInstructions.tsx
Reftype import from preact