A research project for identifying potentially manipulative interface patterns in website and app screenshots using visual, textual, and structural evidence.
Based on Misleading Patterns — CDS Onboarding, including its model-design diagrams.
All four implementation-plan tasks and UI embedding integration completed; the full text baseline is evaluated. The prototype includes eight real ContextDP text examples, a TF-IDF/logistic-regression baseline, a working OmniParser screenshot wrapper, and CLIP screenshot encoding with fusion forward/backward checks. The merged data branch adds 12 labeled e-commerce text examples and their preparation script. Training on the full e-commerce corpus produced 94.16% accuracy and 0.9414 macro-F1 on a page-grouped holdout of 565 texts. The UI adapter encodes real parsed text and boxes with frozen MiniLM and a trainable projection, including padding masks. See the implementation plan and the data, baseline, UI parsing, and visual encoder reports. Multimodal training and evaluation remain deferred.
Use Python 3.11, uv for environment/dependency management, scikit-learn for TF-IDF and logistic regression, and PyTorch for the UI encoder and transformer. The one-hour prototype runs on CPU; no GPU is required.
-
Install uv using the official installation instructions.
-
From the repository root, create the environment and install the locked dependencies:
uv sync --locked
uv manages .venv/ and selects Python 3.11 using .python-version. Commit pyproject.toml, uv.lock, and .python-version; keep the environment and raw dataset/ directory out of Git. See the uv project guide for environment and lockfile behavior.
Run the text baseline and automated checks through the shared environment:
uv run python baseline.py --data examples/baseline_sample.jsonl
uv run python -m unittest discover -s tests -vThe baseline uses a stratified 75/25 split with seed 42 and fits TF-IDF only on
training text. It prints held-out texts, true labels, and predictions. The eight
manual transcriptions are tracked in examples/baseline_sample.jsonl; their
screenshot IDs and inherited labels are documented in the data report. The tiny
sample checks execution rather than model performance.
The Yada et al. corpus provides labeled e-commerce text. Download its pinned TSV and regenerate the separate 12-row sample, then evaluate the full corpus:
uv run python scripts/prepare_real_dataset_sample.py
uv run python baseline.py --data examples/ec_darkpattern_sample.jsonl
uv run python baseline.py --ec-darkpattern dataset/ec-darkpattern/dataset.tsv \
--output dataset/ec-darkpattern/baseline_result.jsonThe full run uses all 2,356 source rows before removing 98 normalized-text duplicates. Pages sharing duplicate text stay together. A fixed grouped split (seed 42) yields 1,693 training and 565 held-out texts, with TF-IDF fitted only on training text. The JSON records dataset checksum, split IDs, leakage checks, metrics, a majority-class comparison, and held-out probabilities. See the results and limitations.
First export the screenshot's UI elements with OmniParser, following the isolated environment and weight setup:
dataset/.venv-omniparser/bin/python parse_ui.py \
--image dataset/web/images/knockaround.com.png --sample-id web:6 \
--output-dir dataset/parsed/web-6The wrapper saves elements.json with normalized [x1,y1,x2,y2] boxes,
native.json with OCR/caption provenance, and annotated.png. The report includes
a screenshot download command and observed parsing errors. OmniParser runs in a
separate Python 3.12 environment because its inference dependencies use
Transformers 4; the shared CLIP/MiniLM environment uses Transformers 5.
Run the screenshot and its exported elements through fusion:
uv run python fusion_model.py \
--image dataset/web/images/knockaround.com.png \
--elements dataset/parsed/web-6/elements.jsonThe first run downloads pinned CLIP and MiniLM checkpoints into the Hugging Face
cache. The demonstration checks finite logits and backward gradients through the
UI projection and fusion model, while both pretrained encoders remain frozen.
It uses a demonstration target; the adapter projection and classifier are untrained.
The real web:6 run produced [1,59,256] UI tokens and [1,1] logits and passed
all forward/backward checks.
ui_encoder.py loads the parser's {sample_id, elements} JSON
and preserves element order, text, and normalized [x1,y1,x2,y2] boxes.
MiniLM-L6-v2
provides a 384-dimensional text embedding using attention-masked mean pooling and
L2 normalization. Each text is truncated to 256 wordpieces. A trainable linear
projection maps the text embedding plus four box coordinates (388 → 256) to a
UI token. OCR text and generated descriptions are embedded as exported;
native.json retains their provenance. No element-count truncation is applied.
load_ui_adapter() returns an adapter whose forward method accepts a list of
records from load_ui_record(path) or parse_ui.export_elements(...):
ui_tokens, ui_padding_mask = adapter(records)
logits = model(ui_tokens, image_features, ui_padding_mask)Tokens have shape [B,N,256], where N is the largest element count in the batch.
The boolean mask has shape [B,N]; True marks padding. Fusion prepends an
unmasked image token, so empty element lists remain valid. An entirely empty-UI
batch uses [B,0,256] tokens. MiniLM remains frozen and in evaluation mode when
the adapter is switched to training; train the adapter's projection alongside
the fusion model.
All 18 automated tests run without model downloads and cover sample integration, grouped splitting, duplicate handling, metrics, pooling, geometry, variable-length batches, empty UI, padding invariance, and gradient isolation.
Compare text-only, vision-only, UI-structure, and multimodal approaches to determine which signals help identify dark patterns. The intended output is a set of pattern predictions with scores and supporting UI elements or regions.
The onboarding document also proposes a smaller click-to-cancel MVP: parse an interface with OmniParser and use a model to assess cancellation-related UI. Verifying that cancellation actually succeeds would require interaction evidence beyond a single screenshot. The MVP and the broader research pipeline are separate possible implementation scopes.
| Candidate category | Example evidence | Interpretation limit |
|---|---|---|
| False urgency | Countdown timers or urgency language | A screenshot can reveal urgency cues, but cannot establish that the urgency is false. |
| Visual interference | A prominent Accept button and a low-visibility Reject button | Assess relative size, contrast, and placement in context. |
| Hidden costs | Fee disclosures in small or low-visibility text | Additional screens or pricing context may be needed to establish that a cost was hidden. |
| Confirmshaming | Emotionally manipulative refusal wording | Retain the exact text and its surrounding choice. |
| Sneaking / preselection | A paid optional add-on already selected | Selection state and pricing context are relevant evidence. |
Multiple categories may apply to one screenshot. An insufficient evidence outcome should be distinct from a negative prediction. The model-design diagram uses related labels such as urgency/scarcity cues and low-visibility disclosures; the final taxonomy must reconcile these observable cues with the broader categories above.
The design combines parsed UI elements with a full-image visual branch that preserves information the parser may miss.
flowchart TD
A[UI screenshot] --> B[OmniParser and OCR]
A --> C[Full-image visual encoder]
B --> D[Element crops, text, roles, and bounding boxes]
D --> E[Visual, text, and structural encoders]
E --> F[Element-level multimodal fusion]
F --> G[Relational transformer]
C --> G
G --> H[Multi-label classification]
G --> I[Evidence localization]
| Component | Responsibility | Candidates in the onboarding document |
|---|---|---|
| Input processing | Load screenshots and optional HTML | Pillow, BeautifulSoup |
| UI parsing | Extract text, UI regions, bounding boxes, and element roles | OmniParser V2, AidUI UIED, OCR |
| Visual encoding | Represent the screenshot and individual element crops | CLIP / ViT, ResNet |
| Text encoding | Represent UI wording | MiniLM, DeBERTa-v3-small, BERT, RoBERTa |
| Structural encoding | Represent geometry, relative prominence, contrast, and roles | OpenCV, bounding-box features, MLP |
| Fusion | Combine visual, text, and structural representations | Concatenation with an MLP; relational or cross-modal attention |
| Prediction | Produce pattern scores and supporting regions | Multi-label sigmoid classifier, element-localization head |
| Results interface | Present predictions and evidence | React with FastAPI, if a demo is included in a later scope |
For element i, the proposed fusion is z_i = MLP([v_i; t_i; s_i]), where v_i, t_i, and s_i are its visual, textual, and structural embeddings. The model-design diagrams propose a shared 256-dimensional representation and a relational transformer with two layers and eight attention heads.
The source contains two related transformer designs: one combines element tokens with full-image patch tokens; the other uses an Element-Aware Multimodal Transformer (EAMT) with a global screenshot token and a maximum of 64 UI elements. These are design candidates, not a finalized implementation. The choice of global representation, element-selection policy, and localization supervision remains open.
ContextDP is the recommended starting point for a screenshot-based experiment. It includes web and mobile screenshots with dark-pattern categories and bounding-box annotations, making it suitable for comparing text-only, visual, and UI-element approaches. The AidUI paper describes its construction and evaluation.
- Access: Download the dataset archive. The images and annotations are also available in the AidUI repository under
evaluation/evaluation_dataset/, separated intowebandmobiledirectories. Each has animages/directory and aresult.jsonannotation file. - Size: The authors report 162 web and 339 mobile screenshots, depicting 301 dark-pattern and 243 non-dark-pattern instances. These are instance counts, not counts of positive and negative screenshots; a screenshot can contain multiple instances.
- Annotations: The JSON files contain image metadata, category IDs, and instance bounding boxes. Categories differ between the web and mobile subsets; both include
NO DP. Review how instance annotations should become screenshot-level labels before training. - Text: The inspected annotation files do not include transcribed UI text. Extract it from screenshots with OCR or OmniParser for the text baseline and review extraction quality.
- Scope and limitations: ContextDP is small and consists of static screenshots. It cannot establish behavior across a user journey, such as whether cancellation succeeds. Check category frequencies and source identities before creating splits; group related screenshots by website or app where that information is available. Confirm dataset reuse terms before redistribution or publication; the repository's code license does not by itself establish rights to every screenshot.
The source README reports 544 instances, while the two annotation files contain 546 annotation records in total. Resolve this discrepancy before dataset-wide evaluation. This milestone verified NO DP-only labels for four negative web examples, transcribed eight screenshots, and processed one screenshot through OmniParser and CLIP. A full dataset audit remains pending.
Inspect ContextDP's annotations and source identities for the first experiment. Consider Dark Patterns at Scale and custom labeled screenshots for later expansion after checking their modalities and annotations.
Define a shared representation for screenshot identity, source domain, extracted text, UI elements, bounding boxes, and category labels. Record which samples have localization annotations and which require additional context. Reconcile label differences before combining datasets.
Create train, validation, and test splits separated by website or app source to reduce leakage. Keep duplicate or closely related screenshots together. Use the validation split for model selection and reserve the test split for final comparisons.
| Experiment | Input | Research question |
|---|---|---|
| NLP-only | Extracted UI text | How much can wording explain? |
| CV-only | Raw screenshot | How much can appearance explain? |
| UI structure | Bounding boxes and geometric features | How much can layout explain? |
| Pretrained VLM | Screenshot with text | How well does pretrained multimodal reasoning perform? |
| Learned multimodal model | Visual, textual, and structural features | Does learned fusion improve detection and localization? |
Use consistent labels and data splits across comparisons. Record differences in available input, pretraining, and supervision so results can be interpreted fairly.
Compare simple MLP fusion with the proposed relational transformer. Evaluate macro-F1, precision, recall, AUROC, and per-category F1. Add a localization metric when bounding-box or element-level annotations support it; choose the metric to match those annotations.
Run ablations that remove the visual, textual, and structural branches individually. Compare the contribution of full-image features with parsed element features, and inspect errors caused by OCR or UI parsing.
Report performance by category, ablation results, and representative errors with supporting screenshots or UI regions. Document dataset versions, splits, preprocessing, model configurations, random seeds, and decision thresholds alongside each experiment. Report insufficient-evidence outcomes separately so they are not silently counted as clean interfaces.
- Choose the next deliverable after this prototype: cancellation-focused MVP, a baseline experiment, or the broader research pipeline.
- Confirm ContextDP's suitability after inspecting its labels, negative examples, source identities, and available reuse terms; reconcile its categories with the project taxonomy.
- Confirm available compute and whether models run locally or use a hosted API.
- Select a fusion architecture and the supervision available for localization.
- Decide whether the deliverable needs a CLI, notebooks, an API, or an interactive demo.
The prototype uses ContextDP screenshots, Microsoft OmniParser, and OpenAI CLIP.
External source checkouts, screenshots, and generated parsing outputs stay under
the ignored dataset/ directory; CLIP weights use the Hugging Face cache.