Skip to content

About

Research on detecting dark patterns in UI screenshots using visual, textual, and structural evidence.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Dark Pattern Detector

A research project for identifying potentially manipulative interface patterns in website and app screenshots using visual, textual, and structural evidence.

Based on Misleading Patterns — CDS Onboarding, including its model-design diagrams.

Status and scope

All four implementation-plan tasks and UI embedding integration completed; the full text baseline is evaluated. The prototype includes eight real ContextDP text examples, a TF-IDF/logistic-regression baseline, a working OmniParser screenshot wrapper, and CLIP screenshot encoding with fusion forward/backward checks. The merged data branch adds 12 labeled e-commerce text examples and their preparation script. Training on the full e-commerce corpus produced 94.16% accuracy and 0.9414 macro-F1 on a page-grouped holdout of 565 texts. The UI adapter encodes real parsed text and boxes with frozen MiniLM and a trainable projection, including padding masks. See the implementation plan and the data, baseline, UI parsing, and visual encoder reports. Multimodal training and evaluation remain deferred.

Development setup

Use Python 3.11, uv for environment/dependency management, scikit-learn for TF-IDF and logistic regression, and PyTorch for the UI encoder and transformer. The one-hour prototype runs on CPU; no GPU is required.

  1. Install uv using the official installation instructions.

  2. From the repository root, create the environment and install the locked dependencies:

    uv sync --locked

uv manages .venv/ and selects Python 3.11 using .python-version. Commit pyproject.toml, uv.lock, and .python-version; keep the environment and raw dataset/ directory out of Git. See the uv project guide for environment and lockfile behavior.

First milestone

Run the text baseline and automated checks through the shared environment:

uv run python baseline.py --data examples/baseline_sample.jsonl
uv run python -m unittest discover -s tests -v

The baseline uses a stratified 75/25 split with seed 42 and fits TF-IDF only on training text. It prints held-out texts, true labels, and predictions. The eight manual transcriptions are tracked in examples/baseline_sample.jsonl; their screenshot IDs and inherited labels are documented in the data report. The tiny sample checks execution rather than model performance.

Full text baseline

The Yada et al. corpus provides labeled e-commerce text. Download its pinned TSV and regenerate the separate 12-row sample, then evaluate the full corpus:

uv run python scripts/prepare_real_dataset_sample.py
uv run python baseline.py --data examples/ec_darkpattern_sample.jsonl
uv run python baseline.py --ec-darkpattern dataset/ec-darkpattern/dataset.tsv \
  --output dataset/ec-darkpattern/baseline_result.json

The full run uses all 2,356 source rows before removing 98 normalized-text duplicates. Pages sharing duplicate text stay together. A fixed grouped split (seed 42) yields 1,693 training and 565 held-out texts, with TF-IDF fitted only on training text. The JSON records dataset checksum, split IDs, leakage checks, metrics, a majority-class comparison, and held-out probabilities. See the results and limitations.

Screenshot and fusion demonstration

First export the screenshot's UI elements with OmniParser, following the isolated environment and weight setup:

dataset/.venv-omniparser/bin/python parse_ui.py \
  --image dataset/web/images/knockaround.com.png --sample-id web:6 \
  --output-dir dataset/parsed/web-6

The wrapper saves elements.json with normalized [x1,y1,x2,y2] boxes, native.json with OCR/caption provenance, and annotated.png. The report includes a screenshot download command and observed parsing errors. OmniParser runs in a separate Python 3.12 environment because its inference dependencies use Transformers 4; the shared CLIP/MiniLM environment uses Transformers 5.

Run the screenshot and its exported elements through fusion:

uv run python fusion_model.py \
  --image dataset/web/images/knockaround.com.png \
  --elements dataset/parsed/web-6/elements.json

The first run downloads pinned CLIP and MiniLM checkpoints into the Hugging Face cache. The demonstration checks finite logits and backward gradients through the UI projection and fusion model, while both pretrained encoders remain frozen. It uses a demonstration target; the adapter projection and classifier are untrained. The real web:6 run produced [1,59,256] UI tokens and [1,1] logits and passed all forward/backward checks.

UI embeddings and padding

ui_encoder.py loads the parser's {sample_id, elements} JSON and preserves element order, text, and normalized [x1,y1,x2,y2] boxes. MiniLM-L6-v2 provides a 384-dimensional text embedding using attention-masked mean pooling and L2 normalization. Each text is truncated to 256 wordpieces. A trainable linear projection maps the text embedding plus four box coordinates (388 → 256) to a UI token. OCR text and generated descriptions are embedded as exported; native.json retains their provenance. No element-count truncation is applied.

load_ui_adapter() returns an adapter whose forward method accepts a list of records from load_ui_record(path) or parse_ui.export_elements(...):

ui_tokens, ui_padding_mask = adapter(records)
logits = model(ui_tokens, image_features, ui_padding_mask)

Tokens have shape [B,N,256], where N is the largest element count in the batch. The boolean mask has shape [B,N]; True marks padding. Fusion prepends an unmasked image token, so empty element lists remain valid. An entirely empty-UI batch uses [B,0,256] tokens. MiniLM remains frozen and in evaluation mode when the adapter is switched to training; train the adapter's projection alongside the fusion model.

All 18 automated tests run without model downloads and cover sample integration, grouped splitting, duplicate handling, metrics, pooling, geometry, variable-length batches, empty UI, padding invariance, and gradient isolation.

Research objective

Compare text-only, vision-only, UI-structure, and multimodal approaches to determine which signals help identify dark patterns. The intended output is a set of pattern predictions with scores and supporting UI elements or regions.

The onboarding document also proposes a smaller click-to-cancel MVP: parse an interface with OmniParser and use a model to assess cancellation-related UI. Verifying that cancellation actually succeeds would require interaction evidence beyond a single screenshot. The MVP and the broader research pipeline are separate possible implementation scopes.

Detection categories

Candidate category Example evidence Interpretation limit
False urgency Countdown timers or urgency language A screenshot can reveal urgency cues, but cannot establish that the urgency is false.
Visual interference A prominent Accept button and a low-visibility Reject button Assess relative size, contrast, and placement in context.
Hidden costs Fee disclosures in small or low-visibility text Additional screens or pricing context may be needed to establish that a cost was hidden.
Confirmshaming Emotionally manipulative refusal wording Retain the exact text and its surrounding choice.
Sneaking / preselection A paid optional add-on already selected Selection state and pricing context are relevant evidence.

Multiple categories may apply to one screenshot. An insufficient evidence outcome should be distinct from a negative prediction. The model-design diagram uses related labels such as urgency/scarcity cues and low-visibility disclosures; the final taxonomy must reconcile these observable cues with the broader categories above.

Proposed architecture

The design combines parsed UI elements with a full-image visual branch that preserves information the parser may miss.

flowchart TD
    A[UI screenshot] --> B[OmniParser and OCR]
    A --> C[Full-image visual encoder]
    B --> D[Element crops, text, roles, and bounding boxes]
    D --> E[Visual, text, and structural encoders]
    E --> F[Element-level multimodal fusion]
    F --> G[Relational transformer]
    C --> G
    G --> H[Multi-label classification]
    G --> I[Evidence localization]
Loading
Component Responsibility Candidates in the onboarding document
Input processing Load screenshots and optional HTML Pillow, BeautifulSoup
UI parsing Extract text, UI regions, bounding boxes, and element roles OmniParser V2, AidUI UIED, OCR
Visual encoding Represent the screenshot and individual element crops CLIP / ViT, ResNet
Text encoding Represent UI wording MiniLM, DeBERTa-v3-small, BERT, RoBERTa
Structural encoding Represent geometry, relative prominence, contrast, and roles OpenCV, bounding-box features, MLP
Fusion Combine visual, text, and structural representations Concatenation with an MLP; relational or cross-modal attention
Prediction Produce pattern scores and supporting regions Multi-label sigmoid classifier, element-localization head
Results interface Present predictions and evidence React with FastAPI, if a demo is included in a later scope

For element i, the proposed fusion is z_i = MLP([v_i; t_i; s_i]), where v_i, t_i, and s_i are its visual, textual, and structural embeddings. The model-design diagrams propose a shared 256-dimensional representation and a relational transformer with two layers and eight attention heads.

The source contains two related transformer designs: one combines element tokens with full-image patch tokens; the other uses an Element-Aware Multimodal Transformer (EAMT) with a global screenshot token and a maximum of 64 UI elements. These are design candidates, not a finalized implementation. The choice of global representation, element-selection policy, and localization supervision remains open.

Research plan

Recommended starting dataset: ContextDP

ContextDP is the recommended starting point for a screenshot-based experiment. It includes web and mobile screenshots with dark-pattern categories and bounding-box annotations, making it suitable for comparing text-only, visual, and UI-element approaches. The AidUI paper describes its construction and evaluation.

  • Access: Download the dataset archive. The images and annotations are also available in the AidUI repository under evaluation/evaluation_dataset/, separated into web and mobile directories. Each has an images/ directory and a result.json annotation file.
  • Size: The authors report 162 web and 339 mobile screenshots, depicting 301 dark-pattern and 243 non-dark-pattern instances. These are instance counts, not counts of positive and negative screenshots; a screenshot can contain multiple instances.
  • Annotations: The JSON files contain image metadata, category IDs, and instance bounding boxes. Categories differ between the web and mobile subsets; both include NO DP. Review how instance annotations should become screenshot-level labels before training.
  • Text: The inspected annotation files do not include transcribed UI text. Extract it from screenshots with OCR or OmniParser for the text baseline and review extraction quality.
  • Scope and limitations: ContextDP is small and consists of static screenshots. It cannot establish behavior across a user journey, such as whether cancellation succeeds. Check category frequencies and source identities before creating splits; group related screenshots by website or app where that information is available. Confirm dataset reuse terms before redistribution or publication; the repository's code license does not by itself establish rights to every screenshot.

The source README reports 544 instances, while the two annotation files contain 546 annotation records in total. Resolve this discrepancy before dataset-wide evaluation. This milestone verified NO DP-only labels for four negative web examples, transcribed eight screenshots, and processed one screenshot through OmniParser and CLIP. A full dataset audit remains pending.

1. Prepare and standardize data

Inspect ContextDP's annotations and source identities for the first experiment. Consider Dark Patterns at Scale and custom labeled screenshots for later expansion after checking their modalities and annotations.

Define a shared representation for screenshot identity, source domain, extracted text, UI elements, bounding boxes, and category labels. Record which samples have localization annotations and which require additional context. Reconcile label differences before combining datasets.

2. Establish evaluation splits

Create train, validation, and test splits separated by website or app source to reduce leakage. Keep duplicate or closely related screenshots together. Use the validation split for model selection and reserve the test split for final comparisons.

3. Implement controlled baselines

Experiment Input Research question
NLP-only Extracted UI text How much can wording explain?
CV-only Raw screenshot How much can appearance explain?
UI structure Bounding boxes and geometric features How much can layout explain?
Pretrained VLM Screenshot with text How well does pretrained multimodal reasoning perform?
Learned multimodal model Visual, textual, and structural features Does learned fusion improve detection and localization?

Use consistent labels and data splits across comparisons. Record differences in available input, pretraining, and supervision so results can be interpreted fairly.

4. Train and evaluate fusion models

Compare simple MLP fusion with the proposed relational transformer. Evaluate macro-F1, precision, recall, AUROC, and per-category F1. Add a localization metric when bounding-box or element-level annotations support it; choose the metric to match those annotations.

Run ablations that remove the visual, textual, and structural branches individually. Compare the contribution of full-image features with parsed element features, and inspect errors caused by OCR or UI parsing.

5. Report findings

Report performance by category, ablation results, and representative errors with supporting screenshots or UI regions. Document dataset versions, splits, preprocessing, model configurations, random seeds, and decision thresholds alongside each experiment. Report insufficient-evidence outcomes separately so they are not silently counted as clean interfaces.

Decisions before broader experiments

  • Choose the next deliverable after this prototype: cancellation-focused MVP, a baseline experiment, or the broader research pipeline.
  • Confirm ContextDP's suitability after inspecting its labels, negative examples, source identities, and available reuse terms; reconcile its categories with the project taxonomy.
  • Confirm available compute and whether models run locally or use a hosted API.
  • Select a fusion architecture and the supervision available for localization.
  • Decide whether the deliverable needs a CLI, notebooks, an API, or an interactive demo.

References

The prototype uses ContextDP screenshots, Microsoft OmniParser, and OpenAI CLIP. External source checkouts, screenshots, and generated parsing outputs stay under the ignored dataset/ directory; CLIP weights use the Hugging Face cache.

About

Research on detecting dark patterns in UI screenshots using visual, textual, and structural evidence.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages