AndroidLife

Real-phone Android agent benchmark: everyday tasks on a live device, scoring task success and phone cost (battery, thermals, $$, steps). Agents are driven by MobileRun / Droidrun over ADB — pinned to mobilerun==0.6.15 (see pyproject.toml + uv.lock) for reproducible harness behavior.

Built for open-weight models — OpenRouter / any OpenAI-compatible API today, with on-device SLMs as the long-term focus. Same harness either way.


Leaderboard / site	androidlife-website.vercel.app
Code	github.com/YuvrajSingh-mist/AndroidLife
Public run artifacts	`YuvrajSingh9886/androidlife-public`
530-task corpus	`YuvrajSingh9886/androidlife-530`

What you get

60-task public benchmark — fully executed on a OnePlus CPH2423. Each task ships trajectory, screenshots/GIF, telemetry, and a manual-audit verdict (ground truth for the leaderboard).
Sources: benchmarks/androidlife-530/AndroidLife_public_v2.json, public.md, public_vars.local.env, multiturn_kb_public.json.
530-task dataset — 216 easy / 242 medium / 72 hard on a fixed 28-day schedule across 31 apps. The schedule is dataset shape (≈10 apps/day), not wall-clock run length. Public 60 is sampled from this set.
Sources: AndroidLife_530_v1.json, tasks_530.md. Spec: docs/benchmark-spec.md.

TEXT (default) = accessibility tree only. VISION = add --vision (a11y + screenshot each step). Report them as separate rows.

Requirements

macOS or Linux host with Python 3.11–3.13 and uv
adb (Android platform-tools)
A dedicated Android phone with USB or wireless debugging (reference device: OnePlus CPH2423)
MobileRun / Droidrun 0.6.15 — exact pin (mobilerun==0.6.15); do not float to a newer wheel unless you intentionally re-lock and re-validate
API keys as needed:
- OPENROUTER_API_KEY — agent via OpenRouter
- OPENAI_API_KEY — simulated ask_user (ASK USER / multi-turn KB tasks)
- omit OpenRouter key if you point --llm-upstream-base at a local OpenAI-compatible server (e.g. llama.cpp)

Tested hosts

Harness install + public runs have been exercised on:

Host	Notes
MacBook Air (M1, 2020)	Apple Silicon, local + cloud LLM
Mac mini (M4, 2025, 16 GB)	Apple Silicon, local llama.cpp (e.g. Qwen3.5-4B) + cloud LLM

Paths below are repo-relative — run every command from the cloned AndroidLife/ root. The only machine-specific value is your ADB serial ($S).

Setup

Everything is uv + pyproject.toml + uv.lock — no pip install -r, no Poetry. Run commands from the repo root so console scripts and relative paths resolve.

git clone https://github.com/YuvrajSingh-mist/AndroidLife.git
cd AndroidLife   # ← stay here for all later commands

# creates .venv from uv.lock (pins MobileRun 0.6.15 + transitive deps)
uv sync --extra dev --extra tracing --extra hf   # or: make sync
uv run python scripts/setup.py                   # or: make setup  (scaffolds .env / config)

cp -n .env.example .env                          # skip if .env already exists
# edit .env → OPENROUTER_API_KEY / OPENAI_API_KEY as needed

# confirm the harness pin from the lockfile env
uv run python -c "import importlib.metadata as m; print(m.version('mobilerun'))"
# → 0.6.15

Preferred entrypoints (defined in pyproject.toml [project.scripts]):

uv run androidlife-tasks --help
uv run androidlife-runner --help
# thin wrappers also work:  uv run androidlife_tasks.py …

Sanity checks:

make test-fast          # unit smoke
make app-audit          # phone has the expected apps
make smoke-test         # LLM + ADB + one real task (optional)

make test-fast never touches the handset. Tests that drive it — launching or force-stopping apps, or anything that reaches reset_app_state — are skipped unless you opt in:

ANDROIDLIFE_DEVICE_TESTS=1 make test-fast   # only when no benchmark batch is running

That default exists because the phone is exclusive to the run: a stray am force-stop / input keyevent HOME steals the foreground from the task in flight, corrupting the benchmark and looking like a flaky test rather than the interference it is.

Connect the phone

adb devices -l
# USB: enable debugging, plug in, accept the RSA prompt
# Wireless: Developer options → Wireless debugging → Pair / Connect
#   adb pair <phone-ip>:<pairing-port>     # 6-digit code (pairing port)
#   adb connect <phone-ip>:<connect-port>  # different port from the IP & port line

# pick the first online device (works on any machine)
export S="$(adb devices | awk '/\tdevice$/{print $1; exit}')"
test -n "$S" || { echo "No adb device in 'device' state"; exit 1; }
echo "Using serial: $S"
adb -s "$S" shell echo OK

Keep $S exported in that shell (or paste the serial into --serial flags).

Run the public 60 (operator path)

Full reset/seed runbook: docs/device-reset-and-seed.md.
GUI-only seeds (Chrome history, Photos captions, Gmail, …): docs/pre-run-checklist.md.

1. Reset + seed

uv run python scripts/seeding/reset_phone.py --serial "$S" --profile public_v2 --apply

uv run python scripts/seeding/seed_data.py --serial "$S" --day 1
uv run python scripts/seeding/seed_data.py --serial "$S" --day 2
uv run python scripts/seeding/seed_data.py --serial "$S" --day 3
uv run python scripts/seeding/enrich_public_notes.py --serial "$S"
uv run python scripts/seeding/fabricate_public_pdfs.py --serial "$S"

# re-seed tomorrow afternoon conflicts (Team Sync / Mentor 1 on 1) — see device-reset-and-seed.md

uv run python scripts/seeding/reset_phone.py --serial "$S" --profile public_v2 --verify-only
for d in 1 2 3; do
  uv run python scripts/seeding/verify_day1_seeds.py --serial "$S" --day "$d"
done

Do not start a scored run if verify fails. Finish the pre-run checklist in the UI next.

2. Launch the batch

OpenRouter (cloud) example — Phoenix optional for tracing:

RUN_TS=$(date +%Y%m%d-%H%M%S)
RUN_ROOT="assets/runs/public/$RUN_TS"

# optional tracing
# uv run python scripts/run/start_phoenix.py --public --run-ts "$RUN_TS"

uv run androidlife-tasks \
  --dataset benchmarks/androidlife-530/AndroidLife_public_v2.json \
  --source public.md --all \
  --serial "$S" \
  --llm-upstream-base https://openrouter.ai/api \
  --model qwen/qwen3.6-plus \
  --ask-user-model gpt-5.4-mini \
  --temperature 0.0 --steps 60 --task-timeout 2400 \
  --save-trajectory action \
  --vars-file benchmarks/androidlife-530/public_vars.local.env \
  --ask-user-kb benchmarks/androidlife-530/multiturn_kb_public.json \
  --run-root "$RUN_ROOT"
  # add --vision for VISION mode
  # add --phoenix-url http://localhost:6006 --phoenix-project androidlife-public if Phoenix is up
  # add --no-tracing to skip tracing entirely

Local llama.cpp (OpenAI-compatible server on :8088):

Model-agnostic launcher + flag table: scripts/llm/README.md.

export PATH="$HOME/local/bin:/opt/homebrew/bin:$PATH"
# any preset: qwen3.5-4b | gemma4-e2b | mai-ui-2b | gui-owl-1.5-2b
# or: bash scripts/llm/serve_gguf.sh /path/to/model.gguf --alias MyModel
bash scripts/llm/serve_gguf.sh qwen3.5-4b
# shared Metal args: -ngl 99 -fa on -c 65536 -b 2048 -ub 512 -ctk/-ctv q8_0 --jinja -np 1
# Qwen also gets --reasoning off; vision presets auto-attach --mmproj

uv run androidlife_tasks.py \
  --dataset benchmarks/androidlife-530/AndroidLife_public_v2.json \
  --source public.md --all \
  --serial "$S" \
  --llm-upstream-base http://127.0.0.1:8088 \
  --model Qwen3.5-4B \
  --ask-user-model gpt-5.4-mini \
  --temperature 0.0 --steps 60 --task-timeout 2400 \
  --save-trajectory action --no-tracing \
  --vars-file benchmarks/androidlife-530/public_vars.local.env \
  --ask-user-kb benchmarks/androidlife-530/multiturn_kb_public.json \
  --run-root "assets/runs/public/$(date +%Y%m%d-%H%M%S)"
# --model must match the server -a alias; add --vision for mmproj models

Public 60-task batch (detached)

Detach so the run survives terminal close (from repo root). Prefer androidlife-tasks; androidlife_tasks.py is an equivalent alias.

RUN_TS=$(date +%Y%m%d-%H%M%S)
nohup uv run androidlife_tasks.py \
  --dataset benchmarks/androidlife-530/AndroidLife_public_v2.json \
  --source public.md --all --serial "$S" \
  --llm-upstream-base https://openrouter.ai/api --model <model> \
  --ask-user-model gpt-5.4-mini --temperature 0.0 \
  --steps 60 --task-timeout 2400 \
  --save-trajectory action \
  --vars-file benchmarks/androidlife-530/public_vars.local.env \
  --ask-user-kb benchmarks/androidlife-530/multiturn_kb_public.json \
  --run-root "assets/runs/public/$RUN_TS" \
  < /dev/null > "assets/runs/public/batch-$RUN_TS.log" 2>&1 &
tail -f "assets/runs/public/batch-$RUN_TS.log"

# optional: add --vision for screenshot-driven runs
# resume: same --run-root + --resume-from <task_id>

Resume an interrupted batch with the same --run-root and --resume-from <task_id> (or an explicit remaining --task-id list).
Flags: docs/cli-reference.md. (uv run dailybench-tasks / androidlife-tasks remain supported aliases — docs/naming.md.)

3. Score + file artifacts

RUN_ROOT=assets/runs/public/<RUN_TS>

uv run scripts/eval/androidlife_report.py --runs "$RUN_ROOT" --source public.md \
  --hallucination-judge-model gpt-5.4-mini \
  --out "reports/metrics/public/public-<RUN_TS>-report.json" \
  --out-md "reports/metrics/public/public-<RUN_TS>-report.md"

uv run scripts/eval/eval_hallucination_controls.py --runs "$RUN_ROOT" --sub public \
  --model gpt-5.4-mini \
  --out "reports/metrics/hallucination/public-<RUN_TS>.json" \
  --out-md "reports/metrics/hallucination/public-<RUN_TS>.md"

make organize-public

Then write the narrative manual audit under reports/public/public-<RUN_TS>.md. Manual audit is the leaderboard ground truth — official self-report tables are inflated.

530-day schedule (optional)

uv run python scripts/run/start_phoenix.py --day 3
uv run androidlife-tasks --serial "$S" \
  --llm-upstream-base https://openrouter.ai/api --model "$MODEL" \
  --day 3 --vars-file benchmarks/androidlife-530/tasks_vars/day_3.env

Repository layout

Path	Role
`androidlife-tasks` / `androidlife-runner`	Console scripts from `pyproject.toml` (`dailybench-*` aliases)
`src/androidlife/`	Harness
`benchmarks/androidlife-530/`	Public 60 + 530 datasets, vars, KB
`scripts/seeding/`	Reset / seed / verify
`scripts/eval/`	Reports, HC judge, KB audit
`scripts/llm/`	Local GGUF download / serve helpers
`assets/`	Runs, seeds, DBs (gitignored)
`reports/`	Metrics + narrative audits
`website/`	GitHub Pages redirect → Vercel
`androidlife-website/`	Local clone of private site repo (gitignored)

Docs

docs/getting-started.md — mental model + artifact map
docs/device-reset-and-seed.md — reset + seed commands
docs/pre-run-checklist.md — UI/cloud seeds ADB cannot plant
docs/cli-reference.md — flags, detach / resume
docs/benchmark-spec-public.md — public 60
docs/benchmark-spec.md — 530 corpus
docs/evaluation-policy.md — grading
docs/reproducibility.md — what “reproducible” means here

Website

Canonical: https://androidlife-website.vercel.app/
(GitHub Pages redirects: https://yuvrajsingh-mist.github.io/AndroidLife/)

Site source is the private repo androidlife-website. Keep a local checkout at androidlife-website/ (gitignored) and push from there.

Citation

If you use AndroidLife — the benchmark, leaderboard, tasks, or results — please credit this work and cite it as:

@misc{singh2026androidlife,
      title={AndroidLife: Real-Phone Android Agent Benchmark for Open-Weight Models and On-Device SLMs},
      author={Yuvraj Singh},
      year={2026},
      howpublished={\url{https://github.com/YuvrajSingh-mist/AndroidLife}},
}

Also see CITATION.cff.

License

Code / harness (src/, scripts, tooling): Apache License 2.0 — keep the copyright notice and NOTICE when you redistribute.
Benchmark datasets & task content (task definitions, schedules, published evaluation artifacts): CC BY 4.0 — free to use and adapt, including commercially, with attribution to Yuvraj Singh (name + link; indicate changes if you modify).

Academic paper citation is a community norm (use the BibTeX above); CC BY is what legally requires credit when the dataset or task content is shared or adapted.

Fuel the benches

Perf benchmarking burns wall-clock, watts, and a lot of coffee. If these numbers helped you pick a board or a model, fuel the next run:

Navigation Menu

Uh oh!

Repository files navigation

AndroidLife

What you get

Requirements

Tested hosts

Setup

Connect the phone

Run the public 60 (operator path)

1. Reset + seed

2. Launch the batch

Public 60-task batch (detached)

3. Score + file artifacts

530-day schedule (optional)

Repository layout

Docs

Website

Citation

License

Fuel the benches

About

Releases

Sponsor this project

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.agents/skills		.agents/skills
.cursor/rules		.cursor/rules
.github		.github
.vscode		.vscode
benchmarks/androidlife-530		benchmarks/androidlife-530
config		config
docs		docs
reports		reports
scripts		scripts
src/androidlife		src/androidlife
tests		tests
website		website
.env.example		.env.example
.gitignore		.gitignore
.python-version		.python-version
CITATION.cff		CITATION.cff
LICENSE		LICENSE
LICENSE-DATASET		LICENSE-DATASET
Makefile		Makefile
NOTICE		NOTICE
README.md		README.md
androidlife_runner.py		androidlife_runner.py
androidlife_tasks.py		androidlife_tasks.py
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

AndroidLife

What you get

Requirements

Tested hosts

Setup

Connect the phone

Run the public 60 (operator path)

1. Reset + seed

2. Launch the batch

Public 60-task batch (detached)

3. Score + file artifacts

530-day schedule (optional)

Repository layout

Docs

Website

Citation

License

Fuel the benches

About

Topics

Resources

Stars

Watchers

Forks

Releases

Sponsor this project

Packages

Contributors

Languages