Skip to content

eval.md

Latest commit

 

History

History
142 lines (97 loc) · 7.1 KB

eval.md

File metadata and controls

142 lines (97 loc) · 7.1 KB
Raw

Evaluation

We provide a unified evaluation script that runs baselines on multiple benchmarks. It takes a baseline model and evaluation configurations, evaluates on-the-fly, and reports results instantly in a JSON file.

Benchmarks

Donwload the processed datasets from Huggingface Datasets and put them in the data/eval directory, using huggingface-cli:

mkdir -p data/eval
huggingface-cli download Ruicheng/monocular-geometry-evaluation --repo-type dataset --local-dir data/eval --local-dir-use-symlinks False

Then unzip the downloaded files:

cd data/eval  
unzip '*.zip'
# rm *.zip # if you don't keep the zip files

Configuration

See configs/eval/moge3.json for an example of evaluation configurations on all benchmarks. You can modify this file to evaluate on different benchmarks or different baselines.

Each entry maps a benchmark name to a config object. Supported keys:

Key Default Description
path required Root directory of the processed dataset.
width, height required Target evaluation resolution.
split .index.txt Index file listing sample directories, relative to path.
depth_unit null Scale factor applied to GT depth. Setting this marks the benchmark as metric; leaving it out disables all *_metric metric groups.
depth depth.png Depth map filename within each sample directory.
has_sharp_boundary false Enables the boundary F1 metrics.
segmentation null Segmentation map filename. Required by the points_local_moge2 group.
normal null Normal map filename.
local_mask null Binary mask filename marking the local detail region. Required by the local metric groups.
local_segmentation null SAM segment-id map filename. Intersected with local_mask to recover per-segment regions. Only read when local_mask is also set.
drop_max_depth 1000. Drop depth beyond this multiple of the 1% depth quantile.
max_segments, min_seg_area 100, 1000 Segment filtering for points_local_moge2.
subset null Take every N-th sample. Useful for quick smoke runs.

Baseline

Some examples of baselines are provided in baselines/. Pass the path to the baseline model python code to the --baseline argument of the evaluation script.

Metric groups

Use --mg to choose which metrics to compute, which is significantly faster when you only care about a few metrics. --mg takes a comma-separated list of suites, categories, or concrete group names.

There are two named suites:

Suite Contents
moge3 Default setting. The metric set reported by MoGe-3: global + metric + local + boundary_f1_r1
moge2 The metric set reported by MoGe-2: global + metric + points_local_moge2 + boundary_f1_r123

The suites are built from these categories, which can also be requested directly:

Category Expands to
global depth_affine_invariant, depth_scale_invariant, disparity_affine_invariant, points_affine_invariant, points_scale_invariant, fov_x
metric depth_metric, points_metric for metric benchmarks
local depth_local, points_local

Run Evaluation

Run the script moge/scripts/eval_baseline.py. For example,

# Evaluate MoGe-3 on the 10 benchmarks with 3 refine steps
python moge/scripts/eval_baseline.py --baseline baselines/moge.py --config configs/eval/moge3.json --output eval_output/moge.json --pretrained PATH_TO_CKPT.pt --resolution_level 9 --version v3 --refine_steps 3

# Same as the first one, but spread over 4 GPUs (see "Multi-GPU Evaluation" below)
python moge/scripts/eval_baseline.py --baseline baselines/moge.py --config configs/eval/moge3.json --output eval_output/moge.json --ngpu 4 --pretrained PATH_TO_CKPT.pt --resolution_level 9 --version v3 --refine_steps 3

# Evaluate MoGe on the 10 benchmarks
python moge/scripts/eval_baseline.py --baseline baselines/moge.py --config configs/eval/moge2.json --output eval_output/moge.json --pretrained Ruicheng/moge-vitl --resolution_level 9

# Evaluate Depth Anything V2 on the 10 benchmarks. (NOTE: affine disparity)
python moge/scripts/eval_baseline.py --baseline baselines/da_v2.py --config configs/eval/moge2.json --output eval_output/da_v2.json

# Only global metrics, skipping the expensive local and boundary ones
python moge/scripts/eval_baseline.py --baseline baselines/moge.py --config configs/eval/moge3.json --output eval_output/moge.json --mg global --pretrained PATH_TO_CKPT.pt --version v3

# The metric set of MoGe-2, for comparison against older results
python moge/scripts/eval_baseline.py --baseline baselines/moge.py --config configs/eval/moge3.json --output eval_output/moge.json --mg moge2 --pretrained PATH_TO_CKPT.pt --version v3

The --baseline --input --output arguments are for the inference script. The rest arguments, e.g. --pretrained --resolution_level, are custormized for loading the baseline model.

Details of the arguments:

Usage: eval_baseline.py [OPTIONS]

  Evaluation script.

Options:
  --baseline PATH       Path to the baseline model python code.  [required]
  --config PATH         Path to the evaluation configurations. Defaults to
                        "configs/eval/all_benchmarks.json".
  -o, --output PATH     Path to the output json file.  [required]
  --ngpu INTEGER RANGE  Number of GPUs to use. Whole benchmarks of the config
                        are distributed over one worker process per GPU, each
                        claiming the next benchmark as it goes. Defaults to 1,
                        i.e. a single in-process run.  [x>=1]
  --oracle              Use oracle mode for evaluation, i.e., use the GT
                        intrinsics input.
  --mg TEXT             Comma-separated metric groups to compute.
  --dump_pred           Dump predition results.
  --dump_gt             Dump ground truth.
  --help                Show this message and exit.

Multi-GPU Evaluation

--ngpu N spawns one worker process per GPU and hands each worker whole benchmarks of the config. Whoever finishes first takes the next unclaimed benchmark, so the GPUs stay busy despite the benchmarks being very unevenly sized.

python moge/scripts/eval_baseline.py --baseline baselines/moge.py --config configs/eval/moge3.json --output eval_output/moge.json --ngpu 4 --pretrained PATH_TO_CKPT.pt --version v3

Wrap a Customized Baseline

Wrap any baseline method with moge.test.baseline.MGEBaselineInterface. See baselines/ for more examples.

It is a good idea to check the correctness of the baseline implementation by running inference on a small set of images via moge/scripts/infer_baselines.py:

python moge/scripts/infer_baselines.py --baseline baselines/moge.py --input example_images/ --output infer_outupt/moge --pretrained Ruicheng/moge-vitl --maps --ply