models.md

Model notes

What the tables in the README cannot say: why a family behaves the way it does, what it costs, and which of its behaviours are deliberate rather than incidental.

Every claim about parity here is recorded in more detail in the family's mozo/vendors/<family>_deploy/PROVENANCE.md, which also names the exact upstream commit the extraction came from.

RF-DETR

Real-time transformer detection and instance segmentation by Roboflow. NMS-free — the head predicts one box per query, so there is no overlap threshold to tune.

Every variant publishes torch, ONNX and CoreML, and they are verified to return the same detections:

model = mozo.get_model("rfdetr/small")                            # torch
model = mozo.get_model("rfdetr", "small", runtime="onnx-fp32")    # ONNX Runtime

Class names ship with the weights, so they are the vocabulary the checkpoint was trained on rather than an assumption. A checkpoint of your own that carries no names returns class_id with class_name unset — pass labels=[...] to name them. Mozo never guesses a name.

keypoint-preview returns person joints. One variant, because Roboflow published one operating point: 576×576, and 71.8 keypoint AP on COCO at 9.8 ms on a T4 under TensorRT FP16 by their measurement.

model = mozo.get_model("rfdetr/keypoint-preview")
found = model.predict("crowd.jpg", threshold=0.5)
found[0].keypoints[0]      # KeyPoint(id=0, name='nose', x=..., y=..., confidence=...)

Seventeen joints per detection in COCO's order, each (x, y, confidence) in source-image pixels, named from the published labels.json the same way class names are. A joint the model cannot see still occupies its slot — it comes back with a confidence near zero and coordinates that mean nothing, so filter on the confidence before reading a position. That is upstream's behaviour and it is reproduced rather than tidied: dropping the invisible joints would renumber the rest, and the index is the joint's identity.

Two things differ from its siblings. Its class-id space is its own — a two-slot head, background at 0 and person at 1 — where the detection variants emit COCO's sparse ids running to 90. And it publishes torch-fp32 only, so runtime="auto" gives you torch and asking for ONNX says so.

The head also predicts a per-joint precision-Cholesky, an uncertainty ellipse rather than a score. Mozo does not return it: there is nothing in a Detections that carries one, and reducing it to a single number the model never predicted would be worse than leaving it out.

YOLOv8, YOLO11, YOLO12, YOLO26

Real-time detection by Ultralytics. Five sizes each: nano small medium large xlarge. YOLO11 and YOLO26 add five instance-segmentation variants apiece, seg-nano … seg-xlarge.

YOLO26 is NMS-free. Its head fires once per object and the network returns a ranked detection list, so there is no overlap threshold to tune. The other three suppress in the usual way. Either way predict takes a confidence threshold and returns the same PixelFlow result.

YOLO11 and YOLO26 also segment. Five seg- variants each, sitting beside the detection ones in the same family under the same task type, because what they add is a field rather than a different question:

model = mozo.get_model("yolov11/seg-nano")
found = model.predict("desk.jpg", threshold=0.25)
found[0].masks          # a boolean array at the source image's resolution

A mask is a yes-or-no per pixel, one per detection, aligned with the box beside it. The seg- variants currently publish torch-fp32 only, so runtime="auto" gives you torch for them and can still give you a graph for their detection counterparts.

Two behaviours are upstream's and are reproduced rather than tidied away. A detection whose mask comes out empty is dropped, so a seg- variant can return fewer objects than its detection counterpart on the same photograph. And the mask crop takes a different branch below 50 detections, rounding box edges to whole pixels where the other compares against the unrounded float — so the mask you get depends slightly on how many objects are in the picture.

Runtimes differ across the four generations. Every variant of all four publishes ONNX, segmentation included — a seg- graph declares two outputs, predictions and the prototypes its mask coefficients belong to, where a detection graph declares one. YOLOv8 and YOLO12 also publish a CoreML artifact, which is by far the fastest way to run them on Apple silicon. YOLO11 and YOLO26 do not: the C2PSA block they share makes Apple's Metal graph compiler abort the process, and the configuration that avoids that is slower than torch on MPS. Their seg- variants are the same backbone with a different head, so they inherit that rather than escaping it. runtime="auto" handles all of this by itself — it only ever chooses among what a variant actually publishes, so nothing in mozo carries a per-family exception.

Class names come from the checkpoint, so a fine-tuned model publishes its own vocabulary.

Licensing. These weights are AGPL-3.0, or covered by a commercial licence from Ultralytics. Mozo's own code is Apache-2.0 — they are separate works travelling together — but anything you export from them inherits their terms, and serving predictions from them over a network places AGPL-3.0 section 13 obligations on you. Complying is the operator's responsibility.

OWLv2

Open-vocabulary detection. Name anything in words and it returns boxes for it — no class list, no fine-tuning, no vocabulary agreed in advance.

base-ensemble and large-ensemble average Google's self-trained and fine-tuned checkpoints and are what the paper reports; base and large are self-training only.

This is mozo's permissively-licensed way to ask a model a question in words. SAM 3 answers a similar question with masks, but its weights carry Meta's SAM License and bind whoever you serve predictions to. OWLv2 is Apache-2.0 on the code and on all four checkpoints.

Boxes only, no masks — pair it with SAM 2 or EdgeTAM if you want those: a box from here is a prompt there.

Phrases go in verbatim, up to 16 tokens, and all of them share one image forward and one text forward, so twenty phrases cost barely more than one. Scores are similarities through a sigmoid rather than class probabilities, so they run low: 0.3 is confident here, and the default floor is 0.1 rather than the 0.5 the closed-vocabulary detectors use.

There is no non-maximum suppression, because the published postprocessing has none. A detection is a patch: the model scores every patch against every phrase and predicts one box per patch, so overlapping boxes on a large object are expected, and suppressing them is the caller's policy rather than something mozo decides.

Inference is a fixed square — 960 for base, 1008 for large — reached by padding the image bottom and right to a square first, so the aspect ratio is preserved and a 4:3 photograph spends a quarter of its patches on padding. On CPU at 2 MP, base-ensemble is about 1.1 s, which is within 1% of transformers running the same weights; the image encode is cached, so a second vocabulary on the same photograph is about 50 ms.

Verified against transformers on every stage — tokenizer, preprocessing, both towers, all three heads, and the final boxes — with no tolerance: 226 comparisons on base-ensemble, every one bit-identical. tools/verify/owlv2.py needs neither a checkout nor a network, only pip install transformers and the weights.

Grounding DINO

Open-vocabulary detection, and mozo's second answer to the same question OWLv2 answers — same task, same endpoint, same response shape, so the two are substitutable.

Reach for it over OWLv2 when the prompt is a description. OWLv2 embeds each phrase independently and compares it to patch embeddings, which works well for nouns. Grounding DINO fuses the text into the image features six times over and lets the decoder attend back to the words, so "the mug on the left" is read as a phrase rather than a bag of words.

It costs more, and how much more depends entirely on whether you re-query one image. On the fixture photograph, MPS, three prompts:

	first look at an image	asking the same image again
owlv2/base-ensemble	458 ms	13 ms
grounding_dino/tiny	472 ms	472 ms
grounding_dino/base	656 ms	656 ms

Cold, they are comparable — 1.0x and 1.4x. Warm, OWLv2 is 40x faster, and that gap is architectural rather than an optimisation anyone could port. OWLv2's towers are independent, so its image encode depends only on the image and owlv2_deploy caches it; a second prompt against one photograph costs almost nothing. Grounding DINO fuses text into the image features, so its image encode depends on the prompt and there is nothing cacheable to keep. If you ask one image many questions, that is the whole comparison.

Two variants, and upstream publishes no others. tiny (Swin-T) is 48.4 box AP on COCO zero-shot; base (Swin-B) is 56.7 for 35% more weights. tiny is 82% of upstream's own release downloads, but 8.3 AP is a wide enough gap that base is not merely the completionist option.

Prompts have three rules. They may not contain . or ? — those separate concepts, and a prompt carrying one would be split in two and its detections reported against the wrong phrase, so it is refused. Case is not preserved: the caption is lowercased before tokenization, as upstream does. And about 60 prompts fit the model's 256-token budget; upstream truncates past it silently, mozo raises.

The name you get back is the phrase you asked for. Upstream decodes the matching tokens back into a string, which can return "yellow school" for "a yellow school bus". mozo instead reports which prompt matched — exactly, from the phrase map the tokenizer already builds — so class_name is your string and class_id is its index in the list you passed. This is the one place the package deliberately differs from upstream, and it is a difference in the string rather than in any number. PROVENANCE.md has the reasoning.

Nothing is suppressed, as with OWLv2. Two prompts describing the same thing can both find it.

Torch only, and that is structural. The image is resized to a short side of 800 with the long side capped near 1333 — aspect preserved, nothing padded — so unlike every other family here there is no fixed input size. A graph exported at one shape cannot take another, so no ONNX or CoreML artifact is published and the adapter declares EXECUTES = ("torch",) so auto cannot pick one.

No transformers at run time, despite the BERT text encoder. The published checkpoint carries its own fine-tuned BERT tower under bert.* — 200 tensors, most of the 694 MB — and upstream's download of bert-base-uncased is only ever used for its shape. mozo rebuilds that shape and ships the WordPiece vocabulary in the wheel, so tokenizing a prompt needs no network.

Verified against the authors' own implementation on every stage — tokenizer, preprocessing, the BERT tower, all three Swin levels, the encoder, all six decoder layers, and the final logits and boxes — with no tolerance: 138 comparisons across both variants, every one bit-identical. tools/verify/grounding_dino.py needs a checkout of upstream and transformers<5.

SAM 2 and EdgeTAM

Promptable segmentation: point at something and get back the thing you pointed at. sam2/{tiny,small,base_plus,large} and edgetam/edgetam.

Every prompt is a set of points. A click is a point with a label — 1 to include, 0 to exclude. A box is spelled as its two corners carrying reserved labels, because neither model has a separate box input; the adapter writes those for you. Points and a box can be combined.

model = mozo.get_model("edgetam/edgetam")

found = model.predict(image, points=[[820, 640]], labels=[1])   # three candidates, best first
found = model.predict(image, boxes=[40, 60, 300, 480])
found = model.predict(image, boxes=[40, 60, 300, 480], points=[[900, 700]], labels=[0])

multimask_output=True (the default) returns three candidate masks with the model's own predicted IoU as the score, ranked. That is the right setting for a single click, which is genuinely ambiguous about whether you meant the handle, the door or the car — take the first row, or show all three. With a box the prompt is usually unambiguous and multimask_output=False is tighter.

Detections come back with class_name=None. A click does not say what it clicked, and mozo will not invent a name for it — a name comes from the weights or from the user. Pass name="cat" if you know what you pointed at. class_id is the index of the prompt that produced the row, so a batch of prompts stays separable.

The image encoder is the cost and it depends only on the image, so it is cached on pixel content and a second prompt on the same photograph pays only for the decoder. On CPU at 2 MP: EdgeTAM encodes in 272 ms and decodes in 33 ms; SAM 2 tiny encodes in 439 ms.

EdgeTAM is SAM 2 distilled for phones — a 9.1M-parameter image path against SAM 2 tiny's 31.4M — and its masks agree with SAM 2 tiny's at 0.94 IoU on box prompts.

Unlike SAM 3, both families' published weights are Apache-2.0, the same as their code.

Only the torch runtime is served so far. SAM 2 also publishes ONNX and CoreML artifacts, which this adapter refuses rather than quietly answering with torch: a promptable model exports as several graphs and needs a runner that keeps the encode and the decode apart.

SAM 3

Meta ships a single model rather than a size ladder, so there is one variant: sam3/sam3.

Two ways to prompt it, off one checkpoint:

• Name a concept — predict(image, "taxi") returns every instance, with a mask, a box and a score. The phrase you searched for is the class name every detection carries, so there is no fixed vocabulary and nothing for mozo to guess. Pass a list — predict(image, ["car", "person", "dog"]) — and you get one result carrying several classes, with class_id indexing the prompts. Instances found by different prompts may overlap: ask for "car" and "vehicle" and the same car comes back under both names.
• Point at one thing — Segmenter.segment(image, points, labels) takes clicks (1 include, 0 exclude), a box, or a previous mask to refine, and returns three candidate masks with predicted IoU. Reached through mozo.vendors.sam3_deploy rather than the adapter for now.

Prompts are up to 32 tokens. Inference is a fixed 1008×1008 square — SAM 3 squashes rather than letterboxing, so aspect ratio is not preserved.

The model wants a GPU. On Apple silicon MPS the image encoder is about 1.2 s and on CPU about 5 s; the encode is cached, so further prompts on the same image cost only their own decode. Concepts do not batch — the head takes one prompt at a time — so N concepts cost one encode plus N decodes: on MPS, three concepts is about 2.1 s cold and 0.35 s each afterwards. Encoded prompts are cached too (33 KB each), so the same three words on the next image skip the text tower entirely.

Licensing — read this before deploying. These weights are not open source. They carry Meta's SAM License, which no other family here does. It restricts what they may be used for — military, nuclear, espionage and weapons uses are prohibited — and those restrictions flow through to whoever you serve predictions to. It binds on use rather than on signing, and it must travel with the weights if you pass them on. Mozo's own SAM 3 code is Apache-2.0 and derived from transformers, not from facebookresearch/sam3; the code and the weights are separate works travelling together. Complying is the operator's responsibility.

ViTPose

Human pose estimation. Give it a frame and the boxes of the people in it, and it hands those same detections back with COCO's 17 joints attached to each.

found = mozo.get_model("rfdetr", "medium").predict(frame)
posed = mozo.get_model("vitpose", "base").predict(frame, found.filter_by_class_id(1))

posed[0].keypoints[0].name        # 'nose'
posed[0].class_name               # 'person' -- still the detector's, not ours

It takes detections, and it is the only family that does

Every other model in mozo produces detections. This one is handed them. That is not an interface quirk, it is what top-down pose estimation is: the model answers a question about a person whose location it was told. Pair it with anything that produces boxes — RF-DETR, a YOLO, a tracker, a rectangle someone drew.

The result is a copy of what you passed in, with joints added. Same boxes, same class ids, same scores, same names, same tracker ids. Rebuilding them would have been easier and would have thrown all of that away.

Both arguments are required. There is no "no detections" shortcut that treats the whole image as one person — that is a guess about intent, and on a street scene it returns one confident skeleton stretched across the frame. An empty Detections is a different thing and is answered with an empty result: a frame with nobody in it is an answer, not an error.

It does not filter what you give it

Pass the box of a car and it returns seventeen confident joints on a car. It will not hedge — a heatmap model puts a nose somewhere and scores it well — so low confidence does not save you.

That is deliberate. Which boxes are people is a fact about your pipeline, not about this model, and a filter here would be mozo deciding it for you. filter_by_class_id already does the job, and note the id is the detector's: RF-DETR emits COCO's original ids where person is 1, while the YOLO families use a contiguous vocabulary where it is 0.

Give it the frame, not a crop

The crop this model wants is larger than the box it is given. Each box is first widened or heightened to the input's 3:4 aspect ratio, then padded by a further 1.25×. For a 50×140 person that is roughly a 131×175 crop — about forty pixels of width and thirty-five of height the detector's box never contained, taken from the surrounding frame.

So passing a pre-cropped person is not the same operation. Those pixels are already gone, the padding falls back to black, and a wrist just outside the box — which the real crop would have recovered — is unrecoverable. If a crop is all you have, pass it with a box covering its full extent and expect slightly worse joints near the edges.

Confidences are per joint, and worth reading

Each joint carries the peak of its heatmap channel. It is not a probability and not comparable across models, but it does track visibility usefully: on the fixture photograph, where five people sit behind a table, the visible faces and shoulders score 0.6–0.97 while the occluded knees sit near 0.4 and the ankles near 0.15. Filter on it before reading a coordinate — a joint the model cannot see comes back with a position that means nothing, because the argmax of a flat heatmap lands somewhere.

Every published variant is ViTPose++

Seven checkpoints exist upstream; mozo publishes the four that are ViTPose++, the mixture-of-experts revision, named by size. The other three are the original ViTPose, which ViTPose++ beats at every size — and the smallest of them is 344 MB against small's 133.

Every block picks one of six dataset experts. mozo always runs COCO's, because COCO's is the one the published heads match; the other five are not alternatives but ways to get a wrong answer, so there is no argument for them.

Cost

One forward pass for the whole frame: N boxes are N crops batched together, so per-image cost depends on how crowded the photograph is and per-person is the number that transfers. Measured on the fixture photograph, five people, torch-fp32, on an M-series laptop:

	MPS ms/person	CPU ms/person
small	9.9	23.1
base	14.8	39.3
large	28.5	111.6
huge	49.5	190.3

The ONNX graphs are for portability, not for speed

small, base and large publish an onnx-fp32 graph as well, verified to return the same joints as the checkpoint it came from. On the machine above it is slower than torch — 35.5 against 23.1 ms per person on small, 299 against 112 on large — because ONNX Runtime has no Metal path and falls back to its CPU provider. auto never picks it on CPU or MPS, so asking for it is a deliberate choice: serving without torch, or reaching a runtime torch cannot.

The graphs are also smaller than the checkpoints — 97 MB against 133 for small — because binding the expert to a constant lets the exporter fold the other five away. That is exactly 5/6 of the expert parameters in every variant.

huge publishes no graph. Its weights are 2.5 GB after folding, past protobuf's single-file ceiling, so ONNX writes them beside the graph rather than inside it — leaving a few hundred kilobytes of stub that loads fine where it was made and fails everywhere else. tools/export/vitpose.py refuses to write that. An artifact in mozo is one file.

No CoreML, and that one is a measurement rather than a gap. It converts directly and the joints agree to 0.0005 px, but it is not faster than torch on MPS: 22.9 ms against 22.6 for five people on small, 44.0 against 44.2 on base. A fixed batch shape does not help, fp16 does not help, and the Neural Engine is three and a half times worse. Publishing it would make it the auto choice on Apple silicon and cost a coremltools install plus a second copy of every variant, for nothing.

This is the opposite of RF-DETR, where CoreML is five times faster — probably because this trunk is a plain ViT, and Metal already runs matmuls and layer norms at full speed.

Parity

Joints land within 0.005 pixels of transformers on every variant — which is not a measurement of this model so much as of PixelFlow, whose coordinates round to 0.01. Half a step is the floor, and every variant sits exactly on it. Underneath that, the heatmaps are bit-identical.

Two pieces of preprocessing were rewritten to avoid a SciPy dependency, and both have a border rule that is easy to get wrong in a way that looks right: SciPy's mode="constant" returns the constant outside the frame rather than blending toward it, and its mode="reflect" repeats the edge sample where torch's skips it. Both are pinned against SciPy in tests/families/test_vitpose.py. See mozo/vendors/vitpose_deploy/PROVENANCE.md.

Moebius

Object removal. Give it a frame and a mask, and it repaints the masked region so the thing was never there.

car   = mozo.get_model("sam3").predict(frame, text="the red car")
clean = mozo.get_model("moebius", "general").predict(frame, car, seed=0, dilate=8)

clean.shape == frame.shape        # True -- same size, and outside the mask, the same bytes

It answers with a picture, and it is the only family that does

Every other model here describes an image. This one rewrites one, so there is no Detections, no Classifications and no confidence number — the model does not estimate anything. It draws a sample. Run it again with seed=1 and you get a different, equally valid removal; neither is more correct than the other, and that is why seed is a first-class argument rather than a hidden constant.

Everything the seam does not reach comes back byte-identical

Upstream returns the decoder's reconstruction of the whole frame, which changes every pixel including the ones nobody selected. mozo composites: the hole is the model's, everything else is your own array, unchanged. That is what makes it safe to point at a photograph you care about.

Precisely, because the difference matters at the border: the seam is feathered with a 3-pixel blur so it reads as a gradient rather than a cut-out, and that blend spreads about 8 px past your selection. Beyond that band the bytes are yours untouched. Pass feather=0 and the untouched region is exactly the mask's own edge -- at the cost of a visible cut-out.

If the thing is still faintly there

Raise dilate. An object's shadow and its antialiased edge sit outside the mask that found it, and removing the object without them leaves an outline. This is the single most common surprise, and 8–16 pixels usually settles it.

The same knob fixes the other cause: on a mask only a few pixels across, the feather pulls the blend below full strength even at the centre, so a little of the original survives everywhere. Growing the mask fixes that; shrinking the feather just makes the seam obvious.

It runs at 512×512 and nothing else

Not a setting — a property of the weights. The cross-attention's positional table is stored with a row per latent cell, so there is nowhere to put the positions of a larger image. predict resizes for you and composites back at your own resolution, which means detail inside the hole is capped at 512. On a large photograph that is visible if you go looking for it.

Two variants

general for arbitrary photographs, places2 for scenes and backgrounds. Upstream also publishes two face-specific checkpoints; mozo does not carry them, and PROVENANCE.md says why.

Over HTTP the selection is a rectangle:

curl -X POST "http://localhost:8000/predict/moebius/general?box=100,200,340,560&seed=0" \
  -F file=@photo.jpg -o clean.png

For a mask that follows an object's outline rather than a rectangle, run a segmenter and use the Python API.

BEN2

Background removal. Give it a photograph and it returns an alpha matte — a per-pixel opacity, not a binary mask.

model = mozo.get_model("ben2")

alpha = model.predict(frame)                    # (H, W) uint8
rgba  = model.cutout(frame, refine=True)        # (H, W, 4) uint8

A matte, not a mask, and that is the whole point

Every other model here that outlines something gives you a decision per pixel: in or out. Threshold that around a head of hair and the hair goes with the background, because a strand thinner than a pixel was never fully either. BEN2 answers with the mixing fraction instead, which is what a compositor actually needs.

This is also why it is worth having alongside SAM 3 rather than instead of it. SAM 3 answers "where is every cow"; BEN2 answers "how much of this pixel is the cow". The first needs a prompt and gives you instances; the second takes no prompt at all and gives you one matte for the whole foreground.

The default alpha is not a probability

The network emits a sigmoid, and upstream's postprocess then min-max stretches it per image, so the most-foreground pixel in this picture becomes 255 and the least becomes 0. mozo reproduces that by default, because it is what the model's own users get back and what its published examples show:

alpha = model.predict(frame)                  # contrast-stretched, upstream's default
alpha = model.predict(frame, stretch=False)   # the calibrated sigmoid, 0.5 means something

Compare a stretched alpha within an image, never across two. An image whose most confident pixel scored 0.6 still comes back with pixels at 255, and one the model was sure about everywhere comes back looking identical. Use stretch=False when you are thresholding, comparing frames, or feeding the matte to something else.

Upstream divides by max - min with no guard, so a frame it reads as uniform produces nan cast to uint8 — silent garbage rather than an error. mozo skips the stretch when the matte is flat and returns the constant.

refine changes colour, not opacity

Along a soft edge every pixel is a mix of foreground and background. Composite it onto something new and the old background shows through as a fringe. cutout(refine=True) estimates what the foreground colour would have been on its own and rewrites RGB accordingly — the alpha is untouched. It costs two full-resolution box blurs, which is why it is off by default.

It also takes a different alpha, and that is upstream's doing rather than a choice here: the plain path resizes bilinearly and then stretches, the refined path casts the raw 1024×1024 sigmoid to uint8 and resizes that. Both are reproduced exactly, which is why stretch has no effect on the refined path.

It runs at 1024×1024 and nothing else, and it squashes to get there

Not a setting. Each image is resized to a square 1024 — aspect ratio not preserved — then split into four 512×512 quadrants which are matted alongside a 512×512 downscale of the whole frame. The backbone therefore sees five images per photograph, and the decoder splits them [4, 1] at every rung, using the global view to gate the quadrants and folding the refined quadrants back in. That five-way split is what the confidence-guided refinement is built on.

A 4000×500 panorama is squashed to a square, matted, and unsquashed. That looks wrong and is correct: the weights were trained that way, and letterboxing instead moves every pixel.

One variant, MIT on both halves

Upstream publishes one checkpoint, and both the code and the weights are MIT — stated in the repository's LICENSE by the copyright holder and again as license:mit on an ungated model card. That is rarer here than it should be: the best-known alternative, Bria's RMBG-2.0, is CC BY-NC and needs a paid agreement for commercial use.

Torch only, and both graphs were measured before that was decided

EXECUTES = ("torch",). Both an ONNX and a CoreML graph were built and neither is published, which is a result rather than a gap — tools/export/ben2.py re-runs the measurement. ONNX is slower than torch (0.88x) and off by 4.9e-05. CoreML is 1.56x faster than torch on Metal, the best number in the family, but 7.8% of its alpha pixels differ by more than a grey level, and the difference sits entirely on the edges — the one part of a matte that is the whole point of running one. Worth revisiting; mozo/vendors/ben2_deploy/PROVENANCE.md has the numbers to start from.

On Apple silicon torch runs this at ~600 ms per 1920x1281 image against ~5.5 s on CPU, so the device matters far more here than the runtime does.

Two things upstream publishes that mozo does not carry: BEN2_Base.pth is a training checkpoint whose 1.13 GB is three quarters optimiser state around the same 535 tensors mozo publishes as 380 MB, and BEN2_Base.onnx is a float16 export of the CUDA path whose own runner script feeds it unnormalised input. mozo/vendors/ben2_deploy/PROVENANCE.md has the measurements for both.

EasyOCR

Text recognition. Finds every line of text on a page and reads it.

model = mozo.get_model("easyocr/english")
found = model.predict("sign.jpg")
found[0].text                # 'EXIT 42' — what it says
found[0].class_name          # None — OCR reads content, it does not pick a class
found[0].segments            # the four corners as read, for rotated text

A variant is a script, not a language: english, latin, chinese-simplified, japanese, korean. latin alone covers 41 languages and reads every character its charset holds. Upstream instead picks a checkpoint from a language list and then suppresses characters outside those languages at decode time, so its output depends on something that is not a property of the weights — ask mozo's latin for café and you get café.

These five are 88% of upstream's own download counts, out of the seventeen recognisers it publishes.

Detections carry text, not a class name. Every other family here names a class from a fixed vocabulary; this one produces content that belongs to no vocabulary, and PixelFlow keeps the two apart. class_id and class_name are None.

The quadrilateral is kept in segments with its axis-aligned hull in bbox, because real-world text is rotated and a box alone throws the orientation away. Level lines come back top to bottom, followed by tilted ones — that is upstream's ordering, and it is not a reading order: a two-column page interleaves.

Two graphs: CRAFT locates the text, a CRNN reads it. There is no NMS — a detection is a connected component of two heatmaps, one scoring "inside a character" and one "between two characters of the same word". About 200 ms a page on CPU and 31 ms on Apple silicon, within 1% and 22% respectively of the published package running the same weights on the same device.

The GPU path is not bit-identical to the CPU one — strings and quadrilaterals are exact, confidences move by up to 2.2e-05. The verification below is a CPU claim; pass device="cpu" if you need exactly those numbers.

Verified against easyocr at every stage — preprocessed tensor, both heatmaps, quadrilaterals, each crop, the decoded string and its confidence — with no tolerance: 1,275 comparisons across the five variants, every one bit-identical. tools/verify/easyocr.py needs pip install easyocr and the weights.

Why no ONNX or CoreML

EasyOCR is the first family in mozo whose input shape is a property of the input, so neither export is publishable.

The detector's input is the page scaled to fit 2,560 and padded to a multiple of 32, so its shape follows the photograph. The recogniser's width is ceil(aspect) × 64 per line, and padding a line out to a fixed width replicates its last column into more decode steps, which can change what it says.

With flexible shapes neither graph converts: the detector's U-Net upsamples to another tensor's runtime shape, and the recogniser hits an adaptive pool that only exports when the input size is known. At fixed shapes both convert and CoreML is the fastest runtime measured — 13.8 ms against torch-MPS's 18.8 for the detector — but parity comes out at 1.6e-05 on the recogniser, above the 1.4e-05 already known to flip a character, and enumerating shapes would mean 6,241 combinations for the detector alone.

Depth Anything V2

Monocular depth estimation, in two groups that are not interchangeable.

Relative depth — output is inverse depth on an arbitrary per-image scale: larger means nearer, and that is all it means. Two images cannot be compared to each other, and no value is a distance.

• small — fastest, lowest memory (Apache-2.0)
• base — balanced (CC-BY-NC-4.0, non-commercial)
• large — best accuracy (CC-BY-NC-4.0, non-commercial)

Metric depth — output is in metres, from fine-tunes on Hypersim (indoor, 0–20 m) and Virtual KITTI 2 (outdoor, 0–80 m). All Apache-2.0 per their model cards.

• indoor-small, indoor-base, indoor-large
• outdoor-small, outdoor-base, outdoor-large

model.unit is "metres" for the metric variants and None for the relative ones — mozo never guesses a unit.

Output is an HxW float32 array at the input's resolution. Over HTTP it is encoded as a 16-bit PNG with the range in the headers, because six of the nine variants predict metres and quantising those to 256 levels would discard the measurement.

CLIP

The first family here that answers with neither a box nor a map, and the only one that will hand back the numbers it works from rather than an answer.

CLIP is two networks trained together until their outputs land in one shared space: an image tower and a text tower, both ending in a 512- or 768-dimensional vector. Nothing about that space is labelled — the image of a forklift and the phrase "a forklift" simply end up near each other. Every use follows from comparing two vectors with a dot product.

• base — ViT-B/32, 512-d. The one almost everyone means by "CLIP"
• base-16 — ViT-B/16, 512-d. Same size, finer patches, slower and stronger
• large — ViT-L/14, 768-d
• large-336 — ViT-L/14 at 336px, 768-d. The most accurate and by far the slowest

All four are MIT, code and weights. The five ResNet variants OpenAI also publishes use a different image tower and are not carried; see the vendor's PROVENANCE.md.

Two products, one checkpoint

/predict classifies. Name your classes in words and each is scored against the image. No training, no labelled data, no fixed class list — which is what makes it the answer for "is there a person in this frame" when nobody trained a person detector for your cameras.

/encode returns the vectors. That is a handoff rather than an answer: a vector is useful only next to other vectors, which means a store mozo does not provide. The shape of the pipeline is embed a corpus once, keep the vectors in a vector database, then encode a query phrase and let the database find the nearest. mozo is the model in that pipeline and nothing else — it does not index, does not search, and does not persist.

The separation matters because searching a million frames for "forklift near a person" is a database query over stored vectors, not a million model runs. The model runs once per frame at ingest, and once per query.

The scores are cosine similarities, not probabilities

They are not softmaxed. They do not sum to one, they can be negative, and they are compressed — a good match sits far below 1.0, so 0.31 does not mean "31% sure". Compare them against each other, or against a threshold you calibrate on your own images.

A softmax would be worse rather than better here: it is relative to whichever phrases you happened to pass, so one phrase always scores 1.00 and adding a phrase moves every other number. Because mozo does not softmax, each phrase is scored independently and adding a phrase does not change the others' scores — which is the property that makes a threshold mean anything at all.

The towers load independently

Asking for phrases never builds the image tower, and vice versa. An ingest job holds the image tower alone; a query service holds 63.4M parameters instead of 151.3M. Nothing to configure — ask for what you need and the rest is never built.

An index is tied to the weights that built it

Both towers come from one checkpoint and were trained together until their outputs agreed. A vector from base means nothing against a vector from large, and nothing against a vector from a different revision of base. So a stored index is tied to the variant and revision that built it, and changing either means re-embedding the corpus. /encode returns both in the response for exactly that reason. This is the operational cost of an embedding pipeline and it surprises people.

Parity

Bit-exact against openai/CLIP at commit d05afc43, on all four variants: 104 comparisons across preprocessing, token ids, image features, text features, cosine similarities and logits_per_image — every one identical, with no tolerance.

Two traps that survive a code review and only a numeric gate catches. The activation is QuickGELU (x * sigmoid(1.702x)), not nn.GELU; substituting the standard one moves the features by 0.4 and neither raises nor warns. And upstream writes logit_scale * image_features @ text_features.t(), where * and @ share precedence — so the scale multiplies the features before the matmul, not the product after it. Same arithmetic, different rounding, 1.9e-06 apart.

SigLIP 2

The second family that hands back vectors, and the answer to the thing CLIP is bad at.

Same shape as CLIP — an image tower and a text tower trained together until their outputs land in one shared space — but trained with a different loss, and that changes what the number at the end means. CLIP was trained to rank a batch: which caption in these 32,768 goes with this image. So a CLIP score is only meaningful against the other scores you asked for. SigLIP was trained pair by pair with a sigmoid loss: does this caption go with this image, yes or no. So its score means something on its own.

• base-224 / base-256 — ViT-B/16, 768-d. The small, fast end
• so400m-384 — the shape-optimised 400M tower at patch 14, 1152-d. The quality point
• so400m16-256 — the same tower at patch 16 and a smaller input, so roughly a third the cost
• giant-384 — a 1536-d image tower paired with the so400m text tower. The strongest, and the most downloaded of all of them

Apache-2.0, code and weights, ungated. Google publishes fifteen fixed-resolution variants and mozo carries these five, which took 89% of the downloads across those fifteen; the other ten need a manifest entry and a checkpoint, not new code. The two -naflex variable-resolution variants are different — they use a different image tower — and are not carried; see the vendor's PROVENANCE.md.

Multilingual by default: the text tower carries Gemma's 256,000-piece vocabulary rather than an English byte-pair one, so there is no separate multilingual variant to choose.

The score is a probability for one pair

sigmoid(cos × exp(logit_scale) + logit_bias), which is what upstream's own examples print. Two consequences, and they are the reason to reach for this family:

A single phrase is a well-posed question. Ask "is there a forklift in this frame" with one phrase and the answer means something. With CLIP you must pass a complete set of classes and read the ranking, because a lone cosine similarity has no absolute zero to sit against.

Everything can be near zero. On the fixture photograph — people at a table with a laptop — "a photo of people" scores 0.035 and "a photo of an elephant in a pool" scores 0.000. A softmax cannot express "none of these", because something always has to win.

Adding a phrase still does not move the others, exactly as with CLIP.

It is not a calibrated class probability. Nothing in the training made classes compete, so the number is how well this phrase matches this image, not P(class | image). Absolute values run low — a good match is often a few percent, not 90% — because the learned bias sits near −17. Read it as a score with a meaningful zero and calibrate a threshold on your own images.

Write your prompts in lowercase, or let mozo do it

SigLIP 2 was trained on lowercased text, and mozo lowercases for you. This matters more than it sounds: "A Photo Of People" scores 0.0349 lowercased and 0.0001 as written, on the same image.

It is also a place where the published tokenizer configuration is misleading — it names a tokenizer class that ignores its own do_lower_case flag, so the obvious way to load it preserves case and quietly ruins every capitalised prompt. mozo follows what the authors trained; the vendor's PROVENANCE.md records the divergence and the evidence for it.

The towers load independently, and it matters more here

Asking for phrases never builds the image tower, and vice versa. The text tower is most of the checkpoint for this family — Gemma's vocabulary alone is 786 MB of a base variant and 1,180 MB of an so400m one — so an ingest job that only embeds images avoids most of the memory.

SigLIP 2 does not replace CLIP

They are separate vendors with separate weights, separate tokenizers and separate gates, and mozo carries both. CLIP is what a great deal of existing tooling pins, and a vector from one means nothing against a vector from the other. Choose SigLIP 2 when you want a score you can threshold; choose CLIP when you need CLIP's own embedding space.