Vision Trial Log

Can an AI model, local or in the cloud, look at product photos and videos and reliably say what's in them (the product, the design, the on-screen text) without a person having already told it? The real question isn't local versus cloud: it's finding whichever model gets this most accurate. Testing local models first is part of that search, not the whole point of it. This matters for two things: reading incoming footage faster, and eventually sorting or matching footage on its own.

The practical goal: have a model read incoming photos and video and tag what's actually in each clip, and roughly where in the clip, so that tagging can drive things downstream, like picking the right footage automatically when generating video.

Vision model is the umbrella term for everything on this page. Under that umbrella, three different kinds of tool are being tested, because they do three different jobs.

A third job, Judge, shows up in the catalog-matching rounds below: given a reader's written description, decide which real product it matches. Text only, never sees the image.

Blind means the model never sees the right answer, and nobody checks its answer until every model has finished.

1. Every clip is copied out fresh: audio stripped, hidden file metadata cleared, renamed to a neutral ID like N07 instead of its real filename, so nothing about it can hint at the answer.
2. Before any model runs, the real answer for each clip goes into a private answer key, written from the database or from watching the source footage. That file is never shown to a model, and it stays closed while the models are running.
3. Each model gets a completely fresh run per clip. No memory of the last clip, no chat history, no hints.
4. Only after every model has answered does the answer key come out, checked against both the recorded facts and the actual video frames.

Same three prompts, word for word, every time, for every model, every clip. Nothing gets adjusted per clip and nothing gets a second try.

The prompts themselves stay private. In short: every model was told to use only what it could see, say "unknown" when unsure, and never guess a brand or material it couldn't see.

Qwen 27B was clearly strongest at picking the right design. But when I read what it actually wrote, both Qwen models still invented things: called products "wooden" when material was never visible, gave a wrong dog breed, cited a timestamp past the end of a clip, and misspelled on-screen text.

On a separate clip of a CNC cutting machine, correction made Sept 28 after a closer frame check: both models did see a real thing, not nothing: a wall sign for the machine's own brand, and a real design being carved (an animal-like graphic with text). Where they failed was narrower than first reported: every model tried on this clip read that carved text as something different, and none of the guesses matched the real word on the wall. Small, hard-to-read text gets a confident wrong answer instead of "unknown."

Both models stayed reliable at reading legible on-screen text, even on the clips where they had no idea what object the text was printed on.

Since the readers missed the same-cooler pattern, one frame from each of the 13 round 2 clips went through the three matching tools instead, comparing pictures to each other, no reading involved.

The split makes sense: the readers were asked to name the object, which needs more than pixels. The matchers were only ever asked whether two pictures are the same thing, which doesn't require naming anything.

Same 13 videos and 3 stills, run through two true video-native cloud models: Gemini 3.1 Pro and Gemini 3 Flash. A third, a free hosted open-source model, was rate-limited on video every attempt and only partly completed on stills. A capacity problem on the free tier, not a rejection of the test.

Real cost for the full run, from the provider's own billing, not an estimate: $0.42.

Every earlier round tested candidates. This round tests the incumbent: the exact model already running a real production alt-text pipeline, same 13 clips and 3 stills, same fixed prompts, read critically rather than taken at its word.

Two conditions, on purpose. Condition A matches what production actually does today: one static thumbnail frame, no video, ever. Condition B gives it the same fair, frame-sampled treatment as Qwen and Gemini.

Cost, real: condition A (today's actual pattern) is the cheapest of every model tried, and also the worst on accuracy in its real form.

Every round before this asked "what is in the frame." This round adds a second stage: once a reader writes its description, who decides which real product it actually matches? Two products pulled live from a store catalog, plus two real decoy products. Same 4 readers described the two real products, blind. Three independent judges, Jev, Codex and Claude, each looked at a reader's description plus the three candidates and picked one, never seeing the images.

A model we hadn't tried yet: Gemini 3.8 Flash, run on the identical 13 clips and 3 stills.

Real cost: $0.066 for all 16 items.

The decisive follow-up. Round 5 proved matching works on easy content. This round tests it on the exact four clips where every reader failed. Reused the readers' original descriptions, pulled the real matching catalog product plus 3 real decoys (including one deliberately harder, similarly personalized one), and ran all three judges again.

Which model for which job: pros and cons, in the same order as the at-a-glance chart above.

• Reading text

  Any of the models tried does the job. The gaps between them are small.

• Naming an object

  Don't trust it alone from a close-up crop, and the model live today is the weakest one tested at it, not merely average.

• Same product?

  That's a matching question, not a reading one. CLIP is the strongest of the three tried.

• Matching a description to the real catalog

  Worked perfectly, twice: once on easy content, once specifically on the hard content that broke every reader (48 for 48). The strongest, most decisive result in the whole trial.

• Switch the reader model?

  Revised: one candidate model clearly outperformed the pack on the core failure, and the model live today's real pattern is the worst tested, not a tie. Worth a real look.

• Cost, if this runs weekly

  At 20–50 videos and 50–100 images a week, every option priced out under about $7/week. Price is not the deciding factor here; accuracy is.

• How this was checked

  An independent audit, fresh context, adversarial, scored everything against the sealed answer keys before reading any prior write-up, and found real things to fix, now corrected above.

• Not yet tried

  Fresh footage nobody has reviewed, real (non-staged) customer photos, and a genuinely hard matching decoy that shares wording with the real product.

New trials, same format: what we tested, what broke, what we'd actually use. No spam, just new findings as they publish.