Where local models hold, and where they break

We ran four open-weight models on one 12 GB graphics card against Claude Opus 5 on the same 95 documents, with the answer keys and the bar for parity written down before either side ran.

Download Benchmark Report No. 02 (PDF, 19 pages, 1.4 MB)

Seven tasks, one scorecard

Best local model per task against Claude Opus 5 on the same documents. Parity, usable with review and gap are defined in the report and were fixed before the runs.

What we found

How it was run

What people ask about this

Which local model should we run?
On this evidence the task matters more than the model. Qwen3.5 9B was the strongest of the four we tested, reaching parity or usable with review on 3 of 7 tasks, but the spread across tasks is far wider than the spread across models.
Does a local model hallucinate more?
Not in the way most people expect. Across the study the dominant local failure is omission rather than invention. The exception is first-pass marketing review, where false flags run high enough that the output is a candidate list for a reviewer and nothing more.
Is this an independent benchmark?
No. Black Lily installs local deployments, so we benefit when local models look good. That is why the study was pre-registered, why the losses are published alongside the wins, and why the whole corpus and grading code is downloadable so you can run it yourself.
Will it work on our documents?
That is the only version of this question that matters, and this report cannot answer it. The method is published so a firm can run the same measurement on its own corpus in about a week.

How a deployment works end to end: private on-premise AI.

Black Lily is an AI implementation consultancy. We install private on-premise AI on hardware you own, for hedge funds, registered investment advisers, and family offices whose best documents are the ones they cannot send to an outside service, and we turn paper, handwriting, and inconsistent spreadsheets into structured data your systems can import. Founded by William Dorman, Founder & CEO.

Phone: (432) 234-3779 · Book a free 30-minute scoping call: cal.com/black-lily/30min

Black Lily home · Hedge funds · RIAs · Family offices · Local AI · Data digitization · Blog · Privacy Policy · Terms of Service