LLM scorer wired up. Run held until an API key lands.
A provider-agnostic scorer with a frame-aware prompt and a three-provider fallback chain is now in place. The 31 May run did not execute; rather than fabricate metrics, the runner wrote a status placeholder.
The previous release shipped a deliberately simple word-list scorer against a real reliability diagram. The natural next step is the model that has to beat it. This release lays the scaffold for that model: a single provider-agnostic LLM scorer with a fixed JSON-mode prompt and a fallback chain across Anthropic (Haiku 4.5), OpenAI (GPT-4o mini), and Gemini 2.5 Flash. The system prompt is written explicitly to handle the case the lexicon mis-scores most often — Chinese press of record celebrating U.S. bear stories, where a U.S. slip reads as a Chinese bull.
The 31 May run did not execute. The Gemini call returned an unsupported-region error and no Anthropic or OpenAI credentials were available. Rather than fabricate numbers or write placeholder metrics, the runner emitted a status record marking the run as pending credentials. The discipline is explicit: no fabricated numbers, even as placeholders. The moment a working key lands, the run fires and the published reliability diagram gains a second LLM series on the same axes as the lexicon baseline.
One guardrail got built into the runner alongside the scaffold: a single-call smoke test fires before scoring all sixty articles. The point is to trip regional blocks or auth failures at zero cost rather than burning partial spend on a doomed batch. The kind of check that pays for itself the first time a provider region rejects you.
Technical detail
- Provider-agnostic LLM scorer with a fixed JSON-mode prompt, a frame-aware system prompt, and a fallback chain: Anthropic, then OpenAI, then Gemini.
- On a run with no working credentials, the runner writes a status placeholder — not fabricated metrics.
- Direct HTTP calls, no SDK install required — the scorer runs on any machine with Python and a key.
- A one-call smoke test fires before the sixty-article batch — regional blocks and auth failures fail fast at zero cost.
What this opens. The substantive LLM result will publish as its own article once the run executes — not as a changelog entry. The lexicon baseline is the bar to clear: mean absolute error of 0.227, a Pearson correlation of +0.76 with human coders, and the empty bear bin to populate. If the LLM clears all three, that is a v0.3 release and a substantive article. Until then, the scaffold sits ready and the discipline holds: no numbers without a real run behind them.