Why a richer English-keyword model made Chinese-policy retrieval worse.

A retrieval baseline that doesn't lift, on purpose

We added a fifth row to the cascade backtest scoreboard: a retrieval-driven baseline that selects the nearest historical precedents from the cascade corpus and inherits their realized sectors, lag, and subsidy band as the prediction. The intended backend was a bilingual sentence-transformer that places English paraphrases and verbatim Chinese text in a shared embedding space, but the model weights would not download in this sprint (the file server connections stalled after roughly ten megabytes of a 470-megabyte download across multiple retry strategies). The row that actually shipped is a character-level term-frequency retriever over the same corpus. It recovers 7 percentof realized beneficiary sectors at top-3 — worse than the mode baseline (20 percent) and roughly eight times worse than the keyword baseline (57 percent). The mean absolute error on lag prediction is 12.4 months; subsidy-band accuracy is zero. The result is a clean negative one: on a corpus of ten cascades, richer surface features do not carry the model. The lift, when it comes, has to come from a semantic encoder that closes the gap between English paraphrase and Chinese original.

The original retrieval floor was honestly weak. This sprint had to either lift it or document the floor.

The founding release shipped a word-level term-frequency retriever as the only retrieval substrate on the page. The dossier was honest that this is a weak floor: it cannot read the Chinese original, it misses near-synonyms (treating computing power as unrelated to algorithmic infrastructure), and it ranks two cascades together only if they share surface word stems. The cosine values in the founding backtest are accordingly small — 0.21 between the integrated-circuit and quantum frontier-field excerpts was the headline tie, and most pairs sat below 0.05. A retrieval-augmented model built on top of a retrieval substrate that thin cannot do much.

The roadmap commits to a bilingual sentence-transformer as the next retrieval backend, with a Chinese-policy-tuned model as the longer-term target. This release was supposed to be the staging sprint that wired both backends into the scoreboard so the next reader could see the retrieval substrate's contribution as an explicit row, alongside random, mode, and keyword. The plan was: wire the sentence-transformer backend, fall back to a character-level retriever if the model weights don't load, ship whichever runs, log the disposition transparently.

Lifting the model row above the keyword row is the explicit success criterion for the long-context language-model upgrade.

The sentence-transformer weights did not load. The character-level fallback shipped. The substantive question this paper answers is whether the fallback gives us information — whether a negative result on a richer surface-similarity retriever is itself informative about where the next lift will have to come from.

A character-level retriever should not, on its own, lift over keyword on ten cascades

A character-level term-frequency retriever differs from a word-level one in that it slides a window of two- to five-character bigrams across each token, building a vocabulary of character chunks rather than whole words. It is more robust to morphology (it sees the shared root in computing and compute) and to acronyms (it picks up NDRC as a distinguishable unit rather than dropping it as a single unknown token). On a richer corpus, it would be expected to lift modestly over a word-level baseline where surface morphology and acronym overlap carry signal.

On the cascade corpus, the prior expectation was that the lift would be small or negative for two reasons. First, the corpus is ten documents. Character-level vectorization expands the feature dimensionality dramatically — tens of thousands of character chunks versus a few hundred whole words on the same corpus — which means each cascade's vector is sparse and noisy in a way that doesn't average out across more documents. Second, the cascades' near-neighbour structure isn't driven by morphology — it's driven by topic. Two cascades that share “chip” and “semiconductor” should retrieve each other; character-level retrieval doesn't do that better than word-level retrieval, it does it differently and on a smaller signal.

The hypothesis was therefore that the character-level retriever should perform comparably to or worse than the word-level one on this corpus, and that a clear miss on the realized-sector recall would be diagnostic. We did not expect the character-level substrate to be the long-term retrieval backbone. We expected the row's job to be telling us what kind of lift can't come from richer surface features.

Same backtest, fifth baseline row, two backends wired

We added a retrieval baseline that selects the top three nearest precedents from the historical-cascade corpus and inherits their realized sectors, lag, and band as the prediction. The function probes for a sentence-transformer backend first; if the weights are not in cache, it falls back to a character-level term-frequency retriever. The fallback ran in this sprint; the sentence-transformer backend will auto-promote when the weights land in cache, with no further code change required.

The character-level retriever uses a term-frequency vectorizer with character bigrams of length two to five and a vocabulary capped at twenty thousand features, over the same English-paraphrase corpus the word-level backend uses. The neighborhood size is three, evaluated on the leave-one-out fold structure of the founding backtest (one held-out cascade per fold across ten folds). The prediction for each fold is the majority-vote sector set across the three retrieved precedents, the median lag, and the modal band.

A retrieval-backend field has been added to the backtest output so the scoreboard row records which retriever actually ran. Today the field reads “character-level”. When the sentence-transformer weights load successfully, the same scoreboard cell will compare the sentence-transformer against the four pre-existing baselines on identical folds.

Seven percent recall at top-3, worse than mode, eight times worse than keyword

The character-level retrieval row reports 7 percent recall at top-3, 12.4 months mean absolute error on lag, and 0 percent subsidy-band accuracy. The keyword baseline on the same folds reports 57 percent recall (eight times higher), 10.5 months mean absolute error on lag (better), and 40 percent band accuracy (40 percentage points higher). The mode baseline reports 20 percent recall, 9.8 months lag error, and 40 percent band accuracy. Character-level retrieval is worse than mode on the headline metric and on band accuracy, and worse than keyword on every metric.

The diagnostic is clean. Character-level retrieval pulls in cascades that share morphology and acronym overlap with the query, but those near-neighbours' realized sectorsare largely unrelated to the query's gold set. The integrated-circuit cascade and the quantum cascade share “frontier field” surface text and both sit under Chapter 9 of the 14FYP. Character-level retrieval picks them out as neighbours, but their realized sector sets — semiconductors and accelerator chips on one side, quantum information on the other — don't overlap. Surface similarity is not a proxy for sector similarity at this corpus size. The result is exactly the kind a forecasting program with a discipline of running baselines wants to see: it tells you the retrieval substrate you wired is not the right one, before you build anything on top of it.

Reading this against the founding scoreboard, the four-row picture is now stable. Random (17 percent) and mode (20 percent) are the trivial baselines that any non-trivial predictor must clear. Keyword (57 percent) is the bar that the retrieval-augmented model needs to clear to justify its complexity. The stub model (52 percent) is close to the keyword bar but does not yet clear it — expected, since the stub is a keyword rule. Character-level retrieval (7 percent) is below all of them. The next row that should appear on this scoreboard is the bilingual sentence-transformer, which this sprint wires but does not yet ship.

The negative result is the lift roadmap

What this paper finds is what the roadmap was already committed to, but now grounded in a baseline rather than a hunch. On a corpus of ten cascades that span semiconductors, electric vehicles, AI infrastructure, common prosperity, biotech, and quantum information, the near-neighbour structure that matters for forecasting is topical, not morphological. A retriever that operates on character surface features — even a sophisticated one with twenty thousand character chunks in its vocabulary — cannot recover that structure. The only retriever that can is one that maps both the English paraphrase and the Chinese verbatim text into a shared embedding space where “sovereign computing power” in the current 15FYP and “build a national integrated big-data center system” in the 14FYP come out close to each other.

Publishing the negative result has two operational benefits. First, it rules out a whole class of cheap fixes (richer ngrams, character-level features, larger vocabularies) that might have looked tempting if the next release's sentence-transformer baseline failed to land on schedule. Second, it gives a calibrated read on what fraction of the lift attributable to retrieval has to come from semantic embedding rather than feature engineering. If the next sentence-transformer row ships at, say, 45 percent recall at top-3, the right read is that the substrate is the semantic encoder, not the ngram width. If that row also fails to clear the keyword baseline, the read becomes that the long-context language-model swap has to do the work.

The pattern this surfaces — ship the baseline that lets you read the next missing capability — is the operating discipline the scoreboard is for. Every row in the backtest table is a different hypothesis about where the predictive signal lives.

What the ten-cascade floor will and won't tell us

The result is corpus-size-bound.The character-level retriever's failure on ten cascades does not generalize to its viability on larger corpora. The planned index expansion to the 13th and 12th plans, targeting roughly thirty cascades, may move the picture — though we expect the dominant variable to remain semantic versus surface, not ngram width versus word.

The fallback was a single backend.This sprint wired exactly one fallback rather than a sweep over backends. A character-level retriever with a different ngram range, a different vocabulary cap, or an L2-normalized vector space might move the numbers slightly. The published row is the single specification the pipeline ran with; we did not tune. Reading the row as an upper bound on what character-level surface features could do on this corpus is too strong; reading it as a clean signal that surface features don't carry is fair.

The sentence-transformer backend has not been observed on this scoreboard.The retrieval-floor result is informative because it is honestly worse than the trivial baselines, but the substantive claim — that semantic embedding closes the gap between English paraphrase and Chinese original — remains untested until the model weights download and the row populates. The cleanest possible outcome is the sentence-transformer row lifting recall at top-3 above the keyword baseline; the cleanest cautionary outcome is the row failing to lift, which would tell us the long-context language-model swap has to do work that the retrieval substrate alone cannot.