The correction that did not reach the picture

Three days ago I found that every externally published number for one of my tools was a generation old, and I corrected all five public surfaces the same day. I wrote that down as finished.

This week I went looking again, and found four places still carrying the old numbers. All four have something in common, and it is not that I missed them by accident.

The first was an architecture diagram. On the product page, the prose says the grade network loses to ordinary kriging. Directly beneath it, an SVG I had hand-authored said grade-nn R2 0.9613 (ties kriging), next to pit-surrogate AUC 0.9811 and CPIT LP bound offline (10.46% gap). Three retired numbers, in one picture, on the same page as their own corrections.

The second was a figure in the technical report. That report had already been reissued as version 2.0 specifically to carry the correction, and the body text carries it properly. Panel (b) of figure 1 still read exact solve, in-browser, sub-second to 14k blocks over timings that were measured under Node, on one machine, and that the correction had explicitly retired as a way of describing the method. The caption underneath repeated it in words.

The third was worse in a quiet way. The figure was generated by a script whose own docstring said it regenerated the figures “from the COMMITTED artifacts”. It did not. It read a JSON file that someone, meaning me, had once copied out of the artifacts by hand, and that nothing regenerated. That file still held solve times of 5.2, 237.3 and 259.3 milliseconds. The shipped artifact said 7.3, 564.6 and 831.0. The relative errors in the same file matched the shipped ones to the last digit, which is exactly the tell I should have read: relative error is a property of the algorithm and reproduces anywhere, while a solve time is a property of one machine on one afternoon. The numbers that could drift had drifted, and the numbers that could not had not.

The fourth was three older posts. Not product pages, not documentation. Ordinary writing from weeks earlier that had mentioned the tool in a list and quoted the tie in passing. Two of them were posts about the discipline of publishing negative results, which had gone on quoting a flattering number for a result that was in fact a loss.

What these four have in common

None of them is a place where I would have typed the number while correcting the product.

A correction sweeps the surfaces you think of as carrying the claim. I thought of the product page, the personal site card, the CV entry, the resume line and the report. That list is a list of prose. It does not include the picture next to the prose, the figure generated from a snapshot of the prose, or the sentence in an old post that mentioned the tool while talking about something else.

The application itself never drifted once, in any of this. It renders every figure from the committed artifacts at run time, so when the artifacts moved, it moved with them, without anybody deciding to. Everything that drifted, drifted because a human had transcribed a number into a place that had no path back to its source.

What I changed

The obvious half was fixing the four. The useful half was removing the transcription.

The figure script now reads the shipped artifacts directly and writes its snapshot file back as provenance rather than reading it as input, so a figure in the report cannot disagree with what the application serves. That is one fewer number in the world that a person types.

The panel that had shown one scheduling scenario while disclaiming the other in the caption now plots both, each against its own bound, each labelled with its own gap: 3.81 percent on the published benchmark scenario, 11.29 percent on a synthetic twin that is not comparable to it. Conflating those two was the defect that version 2.0 existed to correct, so the picture now carries the correction rather than relying on a caption to apologise for it.

And the surfaces got a gate. A small script fails the build on the characters and patterns I keep having to sweep by hand. I tested it against its own subject before trusting it, by injecting the defect and confirming the build went red, because a gate that has never failed is not evidence of anything. It immediately found things the manual sweep had missed, which is the entire argument for having it.

The part I would keep

The record I wrote on the day said “all five public surfaces were corrected”. It was written in good faith and it was wrong, and it was wrong in a way that made it less likely I would look again. That sentence is now amended in the same file, with what it missed.

I do not think the lesson is to search harder. I searched reasonably hard, and a diagram is not hard to find. The lesson is that a number which lives in more than one place will eventually disagree with itself, and the only durable fix is to reduce the number of places. Prose that quotes a measurement should be treated as a cache: useful, faster to read than the source, and guaranteed to go stale unless something invalidates it.

The tools are PitForge and its source, where all of the above is committed, including the corrections to the corrections.