One Week, Two Silent Data-Corruption Bugs: A Field Guide From HFlow

The project

HFlow (Hebbian Robotics, YC S26) is an open-source platform for physical-AI data. Robots record episodes. The pipeline scores those episodes, stores the scores as measurements, and curates a manifest for model training.

I came in as an external contributor and did a read-only audit: read the storage code against the queries that read it back.

I graduated two months earlier. This was my first week contributing to the repo.

Bug 1: a type check that looks correct but isn't

Python has its own number types: int, float. Most ML code runs on NumPy, which ships its own: np.float32, np.int64, and friends. The two families behave almost identically. You can add them, print them, compare them.

Almost. They disagree on exactly one question: “are you a float?”

isinstance(np.float32(0.4), float) returns False. Same for every NumPy scalar except np.float64, which slips through by accident because it subclasses Python's float.

HFlow's write path asked exactly that question. It branched on Python types (int goes to one column, float to another, string to a third) in src/hflow/catalog.py. A NumPy score matched no branch.

Here is the dangerous part: nothing crashed. The row was still written. The key got stored, and every value column got NULL. Like a receptionist who files the envelope but, not recognizing the handwriting, leaves out the letter. The pipeline accepted the measurement and threw away the number inside it. No exception. No warning.

The loss happens later, at read time. Curation builds training manifests with filters like “keep episodes where black_pct is below 1.0.” SQL comparisons have three possible answers: yes, no, and unknown. A filter keeps only yes. Compare NULL to anything and you get unknown. So SQL treats “don't know” as “no,” and drops the row without telling you.

Every episode that stored a NumPy score failed that filter. It vanished from the manifest. No quarantine flag, no log line. The dataset shrank, and curation looked more selective than it was. If you monitor manifest length, you see a clean cut. You never see a type bug.

Two details made this findable. A few lines from the broken branch, fingerprints were routed through repr(), which handles NumPy fine. The codebase knew these values arrive; only the storage branch next door never checked. And an existing test asserted the write “does not crash” without ever reading the value back. The test locked the loss in as passing behavior.

The fix belongs where the foreign type first enters: the boundary. Coercing at read time would hide the write bug forever and leave two code paths to keep in sync. Coerce once at write time: NumPy scalars become their Python equivalents, anything that can't be coerced is refused loudly.

This was issue #126. The fix is PR #127.

Bug 2: a column renamed behind your back

HFlow lets checks name their own measurement keys. One check named its measurement task. The episodes table already had a column called task, holding metadata like fold_napkin.

When the curation layer pivots measurements beside the episode columns, DuckDB has a name collision. Its default move is to quietly rename one side: the measurement becomes task_1. No error. Nothing in the docs you'd think to check. It's like a hotel front desk dealing with two guests named John by re-labeling one of them John-2: mail addressed to John still gets delivered, just to the wrong man.

So SELECT task returns the metadata string, while your actual measurement (say 99.0) sits behind task_1, a name no query will ever ask for. You get the wrong answer, not an error. Worse: identifiers are case-insensitive, so a check emitting Task collides the same way.

The catalog comment claimed the shadowing case was handled. It wasn't. The pivot just did what pivot engines do by default. The lesson: treat keys coming from checks as untrusted input, and validate them against the schema you already have. If a check can name a key that collides, someday one will.

This was issue #130, fixed in PR #133.

The fixes

Both fixes are small, and both sit at the boundary, before any downstream code can read a bad value.

For bug 1, values are normalized once, before the run fingerprint is computed, so storage and idempotence see the same thing. NumPy scalars are coerced to their Python equivalents; anything that cannot be coerced is refused loudly. PR #127 closes #126.

For bug 2, shadowing keys are refused at append time. The reserved set is derived from the episodes schema itself, so new columns get guarded automatically. There is no allow-list to forget. PR #133 closes #130.

Both were reviewed and merged the same day they were filed.

What the maintainer's review taught me

Kingston re-ran the whole suite locally before approving. He independently verified the case-sensitivity claim and left the result in the review: “Pivoting a Task key beside e.task gives columns [episode_id, 'task', 'Task_1'], SELECT task returns the metadata”. He corrected my first fix placement with a line-number argument: the coercion had to move before the fingerprint, not after. And he left two footnotes explicitly marked “neither a change request”.

His closing note on the second PR was short: “Nice touch putting EPISODES_VIEW_STATUS_COLUMN in one place.” He noticed the small thing that prevents the next bug. Same-day merges are not luck. They happen when a report ships repro output, a pinned commit, blast radius, and a fix shape the maintainer only has to ratify. I filed the first report that way; the co-founder filed the second, with the same rigor, and I offered both PRs. The review was a check, not a rewrite.

Three things to steal