i ran segmentation inference over a large batch of echo videos and got back a folder of .npy files and a metrics table.

the table said the model was doing fine. i had no idea whether that was true, because i had not looked at a single output.

this is about building the looking-at-it step into the pipeline instead of doing it by hand afterward.

why metrics aren’t enough

metrics tell you how much you’re wrong. they don’t tell you what kind of wrong.

a few failure modes that a Dice score will happily hide:

  • coordinates are fine but drawn in the wrong coordinate space
  • the mask is right and the boundary you export from it is not
  • the output is structurally valid nonsense (right shape, right dtype, meaningless values)
  • the run silently skipped files

that last one isn’t even a model problem. it’s a pipeline problem, and no metric computed over completed outputs will ever mention the ones that didn’t complete.

‼️ a metric can only score the outputs that exist, on the assumption they were produced correctly. QC is how you check that assumption

the artifact: one overlay per input

the rule i landed on: every inference run emits a human-viewable artifact per input, generated by the same script that produced the outputs.

not a separate QC script you run later. later never happens, and a separate script can drift from the real pipeline and QC something that isn’t what shipped.

so each output folder gets:

coords_raw.npy          # model-space boundary coords
coords_original.npy     # rescaled to the source video
mask_prob_original.npy  # segmentation probabilities
extracted_curves.json   # curves in original coords
qc_overlay.mp4          # <- the thing a human looks at
metadata.json           # run metadata

plus, at the top level of the run:

inference_manifest.csv  # one row per input: success / failure

the overlay is the original video with the predicted boundary drawn on top, per frame:

import cv2

# pts: (200, 2) predicted boundary points for this frame,
# already scaled into original video coordinates.
# first half is the top wall, second half is the bottom wall.
wall_1 = pts[:100].astype(int).reshape(-1, 1, 2)
wall_2 = pts[100:].astype(int).reshape(-1, 1, 2)

cv2.polylines(frame, [wall_1], False, (0, 255, 0),   2)  # green
cv2.polylines(frame, [wall_2], False, (255, 0, 255), 2)  # magenta

two things that made this actually useful:

  • distinct colors per structure. when something is wrong you need to know which contour is wrong, and a single color tells you nothing
  • draw on the source frame, not on a blank canvas. the whole point is seeing the prediction against the anatomy

triage, don’t “review”

“review the outputs” is not a task you can finish. sorting them into buckets is.

i spot-checked a handful and put each in a category:

meaning
✅tracks the structure correctly the whole way through
🟨mostly right, 1–2 bad frames
🟥over half the video on the wrong area
❓output is not interpretable at all

the ❓ bucket is the valuable one. 🟥 is a model that’s wrong. ❓ is usually a bug, and it’s a completely different investigation.

my ❓ cases rendered as a straight diagonal line across the frame. that’s not a bad segmentation, it’s not anatomy at all — it’s what you get when a contour’s points are spread over the frame more or less arbitrarily and you connect them in order.

when that showed up i went and inspected the raw arrays directly:

import numpy as np, json

c = np.load(f"{case}/coords_raw.npy")
print("shape:", c.shape)
print("min:", c.min(), "max:", c.max(), "mean:", c.mean())

m = json.load(open(f"{case}/metadata.json"))
print("coord space:", m["coordinate_space_interpretation"])
print("orig size:", m["original_width"], "x", m["original_height"])

min/max is the fast tell. if coords are supposed to be normalized 0–1 and you’re seeing numbers in the hundreds, or vice versa, you’ve found it without reading any model code.

which is why metadata.json exists: write down what the numbers mean, next to the numbers. an array of floats doesn’t say what space it’s in and you will not remember.

bug 1: the rescale that only scaled one point

model outputs boundary points in its own space. to draw them on the source video they get rescaled.

the coords came back as a flat array — 200 points, x and y interleaved, so a trailing axis of 400. and the rescale did this:

out[..., 0] *= orig_w   # 💀
out[..., 1] *= orig_h

on a flat trailing axis, [..., 0] and [..., 1] are not “all the x’s” and “all the y’s”. they’re the x and y of the first point. every other point went through unscaled.

fix is to reshape so the axis means what the code assumes it means:

pts = out.reshape(*out.shape[:-1], 200, 2)   # [..., 200, 2]
pts[..., 0] *= float(orig_w)
pts[..., 1] *= float(orig_h)

this ran clean. no exception, correct output shape, plausible dtype. every automated check passed. it is only visible as a picture, and as a picture it’s unmissable.

‼️ shape-correct is not value-correct. an assertion on .shape would not have caught this. a person glancing at one frame catches it in a second

bug 2: guessing the coordinate space

the script auto-detected which space the coords were in by looking at their magnitude — roughly, “if the max is below ~1.25 it’s normalized 0–1, otherwise it’s model pixels.”

works until it doesn’t. cases whose max landed just above the cutoff got classified as pixels when they were normalized, and rendered as garbage.

two fixes, and you want both:

  1. an explicit override flag so you can force the right answer without touching the heuristic: 
    python batch_inference.py --coord-space normalized_0_1
  2. record what it guessed in metadata.json, so when an overlay looks wrong you can check the guess instead of re-deriving it

the general lesson: a heuristic that picks how to interpret data is fine, but it has to be overridable and logged. a silent guess is a bug that only appears in the pictures.

QC that isn’t visual

eyeballing doesn’t scale past a sample, so some checks are structural. these are cheap and catch whole-run problems:

# did every input produce an output folder?
ls inference_output | wc -l

# did every one produce the artifact it should have?
find inference_output -name "qc_overlay.mp4" | wc -l

# is anything suspiciously small (= died partway)?
du -sh inference_output/* | sort -h | head -20

the size sort is my favorite. a case that crashed halfway leaves a folder that’s structurally present and much too small, so sorting by size floats every truncated run to the top without opening anything.

one gotcha: du reports allocated blocks, so the same data can report different totals on different filesystems. i spent real time “investigating” a size discrepancy between two copies of the same results that turned out to be nothing but block accounting. use du --apparent-size when you’re comparing across filesystems, and compare file counts before you trust byte totals.

QC artifacts are expensive

the honest tradeoff: this stuff is big. per case, the intermediate heatmap array alone ran to a couple of GB, and the overlay videos add up fast across a full run.

what i do now:

  • generate everything on the first pass
  • review and triage
  • then delete the artifacts that were only there to be looked at:
find inference_output -name "qc_overlay.mp4" -delete
  • keep the small stuff forever — metadata.json, the manifest, the coords. they’re tiny and they’re what lets you regenerate an overlay later without rerunning inference

and give any script that loops over a big list a --resume flag, so re-running to regenerate one case doesn’t redo the whole batch.

takeaways

  • the pipeline that produces the outputs should produce the picture of the outputs, in the same pass
  • write a metadata.json saying what space/units/shape everything is in
  • write a manifest — a run isn’t successful if you can’t show every input got processed
  • triage into buckets, and treat “uninterpretable” as a bug report, not a bad score
  • shape-correct is not value-correct, and no assertion i’d have thought to write would have caught either of the real bugs
  • structural checks (counts, sizes) scale where eyeballing doesn’t
  • delete the big artifacts after review, keep the small ones forever

honestly the whole post reduces to: look at the output. it took me a month of not doing that to find out why it matters.