Saber Sojudi
Back

Checking the data used to train AI

I built a Python tool that checks the images and cell labels in the public LIVECell dataset, then gives scientists a clear list of examples to review.

  • Python
  • pandas
  • scikit-learn
  • Data pipelines
  • Data quality
View on GitHub Hire Me

A real example

Review images
A real LIVECell microscope image with many visible cells. Only one cell has a supplied label, outlined in yellow. The one supplied cell label Only labelled cell

A172_Phase_A7_1_01d04h00m_3.tif1 supplied label · 704 × 520 px

Many cells, just one supplied label. The yellow outline marks it; the ring and pointer help you find it.

Supplied cell labels checked

3 splits

1,662,447

✓ 0 missing images in 5,387 image records

Review images See methods
Images audited

5,239

100% matched to a local file

Reused photos

30

5.3% of val train ∩ val

Sparse-label images

9

≤ 10 labels issue #21

Alternate versions

118

kept issue #14

Reported cases found

top 10

The nine upstream-reported sparse images, placed in each method’s first 10 of 3,067.

Validation reuse

5.3%
30 validation photos also in training, of569

The held-back test split shares 0 filenames with the other two; only validation scores are affected.

Reported cases recovered

9 / 9
9 recovered by the audit rule, of9

Reported upstream in issue #21; each one is present in this snapshot with ≤ 10 supplied labels.

Flagged images

Open image review
Images flagged by the audit, with their cell line, split, supplied label count and finding.

Two examples in detail

Huh7 microscope photo in the training group. 68 supplied training labels.
Training · 68 labels
The same Huh7 photo in the validation group. 83 supplied validation labels.
Validation · 83 labels

30 photos appear in both groups. The same photo has different versions of its labels. Familiar examples can make evaluation less reliable.

Explore this example

What I built

  • A repeatable data pipelineStreams the three COCO label files into DuckDB and a pandas feature table, with automated tests.
  • An honest method comparisonMachine learning did not beat a simple label count. The tests favour counting by design, and the results say so.
  • A report you can verifyEvery number here is copied from one hashed release run and checked again before publishing.

PythonpandasDuckDBscikit-learnpytest

Both findings were first reported on GitHub (#26, #21). My tool reproduces them from the raw files and tracks them with evidence.

Image review

Pick an image and show or hide its cell labels. Flagged images come first, then comparison images.

Select an example

Choose an example from the list above to see its image, evidence and recommended review action.

Search and filter all examples

Verified dataset snapshot · 17 October 2026

Loading the illustrated example selection…

Loading…

Page 1 of 1

Two examples in detail

Each one shows what was found, why it matters, what is uncertain, and what a person should do next.

Confirmed overlap

30 image filenames appear in both the learning and the checking groups

A fair check should use images the model has not already learned from.

A Huh7 cell image from the learning group.

Learning grouptraining split

Images the model learns fromResearchers call this the training split.
The same Huh7 cell image, from the checking group.

Checking groupvalidation split

Images used to check its progressResearchers call this the validation split.

These are the same source image. It appears in both groups, each time with a different version of its cell labels.

Why it matters

Think of checking a student with a question they already practised. Familiar examples can make the result look better than it would on new ones.

What needs attention

Keep each source image in one group for this evaluation protocol, and record how the groups were chosen. Preserve the original data so the change can be checked.

This overlap is called data leakage. It affects validation scores; the separate final test group had no overlapping filenames with either group. The dataset authors annotated some images twice and kept both versions (issue #14); here the two versions ended up in different groups. This project has not measured the effect on a model’s results.

View the evidence behind this example

Loading the audit record…

Download the 157 audit findings · Read the original upstream report

Does machine learning help?

Four ordering methods compared at the same review budget. Both tests are defined by label count, so they favour the simple counting rule by design.

Reported cases inside the budget

Review budget

Simulated label loss

simulation

ROC AUC when labels are removed on purpose; 0.5 is chance. Mean over seeds 7, 42 and 101.

Loading the measured comparison…

Known-case regression: the nine upstream-reported sparse-label images

How many of the nine reported cases each method placed inside the budget. Denominator is nine, not every mislabelled image in the dataset.

Simulated annotation loss: what each method does when labels are removed on purpose

Precision@K and ROC AUC measured on a deliberate structured removal scenario. These are simulation numbers, not real-world error precision.

Where each reported image ranked

Rank within the 3,067-row review pool at the default seed. Lower is a higher review priority.

The two pools, named explicitly

What won

count rule

What lost

Isolation Forest

How to read these numbers

    Evaluation summary · Known-case record · Model card

    Data and sources

    Every review-pool candidate, the evidence files, and how this dashboard was built.

    Full rankings

    Loading the complete ranking table…

    Loading…

    Full ranking table over the review pool.
    Page 1 of 1

    Download the evidence

    Technical details

    Sources and checks

    Annotation files
    3
    Image previews
    39
    Snapshot
    17 Oct 2026

    The report, rankings and image previews use the same recorded dataset snapshot.

    File list and full checksums
    Every verified file behind this site, with its size and SHA-256.

    Source annotation files

    The three COCO files, as recorded by the audit and matched by the ranking provenance.

    Review status, sampling and limitations

    Published limitations of the ranking and evaluation stages.

    The four published review criteria

      Model and method configuration

      Fit / hold-out split per cell line. The hold-out is a practical leakage proxy, not proof of biological independence.
      Plain-language glossary
      What this website contains, and what it deliberately does not