Every ML team eventually hits the same moment. Someone asks, “Can you retrain the model from last quarter?” and nobody can say exactly what went into it. The notebook has changed, the dataset has been overwritten, and the library versions have moved on. Reproducibility isn’t an academic virtue here; it is what lets you debug regressions, answer audit questions and roll back with confidence.

Decide What “Reproducible” Means for You

There are two levels, and mixing them up causes needless frustration.

• Re-runnable: you can repeat the exact procedure, with the same inputs, and get a model of equivalent quality.
• Bit-identical: you get exactly the same weights. This is often impractical on GPUs, because some operations are non-deterministic for speed, and results can differ across hardware or library versions.

For most teams, re-runnable is the right target. Aim for the same procedure and the same inputs, and accept small numeric differences. Reserve strict determinism for cases that truly need it, and expect a speed cost when you enable it. Check your framework’s current documentation for what it supports.

The Four Things to Pin

A run is a function of four inputs. If you can’t recover all four, you can’t reproduce it.

Code

Record the exact commit of the training code. Training from a dirty working directory is the quiet killer: results depend on edits that were never committed. A simple guard is to have the training script refuse to start (or at least flag the run) if there are uncommitted changes.

Data

This is the hardest to pin. Overwriting a dataset in place destroys history. Options include immutable, timestamped snapshots; content hashes of the files; or a data versioning tool that records which version fed which run. Whatever you choose, store an identifier for the data in the run record. Also record how the data was split, including the seed or the explicit list of validation items, so you don’t evaluate on different samples next time.

Environment

Pin library versions in a lock file and, ideally, build a container image with a fixed tag or digest. Capture the language runtime version too. For GPU work, note the driver and CUDA-related versions your stack depends on, and the hardware type, since behavior can differ.

Configuration

Hyperparameters, preprocessing options and flags should live in a config file that is saved with the run, not as ad hoc command-line arguments that vanish with your terminal history.

Handle Randomness Deliberately

Randomness enters through weight initialization, data shuffling, augmentation and dropout, among others. Set seeds for every random number generator your stack uses and log the values. Be aware that seeding alone does not guarantee identical results on GPUs, because certain operations are non-deterministic unless you ask for deterministic variants.

A more useful habit than chasing determinism: run an important configuration with a few different seeds and look at the spread. If results vary a lot between seeds, a single “improvement” of the same size is just noise, and you’ve learned something valuable about your evaluation.

Make the Run Record Automatic

Humans forget; scripts don’t. Have each run write a record automatically, containing at minimum:

• Code commit and whether the tree was clean.
• Data identifier and split definition.
• Environment identifier (image tag or lock file hash).
• Full configuration.
• Seeds.
• Hardware used.
• Metrics and the location of the output artifacts.

Any experiment tracking tool can hold this, or even a JSON file stored beside the model. The tool matters less than the discipline of writing it without manual steps.

A Hypothetical Example

A team ships a model, and weeks later quality dips. They want to retrain from the previous good version to compare. The old run record shows the code commit and config, but the data field says only “latest.” The dataset has since been regenerated with a changed cleaning step, so the original can’t be recovered.

After this, they change two habits. Datasets become immutable snapshots with an ID that every run records, and training refuses to launch with uncommitted code. The next time a regression appears, they re-run the old recipe against the old snapshot in a day and can compare directly, which isolates the cause to a change in data cleaning.

A Reproducibility Checklist

Item	Question to answer	Typical way to capture
Code	Which exact version trained this?	Commit hash; block dirty trees
Data	Which exact data and split?	Immutable snapshot ID or content hash
Environment	Which libraries and runtime?	Lock file plus fixed image tag
Config	Which settings?	Config file saved with the run
Randomness	Which seeds, which determinism flags?	Logged values
Hardware	Which accelerator type?	Recorded in run metadata
Outputs	Where are the artifacts?	Stored with a link to the run

Don’t Let Perfect Block Good

Teams stall because full reproducibility sounds like a big project. It doesn’t have to be. Start with the cheapest high-value step: log the code commit, data identifier and config for every run automatically. Add environment pinning next, then determinism if you need it. Each step pays off independently.

Frequently Asked Questions

Do I need to version large datasets in Git? No. Use object storage with immutable snapshots, or a data versioning tool, and keep only identifiers or small manifests in Git.

Why do my results differ across GPU types? Numeric behavior, kernel choices and library paths can differ between hardware, so tiny differences are normal. Record hardware to explain them.

Is setting a random seed enough? It helps but isn’t sufficient on GPUs. You may also need deterministic-mode settings, at some cost to speed.

How long should I keep old snapshots and runs? Keep those tied to models in production or subject to audit requirements. Check your organization’s retention rules.

What about notebooks? Fine for exploration, but move anything that produces a model you’ll ship into versioned scripts so the run is repeatable.

Next Step

Add three fields to your next training run’s output automatically: code commit, data identifier and config. If any of them is hard to fill in, you’ve found the first thing to fix.