I’d love for a way to write nix expressions that easily lead to reproducible builds. Of course this is possible right now next, but requires substantial legwork to pin Nixpkgs version or add SHA for every tarball. Especially now as the machine learning story for NixOS is becoming strong, it would be awesome to have fully reproducible model training using such a system. This highlights a difference from docker or nvidia-docker, which do not control driver version.
Fortunately, `builtins.fetchTarball` makes this easier. I do this in my `nixcfg` repo so that I can build my entire machine config (patches and enabling Iris in Mesa, and all) on a stupidly-cheap Packet.net VM: https://github.com/colemickens/nixcfg/blob/master/default.ni....
If you follow the rabbit-hole: `default.nix` -> `lib.nix`, etc, you can see that I pin nixpkgs, have an update script that updates the nixpkgs revs I build against, it supports building against a custom, local `~/code/nixpkgs` if it exists, and I have my machine config abstracted out to where I can build a "GNOME instance of my machine" or by default, my machine with Sway and sway related packages installed. Much of this should be easier with flakes, as I understand it.
It's not the cleanest config, the README needs love, but maybe it can be inspiration in the meantime. :) My latest trick was figuring out how to get the new Mesa Intel Iris Gallium driver enabled without rebuilding the world, and I extracted it to what I call a "mixin" that anyone can copy and just use: https://github.com/colemickens/nixcfg/blob/master/modules/mi...
(And technically I still use mozilla/nixpkgs-mozilla to pull the latest Firefox Nightly which is impure and thus not always perfectly, perfectly reproducable. I do however pin the overlays themselves also - like my nixpkgs-wayland overlay that packages HEAD versions of Sway and other Wayland related tools!)
This is interesting. Sounds like you do this on the system level? I’m interested in doing this on a shell or package level eg I’d love to have system on bleeding edge but be able to jump back to same binary blobs associated with a particular revision of software
`nix-env` can be coerced into pinning a package at a specific revision, effectively ignoring channel updates. However, yes, I do all of my nixos package management in my system configuration and try to avoid `nix-env` at all costs as it leads to drift. However, in at least one case, this is largely un-avoidable (in the case of OBS Plugins, there's no NixOS infra to make sure that 'wlrobs' is available at a well-known spot for OBS to pick-up. We'd need to make a plugin-aware wrapper, or `programs.obs` module probably. This is because nix-env does some additional symlinking that isn't done for system-installed packages).
As colemickens mentioned, fetchTarball works pretty well. I have an expression that fetches a specific version of nixpkgs, imports it, and instantiates it with an overlay. That lets me override certain packages to customize them or use a different version than nixpkgs provides, and also to add my own packages.
I periodically update the version of nixpkgs it uses, but since it's pinned, I get exactly the same outputs when building.
Ah do you use fetchTarball for nixpkgs? Although maybe I’m not understanding how you’re using it, could you share an example? Interested in your workflow!
This isn't even simplified. I just cut out the contents of the overlay. For more on how overlays work take a look at these references; they were really helpful to get me started.
Is model training reproducible with identical software? I was under the impression that a number of CUDA expressions were non-deterministic, especially in the least-significant bits. And that's if you're running on the same hardware models, there's more variation between models.
That said, it’s hard enough to even run an old model 2 years later let alone reproduce to significant digits. I’d be happy enough to solve the former and get “close enough” training results
