Less flaky nix build

Don't rely on cabal2nix at build time.
1 job for master in 1 minute and 26 seconds