Running out of memory doing: nix-store --add /tmp/../stack2nix-output.nix

[commandodev]

nix-store --add /tmp/stack2nix-output-dir.SHVZvXZmyE/stack2nix-output.nix
warning: dumping very large path (> 256 MiB); this may run out of memory

I'm pulling in an external package via a git entry in extra-deps. This was working when the thing I'm building was in a directory inside the thing I'm pulling in...

[nh2]

Hey Ben, can you check in /tmp/stack2nix-output-dir.SHVZvXZmyE/stack2nix-output.nix, for the package you want to build, is there src = ./. (which would be wrong)?

If yes, that will be the reason: src = ./. will be src = /tmp/ in that case, so it'll try to pull your entire /tmp into the nix store, thus running out of memory if it is large.

I have yesterday found one such case as well.

In a correctly generated file, src = should point to an absolute directory containing the Haskell proect you want to build statically.

[commandodev]

mkDerivation {
           pname = "rapid-example";
           version = "0.1.0.0";
           src = ././.;

That will do it :-)

But I also have:

  # Fixes we want to make to any Haskell packages go in here.
  static_haskellPackages_with_fixes = static_haskellPackages.override (old: {
    overrides = pkgs.lib.composeExtensions (old.overrides or (_: _: {})) (self: super: {

      rapid = setCleanHaskellSrc super.rapid;
      "${pkgName}" = setCleanHaskellSrc super."${pkgName}";

    });
  });

Where pkgName should be set to rapid-example

[commandodev]

I'm calling this https://github.com/cdodev/rapid/blob/master/default.nix

from this file in my example (which I'll push shortly)

{
  sources ? import ./nix/sources.nix,
  stack2nix-output-path ? "rapid-output.nix"
}:
# with import sources.nixpkgs {};
let
  rapid = import sources.rapid {
            pkgName = "rapid-example";
            compiler = "ghc865";
            projectDir = toString "./.";
            inherit stack2nix-output-path;
          };

in
  {
    build_script = rapid.lambda_function_zip_script;
    inherit (rapid) static_package fullBuildScript;
  }

and I'm using this command: $(nix-build -A fullBuildScript)

[commandodev]

Ugh... I was passing the projectDir as a string. :-/

That seems to have fixed it

[nh2]

Ugh... I was passing the projectDir as a string. :-/

That seems to have fixed it

That makes sense!

[commandodev]

But:

Cloning c33f9c3403a84aad9c3b195cdb3cc334f1de56ae from https://github.com/cdodev/rapid.git
fatal: could not create work tree dir '/nix/store/19mx9kn9n8anvrabwhgrbjvpnw6mw0cv-rapid-example/.stack-work/downloaded/dP2JKZGDaHkw': Permission denied

Looks like I'm trying to build in a store path when I get past this point...

[commandodev]

Is there a reason why we have to use stack for all this? Could I use cabal2nix to get the package for rapid in the default.nix linked above?

[commandodev]

I tried using cabal2nix to provide rapid. No dice as stack2nix wants it when it's preparing the package set I guess. So I guess I'm stuck on being able to use git dependencies for now...

[nh2]

@commandodev On rapid commit c33f9c3403a84aad9c3b195cdb3cc334f1de56ae with this diff:

--- a/default.nix
+++ b/default.nix
@@ -2,8 +2,8 @@
 #
 #     $(nix-build --no-link -A fullBuildScript)
 {
-  pkgName,
+  pkgName ? "rapid",
-  projectDir ? toString "./.",
+  projectDir ? ./.,
   compiler ? "ghc865", # matching stack.yaml
   stack2nix-output-path ? "./rapid-output.nix"
 }:

it starts building for me (not finished yet so I cannot tell yet if it will be fully successful).

Is there a reason why we have to use stack for all this?

The use of stack2nix (which uses stack as a library) is for the case that you want to use the dependencies as declared by your resolver: lts-14.27. Otherwise you are bound to use nixpkgs's Haskell package set (it maintains only a single one, not multiple ones you can pick from).

[nh2]

OK, it built all the way to rapid which failed because of hpack:

configuring
rapid.cabal was modified manually, please use --force to overwrite.
builder for '/nix/store/5z66hsvq95kbqj3vrfqbhvh490x6saza-rapid-0.1.0.0.drv' failed with exit code 1

You removed the .cabal file from the repo in cdodev/rapid@c33f9c3 and kept only package.yaml, so the easiest way to generate a cabal file from that is to let stack2nix do it. For that in needs to be able to write into the directory where the package.yaml is, so we cannot give it a /nix/store path. So instead, we give it --argstr projectDir $PWD:

$ $(nix-build --no-link default.nix -A fullBuildScript --argstr projectDir $PWD)
... build output
/nix/store/z7j8w0k8mbc9fzi5zvdgsbasixm3gaav-rapid-0.1.0.0

So that builds.

It doesn't currently build a static exe though because that's commented out in your package.yaml:

https://github.com/cdodev/rapid/blob/c33f9c3403a84aad9c3b195cdb3cc334f1de56ae/package.yaml#L92-L101

[commandodev]

Thanks Niklas!

I don't think I was clear before. I'm actually trying to run this from the default.nix in another package called rapid-example. I've just pushed that here:

https://github.com/cdodev/rapid-example/blob/master/default.nix

The idea is that it's calling rapid's default.nix to build itself (so that I can share an example that just calls the nix rather than having to contain it).

The error here seems to be when stack is trying to clone rapid:

rapid-example$ $(nix-build -A fullBuildScript --argstr projectDir $PWD)
these derivations will be built:
  /nix/store/492z8z07nvjprcrvgswnvmhnwlc8a36w-stack2nix-build-script.sh.drv
  /nix/store/gwjn9cfk9cc44lblqrcwh6qgzb7i94h3-stack2nix-and-build-script.sh.drv
building '/nix/store/492z8z07nvjprcrvgswnvmhnwlc8a36w-stack2nix-build-script.sh.drv'...
building '/nix/store/gwjn9cfk9cc44lblqrcwh6qgzb7i94h3-stack2nix-and-build-script.sh.drv'...
+ /nix/store/4jl99687r2r5xdxis5g93v7cy5rmzrjb-stack2nix-0.2.3/bin/stack2nix /nix/store/jmcrxlbnd9dp95ykq7pjk2lj3a86zxvk-rapid-example --stack-yaml stack.yaml --hackage-snapshot 2020-02-15T00:00:00Z -o /tmp/stack2nix-output-dir.FgzF1kN13m/stack2nix-output.nix

Ensuring git version is >= 2 ...
Ensuring cabal version is >= 2 ...

Cloning c33f9c3403a84aad9c3b195cdb3cc334f1de56ae from https://github.com/cdodev/rapid.git
fatal: could not create work tree dir '/nix/store/jmcrxlbnd9dp95ykq7pjk2lj3a86zxvk-rapid-example/.stack-work/downloaded/dP2JKZGDaHkw': Permission denied
stack2nix: Received ExitFailure 128 when running
Raw command: /home/ben/.nix-profile/bin/git clone --recursive https://github.com/cdodev/rapid.git /nix/store/jmcrxlbnd9dp95ykq7pjk2lj3a86zxvk-rapid-example/.stack-work/downloaded/dP2JKZGDaHkw
Run from: /nix/store/jmcrxlbnd9dp95ykq7pjk2lj3a86zxvk-rapid-example/.stack-work/downloaded/

edit

stack2nix /nix/store/jmcrxlbnd9dp95ykq7pjk2lj3a86zxvk-rapid-example seems to be the problem. Somehow projectDir isn't being passed on.

[commandodev]

fixed by adding projectDir to the default.nix in example, but:

error: while evaluating the attribute 'propagatedBuildInputs' of the derivation 'rapid-example-0.1.0.0' at /nix/store/gnwhwp0ygya7maw0pqcvyqzkscb1h478-source/pkgs/development/haskell-modules/generic-builder.nix:291:3:
while evaluating 'getOutput' at /nix/store/gnwhwp0ygya7maw0pqcvyqzkscb1h478-source/lib/attrsets.nix:464:23, called from undefined position:
while evaluating anonymous function at /nix/store/gnwhwp0ygya7maw0pqcvyqzkscb1h478-source/pkgs/stdenv/generic/make-derivation.nix:156:17, called from undefined position:
attribute 'callCabal2Nix' missing, at /home/ben/docs/1.Projects/Rapid/rapid/default.nix:86:15

[commandodev]

Boom!

Working:

rapid-example$ ldd build/bootstrap 
	not a dynamic executable

Thanks for the help @nh2!

[nh2]

👍

[commandodev]

Happy New Year @nh2. Thanks again for the help. Looking to release rapid after a bit of a tidy up and getting CI set up :-)