BinaryProvider 0.5.9 breaks packages

[fingolfin]

As the title says. I see errors like this:

   Building Polymake → `~/.julia/packages/Polymake/wYSJS/deps/build.log`
┌ Error: Error building `Polymake`:
│ [ Info: MacOS(:x86_64, compiler_abi=CompilerABI(:gcc8, :cxx03))
│ [ Info: Downloading https://github.com/JuliaPackaging/Yggdrasil/releases/download/MPFR-v4.0.2-1/MPFR.v4.0.2.x86_64-apple-darwin14.tar.gz to /Users/mhorn/.julia/packages/Polymake/wYSJS/deps/usr/downloads/MPFR.v4.0.2.x86_64-apple-darwin14.tar.gz...
│ ERROR: LoadError: LoadError: LibraryProduct(nothing, ["libmpfr"], :libmpfr, "Prefix(/Users/mhorn/.julia/packages/Polymake/wYSJS/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at /Users/mhorn/.julia/packages/BinaryProvider/GeAtj/src/Products.jl:419
│  [3] top-level scope at /Users/mhorn/.julia/packages/Polymake/wYSJS/deps/build_MPFR.v4.0.2.jl:48
│  [4] include(::Module, ::String) at ./Base.jl:377
│  [5] include(::String) at ./loading.jl:1124
│  [6] top-level scope at /Users/mhorn/.julia/packages/Polymake/wYSJS/deps/build.jl:77
│  [7] eval at ./boot.jl:331 [inlined]
│  [8] evalfile(::String, ::Array{String,1}) at ./loading.jl:1120 (repeats 2 times)
│  [9] top-level scope at /Users/mhorn/.julia/packages/Polymake/wYSJS/deps/build.jl:96
│  [10] include(::String) at ./client.jl:439
│  [11] top-level scope at none:5
│ in expression starting at /Users/mhorn/.julia/packages/Polymake/wYSJS/deps/build_MPFR.v4.0.2.jl:48
│ in expression starting at /Users/mhorn/.julia/packages/Polymake/wYSJS/deps/build.jl:77

or this

   Building GAP → `~/Projekte/OSCAR/GAP.jl/deps/build.log`
[ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz to /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/downloads/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz...
[00:52:19] ######################################################################## 100.0%##O#- #    [00:52:20] ######################################################################## 100.0%
ERROR: LoadError: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at /Users/mhorn/.julia/packages/BinaryProvider/GeAtj/src/Products.jl:419
 [3] top-level scope at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build_Zlib.v1.2.11.jl:48
 [4] include(::Module, ::String) at ./Base.jl:377
 [5] include(::String) at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build.jl:43
 [6] top-level scope at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build.jl:43
 [7] eval at ./boot.jl:331 [inlined]
 [8] top-level scope at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build.jl:43
 [9] include(::String) at ./client.jl:439
 [10] top-level scope at none:5
in expression starting at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build_Zlib.v1.2.11.jl:48
in expression starting at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build.jl:36

And on the Julia slack, I saw other people report similar issues.

CC @fieker

[staticfloat]

Can you set verbose = true and do a manual Pkg.build()? I see you have your own hand-crafted build.jl that reads in the other build.jl files, so you'll need to evaluate that inside of the modules you're creating here: https://github.com/oscar-system/GAP.jl/blob/master/deps/build.jl#L41-L43

[fingolfin]

OK, I edited build_Zlib.v1.2.11.jl to hardcode verbose = true. Got this:

(@v1.4) pkg> build -v GAP
   Building GAP → `~/Projekte/OSCAR/GAP.jl/deps/build.log`
[ Info: Directory /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/lib does not exist!
[ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz to /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/downloads/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz...
[01:47:22] ######################################################################## 100.0%##O=#  #   [01:47:22] ######################################################################## 100.0%
[ Info: No hash cache found
[ Info: Calculated hash a28fb652f94f1b1548197a3b3c9a71ec56fa50bb7d39454c5e12320335d30934 for file /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/downloads/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz
[ Info: Installing /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/downloads/Zlib.v1.2.11.x86_64-apple-darwin14.tar.gz into /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr
[ Info: Found a valid dl path libz.1.2.11.dylib while looking for libz
[ Info: /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/lib/libz.1.2.11.dylib matches our search criteria of libz
[ Info: Found a valid dl path libz.1.dylib while looking for libz
[ Info: /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/lib/libz.1.dylib matches our search criteria of libz
[ Info: Found a valid dl path libz.dylib while looking for libz
[ Info: /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/lib/libz.dylib matches our search criteria of libz
[ Info: Could not locate libz inside /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/lib
ERROR: LoadError: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at /Users/mhorn/.julia/packages/BinaryProvider/GeAtj/src/Products.jl:419
 [3] top-level scope at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build_Zlib.v1.2.11.jl:48
 [4] include(::String) at ./client.jl:439
 [5] top-level scope at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build.jl:44
 [6] include(::String) at ./client.jl:439
 [7] top-level scope at none:5
in expression starting at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build_Zlib.v1.2.11.jl:48
in expression starting at /Users/mhorn/Projekte/OSCAR/GAP.jl/deps/build.jl:36
┌ Error: Error building `GAP`:
└ @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:892

BTW, similar error with a package not by us:

(@v1.4) pkg> add Libtask
  Resolving package versions...
  Installed Libtask ─ v0.4.0
   Updating `~/.julia/environments/v1.4/Project.toml`
  [6f1fad26] + Libtask v0.4.0
   Updating `~/.julia/environments/v1.4/Manifest.toml`
  [6f1fad26] + Libtask v0.4.0
   Building Libtask → `~/.julia/packages/Libtask/Q1ugc/deps/build.log`
┌ Error: Error building `Libtask`:
│ [ Info: Downloading https://github.com/JuliaBinaryWrappers/Libtask_jll.jl/releases/download/Libtask-v0.3.1+0/Libtask.v0.3.1.x86_64-apple-darwin14.tar.gz to /Users/mhorn/.julia/packages/Libtask/Q1ugc/deps/usr/downloads/Libtask.v0.3.1.x86_64-apple-darwin14.tar.gz...
│ ERROR: LoadError: LibraryProduct(nothing, ["libtask_v1_3"], :libtask_v1_3, "Prefix(/Users/mhorn/.julia/packages/Libtask/Q1ugc/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at /Users/mhorn/.julia/packages/BinaryProvider/GeAtj/src/Products.jl:419
│  [3] top-level scope at /Users/mhorn/.julia/packages/Libtask/Q1ugc/deps/build.jl:58
│  [4] include(::String) at ./client.jl:439
│  [5] top-level scope at none:5
│ in expression starting at /Users/mhorn/.julia/packages/Libtask/Q1ugc/deps/build.jl:58

[staticfloat]

What happens if you try to dlopen("/Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/lib/libz.dylib")?

[fingolfin]

julia> using Libdl

julia> dlopen("/Users/mhorn/Projekte/OSCAR/GAP.jl/deps/usr/lib/libz.dylib")
Ptr{Nothing} @0x00007fcb71cdf830

[fingolfin]

Here is another one I just run into, with yet another package:

   Building FFMPEG → `~/.julia/packages/FFMPEG/guN1x/deps/build.log`
┌ Error: Error building `FFMPEG`:
│ [ Info: Downloading https://github.com/JuliaBinaryWrappers/Bzip2_jll.jl/releases/download/Bzip2-v1.0.6+1/Bzip2.v1.0.6.x86_64-apple-darwin14.tar.gz to /Users/mhorn/.julia/packages/FFMPEG/guN1x/deps/usr/downloads/Bzip2.v1.0.6.x86_64-apple-darwin14.tar.gz...
│ ERROR: LoadError: LoadError: LibraryProduct(nothing, ["libbz2"], :libbzip2, "Prefix(/Users/mhorn/.julia/packages/FFMPEG/guN1x/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at /Users/mhorn/.julia/packages/BinaryProvider/GeAtj/src/Products.jl:419
│  [3] top-level scope at /Users/mhorn/.julia/packages/FFMPEG/guN1x/deps/build_Bzip2.v1.0.6.jl:48
│  [4] include(::Module, ::String) at ./Base.jl:377
│  [5] include(::String) at /Users/mhorn/.julia/packages/FFMPEG/guN1x/deps/build.jl:44
│  [6] top-level scope at /Users/mhorn/.julia/packages/FFMPEG/guN1x/deps/build.jl:45
│  [7] include(::String) at ./client.jl:439
│  [8] top-level scope at none:5
│ in expression starting at /Users/mhorn/.julia/packages/FFMPEG/guN1x/deps/build_Bzip2.v1.0.6.jl:48
│ in expression starting at /Users/mhorn/.julia/packages/FFMPEG/guN1x/deps/build.jl:38
[09:42:33] ######################################################################## 100.0%##O#- #    [09:42:34] ######################################################################## 100.0%
└ @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:892

[fingolfin]

It seems this issue only happens for me on macOS; I just tried it on a Linux server, and everything (?) seems fine so far

[fingolfin]

BTW, is it on purpose that the master branch is stuck at v0.5.5? The only branch containing 0.5.9 seems to be rollback, is that the official "development" branch then?!

[MagnusKoudahl]

Chiming in to report similar issues with Libtask on Arch Linux. Manually running the build file, ie julia build.jl allowed the package to build as normal. FFMPEG works out of the box

[giordano]

BTW, is it on purpose that the master branch is stuck at v0.5.5? The only branch containing 0.5.9 seems to be rollback, is that the official "development" branch then?!

History is a bit convoluted, there are two branches: master has the yet-to-be-released v0.6.0 which will require Julia v1.3, while rollback has the v0.5 series which supports also older versions of Julia.

Chiming in to report similar issues with Libtask on Arch Linux. Manually running the build file, ie julia build.jl allowed the package to build as normal. FFMPEG works out of the box

I also use Arch Linux, but I can't reproduce any of the issues reported. For example, Libtask:

(tmp) pkg> add Libtask
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
  Installed Libtask ─ v0.4.0
   Updating `/tmp/Project.toml`
  [6f1fad26] + Libtask v0.4.0
   Updating `/tmp/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.9
  [6f1fad26] + Libtask v0.4.0
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 
   Building Libtask → `~/.julia/packages/Libtask/Q1ugc/deps/build.log`

(tmp) pkg> build Libtask
   Building Libtask → `~/.julia/packages/Libtask/Q1ugc/deps/build.log`

(tmp) pkg>

[yufongpeng]

(@Strings) pkg> build
   Building PCRE2 → `C:\Users\sciph\.julia\packages\PCRE2\fIxtF\deps\build.log`
┌ Error: Error building `PCRE2`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libpcre2-8"], :libpcre2_8, "Prefix(C:\\Users\\sciph\\.julia\\packages\\PCRE2\\fIxtF\\deps\\usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at .\error.jl:33
│  [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at C:\Users\sciph\.julia\packages\BinaryProvider\GeAtj\src\Products.jl:419
│  [3] write_deps_file(::String, ::Array{LibraryProduct,1}) at C:\Users\sciph\.julia\packages\BinaryProvider\GeAtj\src\Products.jl:400
│  [4] top-level scope at C:\Users\sciph\.julia\packages\PCRE2\fIxtF\deps\build.jl:43
│  [5] include(::String) at .\client.jl:439
│  [6] top-level scope at none:5
│ in expression starting at C:\Users\sciph\.julia\packages\PCRE2\fIxtF\deps\build.jl:43
└ @ Pkg.Operations D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Operations.jl:892

I encounter the same issue, with Window10, and julia v1.4.1.

[fingolfin]

I've now also experienced this on a Gentoo machine:

(v1.4) pkg> add GAP
  Resolving package versions...
  Installed GAP ─ v0.4.0
   Updating `/var/autofs/nfs/cip/alggeom/horn/.julia/environments/v1.4/Project.toml`
  [c863536a] + GAP v0.4.0
   Updating `/var/autofs/nfs/cip/alggeom/horn/.julia/environments/v1.4/Manifest.toml`
  [c863536a] + GAP v0.4.0
   Building GAP → `~/.julia/packages/GAP/vvuPQ/deps/build.log`
┌ Error: Error building `GAP`:
│ [ Info: Downloading https://github.com/bicycle1885/ZlibBuilder/releases/download/v1.0.4/Zlib.v1.2.11.x86_64-linux-gnu.tar.gz to /users/cip/alggeom/horn/.julia/packages/GAP/vvuPQ/deps/usr/downloads/Zlib.v1.2.11.x86_64-linux-gnu.tar.gz...
│ ERROR: LoadError: LoadError: LibraryProduct(nothing, ["libz"], :libz, "Prefix(/users/cip/alggeom/horn/.julia/packages/GAP/vvuPQ/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] write_deps_file(::String, ::Array{LibraryProduct,1}; verbose::Bool, isolate::Bool) at /users/cip/alggeom/horn/.julia/packages/BinaryProvider/GeAtj/src/Products.jl:419
│  [3] top-level scope at /users/cip/alggeom/horn/.julia/packages/GAP/vvuPQ/deps/build_Zlib.v1.2.11.jl:48
│  [4] include(::Module, ::String) at ./Base.jl:377
│  [5] include(::String) at /users/cip/alggeom/horn/.julia/packages/GAP/vvuPQ/deps/build.jl:43
│  [6] top-level scope at /users/cip/alggeom/horn/.julia/packages/GAP/vvuPQ/deps/build.jl:43
│  [7] eval at ./boot.jl:331 [inlined]
│  [8] top-level scope at /users/cip/alggeom/horn/.julia/packages/GAP/vvuPQ/deps/build.jl:43
│  [9] include(::String) at ./client.jl:439
│  [10] top-level scope at none:5
│ in expression starting at /users/cip/alggeom/horn/.julia/packages/GAP/vvuPQ/deps/build_Zlib.v1.2.11.jl:48
│ in expression starting at /users/cip/alggeom/horn/.julia/packages/GAP/vvuPQ/deps/build.jl:36

The same (add GAP) worked fine on another machine (running under ArchLinux, not that I think it necessarily matters).

[giordano]

Can someone please try and debug the issue on the platforms where they can reproduce it? It's a bit hard for me to understand what's going wrong without being able to experience it.

[fingolfin]

I'd love to but I have no idea where to start? I guess the "satisfied" function needs to be scrutinized? I'll try tonight, but some guidance would be appreciated