cargo update hangs forever

[mhristache]

I update to latest nightly and I have issues building my project. It seems cargo update hangs when updating Cargo.lock. Here is the output of strace after I manually removed the existing Cargo.lock:

...
madvise(0x7f4e9cb73000, 28672, MADV_DONTNEED) = 0
open("/cs/max/tools/test/Cargo.lock", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/cs/max/tools/test/Cargo.lock", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 3
ioctl(3, FIOCLEX)                       = 0
flock(3, LOCK_EX|LOCK_NB 

In hangs forever after the flock call.

Versions:

cargo 0.11.0-nightly (867627c 2016-04-21)
rustc 1.10.0-nightly (b5ba5923f 2016-04-21)

Details about the system where I am trying this:

Linux <removed> 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.4 (Santiago)

[alexcrichton]

Cargo should in theory print out what it's about to wait for before it does so, could you gist the full output of Cargo? Also, do you have a project I could reproduce this against?

[mhristache]

I can reproduce it even with a new project with no dependencies, e.g.:

cargo new test_proj
cd test_proj
cargo update

Running with debug does not give much more:

# RUST_LOG=trace cargo update
DEBUG:cargo::update: executing; cmd=cargo-update; args=["cargo", "update"]
TRACE:cargo::ops::cargo_read_manifest: read_package; path=/cs/max/tools/test_proj/Cargo.toml; source-id=file:///cs/max/tools/test_proj

Is there some other way to get more debug output?

[alexcrichton]

Hm this seems pretty worrisome... The strace logs you've pasted show Cargo hanging at

flock(3, LOCK_EX|LOCK_NB 

However LOCK_NB should indicate that the call should not block...

Could you gist the full strace logs as well as uname -a?

[mhristache]

Might be some file system issue on my sytem. FWIW, it's an NFS mount. I will re-run it on a local partition and come back with the result.

[mhristache]

I cannot reproduce the issue when using a local partition so most likely it's an issue with the NFS mount I am using.

The strange thing is that I can create and update files on that NFS partition.

Here is the strace gist when running cargo update on the NFS partition: https://gist.github.com/maximih/224349198fd401ec597170d08310bd6a

[alexcrichton]

This is... interesting! It appears that flock is not only working over NFS, it's blocking indefinitely over NFS...

Do you know if your NFS setup is "weird" in any way? e.g. using somewhat nonstandard software or anything like that? If not, what's your setup look like? Perhaps I can search around to see if there are open bugs or workarounds on this elsewhere?

[mhristache]

I checked my server and it's using NFSv3 which does not support locking.

From the NFS docs:

NFS Versions 2 and 3 are stateless protocols, but NFS Version 4 introduces state. An NFS Version 4 client uses state to notify an NFS Version 4 server of its intentions on a file: locking, reading, writing, and so on. An NFS Version 4 server can return information to a client about what other clients have intentions on a file to allow a client to cache file data more aggressively via delegation. To help keep state consistent, more sophisticated client and server reboot recovery mechanisms are built in to the NFS Version 4 protocol.

NFS Version 4 introduces support for byte-range locking and share reservation. Locking in NFS Version 4 is lease-based, so an NFS Version 4 client must maintain contact with an NFS Version 4 server to continue extending its open and lock leases.

I found a similar issue reported on rubygems: ruby/rubygems#806

If I understand correctly, rubygems stopped using flock when an NFS file system is detected.

[alexcrichton]

Yeah that workaround seems reasonable, but it looks like it may not work in this case? It seems that in Ruby the flock call was raising the ENOLCK exception, but here it's just blocking indefinitely (never returning). Cargo can't even see if you're using NFS to fall back to not using locks :(.

I wonder though if this is related to the LOCK_NB flag? Would you be able to test out a few cases locally to see if when you omit the LOCK_NB flag if flock returns with an error?

[mhristache]

I will try to find some time tomorrow. I will be back :)

[mhristache]

I tried omitting the LOCK_NB flag but flock still hangs:

write(2, "Locking the file\n", 17Locking the file
)      = 17
flock(3, LOCK_EX

[mhristache]

What about using statfs system call to check if the file is on an NFS partition?

Something like this:

#include <sys/statfs.h>
#include <stdio.h>

int main()
{
    struct statfs my_file;
    statfs ("Cargo.lock", &my_file);

    if (my_file.f_type == 0x6969) {
        printf("NFS file system\n");
    }
}

[alexcrichton]

That'd do it! I'd be totally game for that.