I had some free time this weekend and I've spent some of it trying to learn Go since mlmym seems to be unmaintained and I'd like to try to fix some issues in it. I ran into a stumbling block that took a while to solve and which I had trouble finding relevant search results for. I've got it solved now, but felt like writing this up in case it helps anyone else out.
When running most go commands I tried (e.g. go mod init example/hello or go run hello.go or even something as seemingly innocuous as go doc cmd/compile when a go.mod file exists) the command would hang for a rather long time. In most cases, that was about 20~30 seconds, but in one case -- trying to get it to output the docs about the compile tool -- it took 1 minute and 15 seconds! This was on a relatively fresh Linux Mint install on old, but fairly decent hardware using golang-1.23 (installed from apt).
After the long wait, it would print out go: RLock go.mod: no locks available -- and might or might not do anything else depending on the command. (I did get documentation out after the 1min+ wait, for example.)
Now, there's no good reason I could think of why printing out some documentation or running Hello World should take that long, so I tried looking at what was going on with strace --relative-timestamps go run hello.go > trace.txt 2>&1 and found this in the output file:
0.000045 flock(3, LOCK_SH) = -1 ENOLCK (No locks available)
25.059805 clock_gettime(CLOCK_MONOTONIC, {tv_sec=3691, tv_nsec=443533733}) = 0
It was hanging on flock for 25 seconds (before calling clock_gettime).
The directory I was running in was from an NFS mount which was using NFSv3 unintentionally. File locking does not work on NFSv3 out of the box. In my case, changing the configuration to allow it to use NFSv4 was the fix I needed. After making the change a clean Hello World build takes ~5 seconds -- and a fraction of a second with cache.
After solving it, I've found out that there are some issues related to this open already (with a different error message -- cmd/go: "RLock …: Function not implemented") and a reply on an old StackOverflow about a similiar issue from one of the developers encouraging people to file a new issue if they can't find a workaround (like I did). For future reference, those links are:
I was concerned, until I saw you mention NFS.
For sure, Go should play nicely with NFS, but I'm my 25+ years of experience with it, it's best for browsing but painful for working in. Like, I avoid cd'ing into NFS shares because I don't want to be trapped in them when they inevitably lock up while they try to figure out what time everyone agrees to. I've had vi lock up because there was a minor network glitch that threw NFS all out of wack, and lost documents.
If you need to use NFS, then I'm sorry for you. NFS was once the only game in town, but there are far better options now. I'm not surprised that compiling on NFS, especially in an environment that heavily used caching, exhibited divergent behavior.
(Heavily edited because apparently I was having a stroke when I first wrote this)
I've got a similar length of experience and plenty of my own horror stories about NFS... but yeah, it's the most effective way to get what I need done at the moment. (When I outgrow my current setup though I'll wrestle the tentacle monster and replace it with Ceph... probably.)
Honestly, this is a topic I've spent hours on.
bcachefs has this mode I'm pretty excited about - when it's to a state I fell like I can trust it enough to use it - where it has replication levels. Like, you can RAID1 an SSD and (say) an sshfs mount, and configure bcachefs to use the SSD and only opportunistically replicate to the sshfs mount. It means the sshfs is only eventually consistent, but it's still a fantastic feature I have always wanted. Note that I don't know that you can literally use a remote network mounted share; it might be limited to devices. The examples I've seen usually talk about things like mixing M.2 and hard drives, or M.2 and SCSI, or IDE and a USB drive - devices with different speeds, but all physical media. Still, I intend to stress test that aspect because it's something I've wanted for years.
I've tried so many... so... many network filesystems. They all suck in some way. Many, because they insist on owning the storage partition, or file, such that data isn't accessible without spinning up the service. Most because they require a horrible amount of setup, server software, and general PITA-edness. There's a network FS(?) called Upspin that talks about a design that sounds perfect, but actually running it is a terrible experience, and the instructions feel like it was designed to sell Google network storage services.
I don't know of a good solution; I've just had worse experiences with NFS. Even Samba/CIFS was more fault tolerant; it just lacked basic fill filesystem features, like file permissions.
Definitely agree with you on that! Distributed systems are just fundamentally challenging. Network filesystems try to hide some of that complexity, but it inevitably leaks through in one way or another...
I deal with Ceph at work. Thankfully I'm not the one in charge of its configuration, but I've seen enough of the headaches the admin had to go through to get it working right... Once we got it though, it's been working pretty damned well -- but you basically need a full-time sysadmin (or will become one yourself...) when you're dealing with the kind of scale we've got. (My home needs are not quite as insane though; mere dozens of TB instead of a dozen or so PB...)
SeaweedFS is another I've got in the back of my mind if I ever outgrown my NAS, but I haven't worked with it personally.
I spent some time looking at all of these again and think I'm going to try XtremeFS next. It doesn't check all of the boxes, but it seems simple enough. The alternative is a LAN-only IPFS; one of the things I'm trying for is a controllable local cache, like, a size-limited LRU.