I’m kinda surprised someone hasn’t integrated the buildbarn nfs v4 stuff into docker/podman - the virtiofs stuff is pretty bad on osx and the buildbarn nfs 4.0 stuff is a big improvement over nfs v3.
Anyhow I digress. Can’t wait to take it for a spin.
[0] https://github.com/Barre/zerofs_nfsserve
[1] https://github.com/buildbarn/bb-remote-execution/tree/master...
I'm not too sure about NFSv4 for ZeroFS specifically, the benefits don't seem _that_ obvious to me, especially compared to something like 9P, where server-side locking in the ".L" version is native anyway. Especially considering that most ZeroFS users are mounting the fs on the same box as the server is running and so local locks are working.
MacOS is a bit of a blind spot for 9P though, and a client there would be great.
What is the durability model? The docs don't talk about intermediate storage. Slatedb does confirm writes to S3 by default, but I assume that's not happening?
edit: A quirk with the use of NFSv3 here is that there's no specific close op. So, if I understand right, ZeroFS' "close-to-open consistency" doesn't imply durability on close (and can't unless every NFS op is durable before returning), only on fsync. Whereas EFS and (I think?) azure files do have this property.
I'd recommend using 9P when that matters, which has proper semantics there. One property of ZeroFS is that any file you fsync actually syncs everything else too.
I am in no way affiliated with JuiceFS, but I have done a lot of benchmarking and testing of it, and the numbers claimed here for JuiceFS are suspicious (basically 5 ops/second with everything mostly broken).
JuiceFS also supports multiple concurrent clients making their own connection to the metadata and object storage, allowing near instant synchronization and better performance, where this seems to rely on a single service having a connection and everyone connecting through it with no support for clustering.
Using a bunch of clients on any system hides the latency profile. You could even get your "thousands of operations per second" on a system where any operation takes 10 seconds to complete.
This is a JuiceFS setup with 10TB of data, where JuiceFS was specifically chosen because of its minimal latency compared to raw object storage.
Again, I don't see any problems with "thousands of operations a second" if your single client is issuing parallel requests. Try any sequential workload instead.
> because of its minimal latency compared to raw object storage
That's not how it works. JuiceFS hits the object store for every file operation (except metadata, because it's backed by an external database). Latency cannot be lower, especially as JuiceFS doesn't seem to be implementing any buffering.
Above you also claim "allowing near instant synchronization and better performance", but near instant synchronization of what exactly? Metadata? Data blocks? And under what consistency model?
If you wish to run the benchmarks in bench/ on your setup and show better numbers, I'd happily update the JuiceFS benchmarks.
The data and method can be found here: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021
Every software's perf is optimized for a usage pattern and maybe TFA's benchmark or torrenting isn't it, but i was certainly disappointed in the performance i found on 12 core / 144GB / NVME (for metadata) / 6x8TB HDD (for data). In the end we didn't move to S3 we stayed with good old ZFS storage.
I'd be curious to reproduce my benchmarks with ZeroFS when i find the time.
JuiceFS maps operations 1:1 to s3 chunk wise, so the performance profile is entirely unsurprising to me, even untaring an archive is not going to be a pleasant experience.
Looking at those benchmarks, I think you must be using a local disk to sync writes before uploading to S3?
Questions:
- I see there is a diagram running multiple Postgres nodes backed by this store, very similar to horizontally distributed web-server. Doesn't Postgres use WAL replication? Or is it disabled and they are they running on same "views" of the filesystem?
- What does this mean for services that handle geo-distribution on app layer? e.g. CockroachDB?
Sorry if this sounds dumb.
See: https://learn.microsoft.com/en-us/azure/storage/blobs/storag...
Are we prevented from implementing features when the platform itself provides adjacent tech?
Page blobs are 2x+ more expensive ($0.045/GB vs $0.02/GB) and would require Azure-specific code. We don't dream of maintaining separate implementations for each cloud provider.
The approach works identically on S3, GCS, Azure, and local storage. If anything, ZeroFS helps you distance from vendor lock-in, especially if you start to run "usually managed" components on top, say databases.
Speaking of Azure's "native" solutions - we benchmarked ZeroFS against Azure Files. ZeroFS is 35-41x faster for most operations and 38% cheaper. If Azure Files performs that poorly, I don't hold much hope for page blobs either: https://www.zerofs.net/zerofs-vs-azure-files
Note that Azure Files is a significantly different offering than block storage, it is essentially a Windows file server cluster and uses the SMB v3.1.1 protocol. It's slow, but not that slow.
It looks like you benchmarked against the HDD version of it, which few people use. Everyone who cares about its performance uses Azure Files Premium, which is the pure-SSD version of the service. They recently updated it to have flexible performance that can be cheaper than the older HDD storage: https://techcommunity.microsoft.com/blog/azurestorageblog/lo...
You're also mounting it from Linux, which is just... weird. Nobody (for some values of nobody) does this. The sole purpose for this service is to provide backwards-compatible file shares for Windows servers and desktops.
Benchmarking cloud services can be tricky because they have odd quirks such as IOPS-per-GB ratios, so nearly empty storage can be much slower than you'd expect!
That's not what Azure's marketing material says:
Serverless file shares
"Store your files on a distributed file system built from the ground up to be highly available and durable."
"Take advantage of fully managed file shares mounted concurrently by cloud or on-premises deployments of Windows, Linux, and macOS."
"Use integration with Azure Kubernetes Service (AKS) for easy cloud file storage and management of your data using NFS or SMB file shares."
Azure explicitly markets this for Linux and Kubernetes. If it performs terribly on Linux, that's Azure's problem, not mine.
https://azure.microsoft.com/en-us/products/storage/files
> Benchmarking cloud services can be tricky because they have odd quirks such as IOPS-per-GB ratios, so nearly empty storage can be much slower than you'd expect!
These managed services charge premium prices - they should deliver premium performance. At those prices, "quirks" and "you need to fill it up first" aren't acceptable excuses.
I've had similar issues with Amazon's equivalent too, where I tried to use EFS for a tiny website and I got something like 0.1 IOPS!
I do like the design of ZeroFS, and it irks me that competing solutions make local caching and buffering weirdly difficult. For example, Azure Premium SSD v2 disks no longer support local SSD caches! It's also annoyingly hard to combine the local cache disks you do get with remote disks using Windows Server, it insists on trying to detect the "role" of each disk instead of just letting admins specify that.
Using something like ZeroFS could be a nice trick for getting decent price/performance on public cloud platforms. My only concern is integrity and durability: what testing have you done to validate that it doesn't lose data even in corner-cases?
There's always going to be uncertainty with software that young, but the CI is pretty expensive and we didn't yet receive corruption-related reports. https://github.com/Barre/ZeroFS/tree/main/.github/workflows
It also appears to support a great deal of IO and permission complexity, which makes sense as it refers to itself as a filesystem. It also purports to be extremely fast. In the end it looks to function like an actual filesystem with all that entails.
rclone (my first time hearing about it) appears to be a much simpler solution with different design goals. It looks more like rsync in most instances, or a user-space link to a remote filesystem in others (like ftp or smbclient). The mount capability, while similar to zerofs, looks more like something much simpler and targeted for less-sophisticated needs.
Not saying one's better than the other; merely that they really look to be serving different needs/goals while having some overlap.
Incredible performance figures, rocketing to probably the best way to use object storage in an fs like way. There's a whole series of comparisons, & they probably need a logarithmic scale given the scale of the lead slatedb has! https://www.zerofs.net/zerofs-vs-juicefs
Speaks 9p, NFS, or NBD. Some great demos of ZFS with l2arc caches giving a near local performance while having s3 persistence.
Totally what I was thinking of when in the Immich someone mentioned wanting a way to run it on cheap object storage. https://news.ycombinator.com/item?id=45169036
It still sucks that S3 is ~$20/mo/TB basically "in perpetuity", while random SSD drives are ~$80/TB and I'd feel comfortable effectively amortizing them out at ~$20/yr for local storage instead of $20/mo for S3. :-/
ZeroFS will work with any S3 implementation that supports conditional writes. Even something a bit premium-priced like CloudFlare R2 goes for around $15/TB and has no egress costs. If you want to go for as cheap as possible, I'd throw minio or similar on a few cheap Hetzner storage servers https://www.hetzner.com/dedicated-rootserver/matrix-sx/