This is exactly right, and something that we think is particularly important for applications that care about data consistency. Often times, we see that customers want to be able to quickly hand off tasks from one instance to another which can be incredibly complex if you don't have guarantees that your new operations will be seen by the second instance!

The headline seems misleading, then.

rclone can work with AWS' different offerings, some of which at least partially address this: https://aws.amazon.com/blogs/aws/new-amazon-s3-express-one-z...

Thanks, this was my thought as well. I use and love rclone and it wasn't immediately clear what this offered above that

I suppose rclone doesn't provide byte range file locking? Running sqlite over rclone would be a disaster.

Wow, thanks for the nice note! No questions are silly, and I'll also note that we now have a docs site (https://docs.regattastorage.com) and feel free to email me (hleath [at] regattastorage.com) if I don't fully address your questions.

> If I had a local disk which was 10 GB, what happens when I try to contend with data in the 50 GB range (as in, more that could be cached locally?) Would I immediately see degradation, or thrashing, at the 10 GB mark?

We don't actually do caching on your instance's disk. Instead, data is cached in the Linux page cache (in memory) like a regular hard drive, and Regatta provides a durable, shared cache that automatically expands with the working set size of your application. For example, if you were trying to work with data in the 50 GiB range, Regatta would automatically cache all 50 GiB -- allowing you to access it with sub-millisecond latency.

> Does this only work in practice on AWS instances? As in, I could run it on a different cloud, but in practice we only really get fast speeds due to running everything within AWS?

For now, yes -- the speed is highly dependent on latency -- which is highly dependent on distance between your instance and Regatta. Today, we are only in AWS, but we are looking to launch in other clouds by the end of the year. Shoot me an email if there's somewhere specifically that you're interested in.

> I've always had trouble with FUSE in different kinds of docker environments. And it looks like you're using both FUSE and NFS mounts. How does all of that work?

There are a couple of different questions bundled together in this. Today, Regatta exposes an NFSv3 file system that you can mount. We are working on a new protocol which will be mounted via FUSE. However, in Docker environments, we also provide a CSI driver (for use with K8s) and a Docker volume plugin (for use with just Docker) that handles the mounting for you. We haven't released these publicly yet, so shoot me an email if you want early access.

> Is the idea that I could literally run Clickhouse or Postgres with a regatta volume as the backing store?

Yes, you should be able to run a database on Regatta.

> I have to ask - how do you think about open source here?

We are in the process of open sourcing all of the client code (CSI driver, mount helper, FUSE), but we don't have plans currently to open source the server code. We see the value of Regatta in managing the infrastructure so you don't have to, and if we release it via open-source, it would be difficult to run on your own.

> Can I mount on multiple servers? What are the limits there? (ie, a lambda function.)

Yes, you can mount on multiple servers simultaneously! We haven't specifically stress-tested the number of clients we support, but we should be good for O(100s) of mounts. Unfortunately, AWS locks down Lambda so we can't mount arbitrary file systems in that environment specifically.

> efs performance

Yes, the challenge here is specifically around the semantics of NFS itself and the latency of the EFS service. We think we have a path to solving both of these in the next month or two.

Yes, you should be in our target market. I don't think that I can give a cost estimate without having a good sense of what percentage of your data you're actively using at any given time, but we should absolutely support the performance numbers that you're talking about. I'd love to chat more in detail, feel free to send me a note at hleath [at] regattastorage.com.

Out of curiosity, why not go bare metal in a managed colocation? Is that for the geographic spread? Or unpredictable load?

Every few months of this spend is like buying a server

Edit: back at my pc and checked, relevant bare metal is ~$500/m, amortized:

https://baremetalsavings.com/c/LtxKMNj

Edit 2: for 100tb..

Great to hear from you, I think cunoFS is doing a lot of things right! It’s certainly a fun problem space!

Is that Gweo? Didn't know you were in the storage space, good to see you!

Hey -- I think this is something that's in-scope for our custom protocol that we're working on. I'd love to chat more about your needs to make sure that we build something that will work great for you. Would you mind shooting an email to hleath [at] regattastorage.com and we can chat more?

Wow, thanks for coming out! I hope that you're heartened to see the number of people who immediately think of JuiceFS when they see our launch. I totally agree with you, storage is such an interesting space to work in, and I'm excited that there are so many great products out there to fit the different needs of customers.

As someone who is already happily using JuuceFS, perhaps you can provide a short list of differences (conceptual and/or technical). Thanks for a great product.

I'm a happy and satisfied JuiceFS user here, so I too would be interested in the difference between these. Is the Regatta key point caching?

> That is, you aren't treating the object store as the source of truth. If your caching layer goes down after a write is ack'ed but before it's "replicated" to S3, people lose their data, right?

This is exactly why we're building our caching layer to be highly-durable, like S3 itself. We will make sure that the data in the cache is safe, even if servers go down. This is what gives us the confidence to respond to the client before the data is in S3. The big difference between the data living in our cache and the data living in S3 is cost and performance, not necessarily durability.

> I think you'll end up wanting to offer customers the ability to do strongly-consistent writes (and cache invalidation). You'll also likely end up wanting to add operator control for "oh and don't cache these, just pass through to the backing store" (e.g., some final output that isn't intended to get reused anytime soon).

I think this is exactly right. I think that storage systems are too often too hands off about the data (oh, give us the bytes and we will store them for you). I believe that there are gains to be had by asking the users to tell you more about what they're doing. If you have a directory which is only used to read files and a directory which is only used to write files, then you probably want to have different cache strategies for those directories? I believe we can deliver this with good enough UX for most people to use.

> Finally, don't sleep on NFSv4.1! It ticks a bunch of compliance boxes for various industries, and then they will pay you :). Supporting FUSE is great for folks who can do it, but you'd want them to start by just pointing their NFS client at you, then "upgrading" to FUSE for better performance.

I certainly don't, and this is why we are supporting NFSv3 right now. That's not going away any time soon. We want to offer something that's highly compatible with the industry at large today (NFS-based, we can talk specifics about whether or not that should be v3 or v4) and then something that is high-performance for the early adopters who can use something like FUSE. I think that both things are required to get the breadth of customers that we're looking for.

> It's pure magic when you mount a filesystem on clean machine and all your data is "just there."

I totally agree! I am hoping that Regatta can power a future where teams don't need more than ~8 GiB of local storage for their operating system, and can store the rest on something like Regatta to get rid of the waste of overprovisioned block volumes.

The funny thing about storage is that all of the problems are the same! Ultimately, there is no problem that cannot be solved with caching, journaling, write-ahead logging, etc. I think what makes the problem space so interesting is how a million different products can make a million different trade offs with these tools to deliver on their customer needs. File systems are awesome.

wow gmail drive is a walk down memory lane :) really is amazing how far we've come since then!

Yeah, that's exactly right. I had some... experiences with Spark recently, that convinced me that this is something that could really help. I also really like the idea that organizations can continue to use S3 as the source of truth for their data (as you mention, it means that you can continue to use Access Logs, which would capture all usage of your S3 bucket across your applications).

> Will the mounted system _really_ be accessible as local fs and seen as such to all running processes? No surprises? No need for python specific filesystem like S3Fs?

Ha, well it depends on what you mean by surprises. We won't have a Python-specific file system. Our client is going to come in two flavors. Today, you can mount Regatta over NFSv3 (which we wrap in TLS to make it secure). This works for some workloads, but doesn't provide like-for-like performance with EBS. Over the next month, we plan to release the "custom protocol" that I wrote about above, that we expect to send to customers in the form of a FUSE file system.

Either way, it should be one package, you shouldn't need to worry about versioning, and it will appear as a real, local file system. :D

I think this is a great insight, and something that I think about often. The challenge that I see is that the scientist archetype (whether it's data science, AI researcher, or anything else) isn't really interested in doing software development for these kinds of things. They just want the data to be there, and it's super nice to be able to click through the S3 console to be able to see and share the data their using. I think that what you're doing is a great idea for folks who are accessing their data primarily through Python programs!

Yes! This is my expectation. Lots of the big companies have already done this with in-house architecture. With Regatta, we want to democratize building stateless applications that can take advantage of the low-cost storage of S3.

Curious as to why you would want to build that yourself when so many solutions already exist (Supabase, NeonDB, AWS Aurora or RDS, etc.)?

Super interesting. What did you have to modify and tune in goofyfs and catfs? Did you consider using Lustre at all?

That's exactly the kind of thing that I've been hearing lots of teams having to solve individually, and I'm glad that this set up worked out for you. Would love to see you try it for bioinformatics (another industry where this problem seems to show up frequently), feel free to reach out with any questions when you start that.

Awesome! Great to meet you, so happy that so many folks in the space are here.

Hey there, our file system is strongly consistent for all connected file system clients. For example, if User A and User B are both connected via Regatta, then this works like any other NFS file system (in that they can use file locks, atomic renames or other techniques to ensure that one write wins). However, if User A and User B are accessing the data through different protocols (for example User A is using Regatta and User B is accessing the data through S3), then it's possible to get undefined behavior by attempting to simultaneously update the same piece of data from both places. We think that these applications are rare, and (almost by definition) likely don't exist right now. For the most part, customers use file storage as a "stage" in a broader workflow (for example, customers may ingest data through S3 and then process it on a file system), and that is totally consistent.

This is accurate! A lot of people have spent a lot of time trying to build a good file system abstraction on cheap, S3 storage. However, Regatta differs from these solutions in two important ways. First, Regatta is a shared, durable caching layer that sits between your instances and S3. This means that Regatta is able to efficiently perform operations (like directory renames) and provide strong consistency to other file system clients (whereas s3fs and other FUSE file systems would need to actually perform those operations in S3 for other clients to see the output). Secondly, Regatta is designed to support all file system operations. This means that you can do file locking, random writes, appends, and renames -- even when they aren't efficient to perform on S3.

Thanks for the question!

> How do you handle the cache server crashing before syncing to S3? Do the cache servers have local disk as well?

Our caching layer is highly durable, which is (in my opinion) the key for doing this kind of staging. This means that once a write is complete to Regatta, we guarantee that it will eventually complete on S3.

For this reason, server crashes and intermittent S3 availability issues are not a problem because we have the writes stored safely.

> What are the fsync guarantees for file append operations and directories?

We have strong, read-after-write consistency for all connected file system clients -- including for operations which aren't possible to perform on S3 efficiently (such as renames, appends, etc). We asynchronously push those writes to S3, so there may be a few minutes before you can access them directly from the bucket. But, during this time, the file system interface will always reflect the up-to-date view.

Thanks for the note, great to hear from you! I think that what Clickhouse does is great, and I expect that more applications want to take advantage of the low prices of S3 cold storage without needing to build their own application-level abstractions. I'm hopeful that this allows more of these next-generation serverless data products to exist.

there are more ways to achieve consensus than raft

Great call out. Some kinds of data, like ACLs and specific kinds of metadata, don't live in S3. Full disclosure, we don't support ACLs today (but plan to soon). We keep file system metadata in the durable cache. For some files (where users haven't changed permissions, etc), we are able to release that cached metadata when the file is no longer in use. For other files (where permissions have been changed by the user), that metadata must live in the cache long-term.

We selected NFSv3 due to it's broad compatibility with different compute environments. For example, Windows has an NFSv3 client in it, but doesn't have an NFSv4 client. There are lots of enterprise workloads which needs simultaneous access to file data from both Windows and Linux, and supporting NFSv3 was the easiest path to support those workloads.

We choose NFS purely because it's the fastest way to get broad compatibility with most operating systems (NFSv3, for example is supported on both Linux and Windows). However, I have great news for you! We're simultaneously working on a custom protocol (over FUSE today) that is going to solve the small file problem for things like cloning the Linux kernel git repo. You can actually see in our demo video (https://youtu.be/xh1q5p7E4JY?feature=shared&t=170) that we untar the Linux kernel on Regatta in under 12 seconds. We're hopeful that this performance makes file storage useful for a broader set of workloads.

It's similar to JuiceFS, but JuiceFS writes and reads data from S3 in a proprietary block format. This means that you cannot connect JuiceFS to existing data sets in S3, and you cannot use data written through JuiceFS from the S3 API directly. On the other hand, Regatta reads and writes data to S3 using it's native format -- so you can do these things!

After reading the document, I think "proprietary block format" is a design. Obviously this design can provide parallel data flush back to S3 quickly, and we can cache the reading block and prefetch the next block for more cache friendly.

That's exactly right, I've spoken with a ton of folks who have had a good experience with Lucid Link. I think that we are in a slightly different part of the market (in that we aren't targeting video editors, and more of data-intensive applications which may use thousands of IOPS), but I appreciate that the technology is likely similar.

This is true, and I think that there are consumer (or at least "run on your laptop") versions of this that could make sense. However, the technology underneath would have to look very different. For example, these protocols are designed for online file systems (e.g. you must be connected to the file system directly in order to list what's in a directory). This works great in a data center, but doesn't work great on your laptop.

On the other hand, something like Dropbox is actually a program running on your laptop that simulates a file system, and then does the synchronization at the file level as needed. I think that there's probably some latent demand for a similar product for developers to access their S3 buckets easily from their laptops, and it's something we might look into as we get farther along.

Ha, thank you for the FTP comment, I was hoping someone would make it.

> I really wish that NFSv3 and Linux had built-in file hashing ioctls that could delegate some of this expensive work to the backend as it would make it much easier to use something like this as a backup accelerator.

Tell me a bit more about what you mean here. We're interested in really pushing the limits of what a storage system can do, so I'd be potentially interested.

I totally agree that we're continuing to see a trend of applications which are designed to work directly on S3.

However, like the S3 protocol, I think that the file protocol is cemented in time as something that we will be using 100 years from now. For example, most AI applications do still download data sets to local file system devices to actually load and use, this is why you see a lot of HPC workloads use things like Lustre. Postgres, SQLite, etc all use file system semantics to operate the database.

I totally respect folks who rewrite their applications to work directly with S3, but as you point out, it comes with a different set of challenges (around caching and chunking).

All data cached in Regatta is also encrypted with AES-256

Re: bring your own compute: It’s certainly something we’re thinking about. We are in discussions with a lot of customers running GPU clusters with orphaned NVMe resources that they would like to install Regatta on. We’d love to get more details on who’s out there looking for this, so please shoot me an email at hleath [at] regattastorage.com

You are correct in that NFS is not strictly-speaking POSIX compliant to the letter of the law, due to the caching behavior. This is an NFSv3 file system, so it shares those semantics. The point that I'm trying to emphasize is that the file system supports standard file operations which aren't possible through other FUSE adapters, or possible to perform efficiently on S3 (such as append, rename, and symbolic links) -- which provides broad compatibility with file-based applications.

All connected file system clients see read-after-write consistency, so you see the up to date file data!

Sure can, full disclosure, copied from a comment below:

Thanks for the question! Mountpoint for Amazon S3 is a FUSE layer that doesn't support full POSIX semantics. For example, you can't use Mountpoint for Amazon S3 for random writes to existing files, appends, or renames. This means that you have to carefully instrument your application to understand whether or not it's compatible with Mountpoint, which can be error-prone. Regatta, on the other hand, provides full POSIX compatibility for the file interface, which means that it works out-of-the-box with all file based applications.

Hey, thanks for the question. From what I can tell (and this could be wrong), but it looks like s3ql is using S3 as a block layer. Regatta, on the other hand, allows you to read and write files in their native format. I agree that it's harder to implement than just using S3 for block storage, but I think that it unlocks a lot of potential use cases for customers. With Regatta, we make these semantics performant, which is a huge improvement on the prior art.

I think this is a great call out, and you're correct. One example that comes to mind is that NFSv3 doesn't support flags on the rename() operation (such as RENAME_WHITEOUT), which means that you can't use them as an overlay upperdir (which is desirable for building container runtimes). To solve this, we're working on a custom protocol that we intend to place in the Linux kernel which will expose a broader set of features than we can get in NFS. As I tell people, this is the worst version of Regatta that will ever exist -- we're going to make it better every day.

You can implement a single client NFSv3 server that provides stronger than expected (of NFSv3) guarantees and if you implement the "optional" companion protocols it should come closer to local filesystem semantics than most network filesystems. What would be neat about such a solution is that you can run the server either locally or remotely (same site, high bandwidth, low latency) and at the same time clients would have to a custom FUSE server or even worse load an (from the customer's point of view) experimental vendor kernel module. Upgrading from NFSv3 to NFSv4 would get you a bit closer to POSIX semantics, but of course it would still be NFS just not over a congested, jittery link to a shared server. Especially NFSv4 delegations could be a nice way to let the clients kernel buffer a lot of bursty async I/O locally. Just keep in mind how little POSIX really guarantees instead of assuming it will behave like ext4/XFS or even better ZFS on a laptop NVMe with two levels of power loss protection (big caps in the drive and the laptop battery).

I know a lot of folks have asked me for local support, and while I can share that this would work from OS X -- it's not something that I would recommend doing outside of a data center because the semantics of a networked file system on a sporadic internet connection (when compared to a data center) aren't great -- unless you're doing something higher level like Dropbox. However, it's something we're considering for next year.

You can use rclone mount, depends on how much you're flipping through files or actually doing lots of IO

I wouldn't want to host fastqs or something and use this for alignment, but for spot checking raw fastqs it could be nice

I think that "owning the storage" is such an important part of this. I'm excited that folks who use this will continue to have access to their data directly through S3, so if they ever decide to move off of Regatta, all their data is still right there. This is also important at large companies which already have compliance and governance workflows that connect to data in S3 -- Regatta enables them to continue to use those workflows without having to think about another primary storage system.

Hey, thanks for reaching out. The caching layer does return success before writing to S3 -- that's how we get good performance for all operations, including those which aren't possible to do in S3 efficiently (such as random writes, renames, or file appends). Because the caching layer is durable, we can safely asynchronously apply these changes to the S3 bucket. Most operations appear in the S3 bucket within a minute!

Great question! We fill the same role as AWS Storage Gateway (and I used to work closely with that team when I was at AWS, lots of respect for what they do). AWS Storage Gateway is built primarily as an appliance to be installed on instances in your own data center to ease migration to the cloud. Many customers do deploy Storage Gateway on EC2 because they want these features in the cloud itself. However, the "appliance" design of Storage Gateway makes it unsuitable for this purpose. For example, Storage Gateway is not designed to run in a cluster for high-availability and doesn't have access to durable, long-term storage to stage and cache writes.

On the other hand, Regatta is designed as a cloud-native gateway product. Regatta's elastic, durable caching layer allows us to efficiently cache large data sets without thrashing, and always efficiently perform writes. Because Regatta is designed to be highly-available, customers don't have to worry about downtime for patching or deployments.

Thanks for the feedback. If price is the single blocker for teams to try the product, I'd love to discuss more. Please send me an email at hleath [at] regattastorage.com.

> If i'm on one cloud provider, the traffic between their S3 compatible solution and my infrastructure is most of the time in the same cloud provider

This is exactly right, and it's why we're working to deploy our infrastructure to every major cloud. We don't want customers paying egress costs or cross-cloud latency to use Regatta.

> I also don't get your calculator at all.

This could probably use a bit more explanation on the website. We're comparing to the usage of local devices. We find that, most often, teams will only use 15% of the EBS volumes that they've purchased (over a monthly time period). This means that instead of paying $0.125/GiB-mo of storage (like io2 offers), they're actually paying $0.833/GiB-mo of actual bytes stored ($0.125/15%). Whereas on Regatta, they're only paying for what they use -- which is a combination of our caching layer ($0.20) and S3 ($0.025). That averages out closer to $0.10/GiB stored, depending on the amount of data that you use.

Thank you so much! It’s been amazing to see what you all have built over the years, and it’s (of course) been inspirational for me.

FlexFS kicks ass. I benchmarked it for our data storage and processing layers in value.space (satellite data processing and analysis) and we will most likely migrate to FlexFS in the near future.

Out of curiosity, why did you choose EFS, it's insanely expensive at even modest scales?

Yes, I think it's similar product, but we're looking to provide high performance on all dimensions (latency, throughput, and IOPS). I totally agree with you that Elastic Throughput solves this problem, but it can be expensive for many workloads!

Yeah, I like to think of it in a similar vein. We want to empower people to create stateless workflows where they may have previously needed to think about state management. Today, Regatta is an NFS file system where the cache lives on our shared infrastructure. However, when we complete the work on our custom protocol, that will be a FUSE file system which offers additional caching on your instances to enable truly local-like performance.

Thanks for your note! We're really hopeful that our "local-like performance" is part of the story that distinguishes us from other file system solutions. I envision a world where people don't have to overprovision block storage volumes, and can just use this instead -- with the ability to easiy grab their data from S3.

It's certainly my hope to be cost effective, but I understand the worry and I'm sorry that you had that experience with the PMs of that time. At the end of the day, I see my target customers as those who aren't interested in running their own infrastructure and having to manage availability and durability (in memcached case, things like needing to pre-warm the cache). I understand that it still may be possible to be more cost effective if you're willing to trade off ease of use for dealing with those other concerns.

:sunglasses: We think it's important to be where our customers are, and we're looking to prioritize the next regions that we launch in based on customer demand. We expect to be in more regions by the end of the year.

> given what I know...

Given Hunter worked at AWS, I bet they are way too familiar with IAD.

Total curiosity, but what’s the limiting factor of scaling out to multiple regions day one.

It depends on what you're doing with EFS! For the most part, I would expect to be lower cost than EFS. If you're doing where individual files are primarily written or accessed from an individual instance, I would expect a significant improvement in performance. If you have some time, I'd love to chat more deeply about what you're doing. Feel free to grab some time on my calendar from the Demo link on the Regatta home page or shoot me an email at hleath [at] regattastorage.com.

This isn't a technical limitation, per se, but a time limitation in terms of getting to the place where we feel comfortable supporting those environments for the public. I still wouldn't recommend mounting it from a local environment (because NFS behaves pretty poorly when it can't connect to the server), but we do have a CSI driver for containers running in K8s. We expect that customers will get the best experience if their instances are very close (latency-wise) to our instances, which is why we only support access from us-east-1 in AWS. We expect to launch in more regions and clouds in the coming months.

If you want early access to other clouds or the CSI driver, feel free to email hleath [at] regattastorage.com.

I spent several years working in the Northeast, and I developed an appreciation (but not a skill) for sailing. In some sense, I think of Regatta as high-speed sailing on top of customer data lakes.

One of the fun parts about working on storage and file systems in particular, is that these techniques are old as time. Log-structured writes, journals, caching, etc -- are all non-novel. However, the benefit to our customers is in how easy we make it for them to use something like this without having to deploy or build it themselves.

Sure, and we have more details on pricing here which may answer your questions: https://docs.regattastorage.com/details/pricing

We need to update the home page with these details, but $0.05 is only charged on transfer between Regatta and S3. We calculate your cache usage minutely and tally it into a monthly usage amount that we then bill for.

That’s true, we could just release the software open source, but that doesn’t help our customers who don’t want to run and manage their own infrastructure. Our customers tell us that the value of the product comes from it being fully managed — they simply need to click a button, and all of this works out of the box.

Hey there, thanks for your note. I think that the answer here (as with all good questions) is "it depends".

I agree with you, Object Storage accels at making the storage interface super simple to use (POSIX is incredibly complex). However, that doesn't change the reality that nearly all software still reads and writes data from a local file system interface.

The specifics of whether or not using a translation layer will save you costs comes down a lot to what you're comparing it to. If you have an EBS volume that's 20% full, then I guarantee you that Regatta's storage costs will be cheaper than EBS, even if you don't ever tier to S3. It's just a cherry on top for workloads which may have unpredictable access patterns and don't want all of their data to be hot when not in use.

Trust me, I feel the same way. The problem with these things is that you end up building a company because you get so much conviction that what you're doing is the right thing for customers, and you end up shocked that this isn't the default for everyone.

This is more or less exactly what I'm hoping for. I think that people are excited to build stateless applications, but often that requires really specialized application and storage knowledge to pull off. My hope is that people can use this generic storage layer to build the next generation of stateless applications (including things like databases) without having to become storage experts themselves. I'm also excited to see what they build.

It’s not as clear, but it’s certainly something we are considering. If you’d like to use us on-prem, I’d love to hear more. Can you shoot me an email with details at hleath [at] regattastorage.com?

There is room in storage for many kind of products with different sets of tradeoffs. And don't underestimate the ability of a startup to move much faster than AWS.

It depends on what you mean by crash-consistent. I would expect that we handle crash-consistency at the client fine (since it is the same crash-consistency of NFSv3) and craash-consistency at the server also fine (since we are able to detect using etags what version of an object is in the backing data storage). Tell me a bit more about what you're thinking.

Hey, thanks for asking. It very much depends on which aspect of Regatta you're interested in using. I know of a couple of different architectures -- some folks wrote in "rclone" in the thread, I know of people using SeaweedFS if you want to host storage infrastructure yourself, etc.

I'd love to know a bit more about why you're looking for an open source alternative. Is it because of costs (i.e. you'd like an open source alternative that doesn't require you to pay) or if it's because of the operating environment (i.e. you want an open source alternative so that you can deploy it to your own infrastructure)? There are some things that we are exploring around deploying onto your own infrastructure over the next 12 months, but I'd love to learn more. Feel free to respond here or email me at hleath [at] regattastorage.com.

Thanks for the question! Mountpoint for Amazon S3 is a FUSE layer that doesn't support full POSIX semantics. For example, you can't use Mountpoint for Amazon S3 for random writes to existing files, appends, or renames. This means that you have to carefully instrument your application to understand whether or not it's compatible with Mountpoint, which can be error-prone. Regatta, on the other hand, provides full POSIX compatibility for the file interface, which means that it works out-of-the-box with all file based applications.

Just the theme that we ended up using for the marketing site. We will likely build something less janky post-batch, but right now -- just trying to get the information out there.

It can offer an advantage over the built-in caching, but it depends on your exact access patterns. For example, if you are running ClickHouse on multiple servers and accessing the same reference data, it's more efficient to cache that data in a centralized location (like Regatta) instead of on the disk of each individual instance.

Philosophically, our goal is to build a standard that can be used in these kinds of applications moving forward, so that application developers don't need to build streaming over and over again and users don't need to learn how to configure each individual systems' caching.

There isn't any reason that it shouldn't be a supported use case, depending on your exact performance needs and workflow. It's very similar to ObjectiveFS except that it operates on the data in your S3 bucket in it's native format, so you can point it at existing data sets, and use the newly written data directly from S3.

We don’t actually, but thanks for pointing that out!

Yes, you can absolutely get similar functionality with rclone. However, what we are solving for our customers is the ability to do this without thinking about infrastructure or deployments. Customers don't need to worry about data durability, replication, recovering off of failed drives, or availability through deployments or patches.

It is! Let me share some details in differences that I’ve posted elsewhere:

It's similar to JuiceFS, but JuiceFS writes and reads data from S3 in a proprietary block format. This means that you cannot connect JuiceFS to existing data sets in S3, and you cannot use data written through JuiceFS from the S3 API directly. On the other hand, Regatta reads and writes data to S3 using its native format -- so you can do these things!

Huh, that's interesting. I wouldn't imagine that there were security problems specific to FUSE compared to any other software that you would run on your servers. Regardless, I see FUSE as the fastest path to getting our protcol in the hands of our customers. In the fullness of time, I hope that we can deliver it as either a kernel-module or in-tree.

I think we’re moving in that direction. I’m really interested to do more in the API space than traditional storage has allowed. Tell me a bit more what you mean by “transactional file system”?

      1. fsync, fdatasync - synchronize a file's in-core state with
       storage device

Those costs only apply to data transfer into and out of AWS. If you're running EC2 instances in AWS, your Regatta file system is in AWS, and your S3 bucket is in AWS -- then you shouldn't incur additional data transfer fees.

    "When you’re done editing data, it automatically flows back to S3 within a few minutes"

Thanks for the question!

Snowflake and Databricks aren't storage products, but are managed compute platforms on top of storage that probably looks a lot like this. Snowflake allows you to easily connect different data sets to your data warehouse, and Databricks provides a managed analytics (Spark) offering.

Regatta, on the other hand, would allow you to more easily build the next Snowflake or Databricks by taking advantage of the same low-cost, unlimited storage in S3 that they likely use.

Thank you! We have docs at https://docs.regattastorage.com. There is an architecture page which might answer your questions. If you have deeper questions, feel free to ask in the thread or shoot me an email at hleath [at] regattastorage.com.

Thank you! That’s exactly my hope! How can we make these technologies as easy as possible for teams to use so that they can build the best <training> <analysis> or whatever applications without becoming storage experts.

One thing that we’re considering for these kinds of use cases is providing the ability for users to either (a) request that their data always stays in the hot storage or (b) provide the ability for users to ask us to pre-load their data sets when they begin analysis (and I expect that our large fleet of instances can do this preloading much quicker than any individual instance download). Would either of those options make the product more helpful for you?

Yes! I don't know of any reason why that wouldn't work. I've worked with lots of customers who need simple, low-cost storage for database backups.

We are looking at launching in Azure Cloud with support for Azure Blob Storage as the backend within the next 6 months. If there's a specific use case that you have, it would be helpful to share it with me at hleath [at] regattastorage.com so we can appropriately prioritize Azure against other cloud vendors and regions.

That's correct -- every file is an S3 object. If you change the middle of a large file, Regatta will store the change on our durable caching layer efficiently (and most writes complete in under 1ms). Regatta will then asynchronously update the large object in S3, which may take longer. We automatically batch multiple changes together to minimize the number of operations to your S3 bucket!

Thanks for the question. Full disclosure, I'm grabbing this response from another comment:

I have mutual friends with some of the Nasuni folks, and I have a lot of respect for what they do. In particular, Nasuni stores data in a proprietary block format in your S3 bucket, so you can't connect it to existing data sets or use that data directly from S3 out the other side. Whereas with Regatta, we store data in its native format in S3 so you can do these things.

I could totally be misreading DirectPV, but it appears to be a way to use K8s Persistent Volumes to manage things like NVME drives which are attached to each node, and doesn't provide any tie in to S3 (outside of the fact that it's built to power MinIO).

Great question! Full disclosure, answer copied from a another comment:

It's similar to JuiceFS, but JuiceFS writes and reads data from S3 in a proprietary block format. This means that you cannot connect JuiceFS to existing data sets in S3, and you cannot use data written through JuiceFS from the S3 API directly. On the other hand, Regatta reads and writes data to S3 using it's native format -- so you can do these things!

Well, I think this is the benefit that our customers are looking for. They aren’t interested in becoming storage administrators, and running Regatta as a service allows them to not. There are, of course, other teams who do want to do that. It’s great that both kinds of products can exist.

Thanks for the question. We stage writes to a durable, shared caching layer. This allows us to respond quickly to your application when it performs these operations (<1ms), but then asynchronously send those operations to S3 later. When connecting through Regatta, all file system clients see a strongly consistent read-after-write view of the changes on the file system, even if they haven't yet propagated to S3.

(full disclosure, reposted from a comment below)

Great question! We fill the same role as AWS Storage Gateway (and I used to work closely with that team when I was at AWS, lots of respect for what they do). AWS Storage Gateway is built primarily as an appliance to be installed on instances in your own data center to ease migration to the cloud. Many customers do deploy Storage Gateway on EC2 because they want these features in the cloud itself. However, the "appliance" design of Storage Gateway makes it unsuitable for this purpose. For example, Storage Gateway is not designed to run in a cluster for high-availability and doesn't have access to durable, long-term storage to stage and cache writes.

On the other hand, Regatta is designed as a cloud-native gateway product. Regatta's elastic, durable caching layer allows us to efficiently cache large data sets without thrashing, and always efficiently perform writes. Because Regatta is designed to be highly-available, customers don't have to worry about downtime for patching or deployments.

Hey there, I have mutual friends with some of the Nasuni folks, and I have a lot of respect for what they do. In particular, Nasuni stores data in a proprietary block format in your S3 bucket, so you can't connect it to existing data sets or use that data directly from S3 out the other side. Whereas with Regatta, we store data in its native format in S3 so you can do these things.

What's cool about the storage market is that there are so many impressive companies because there are so many varied needs from customer applications! We're hoping to become a simple "default" for teams who are writing applications in the cloud.

Sorry about that! It's on our list to fix once we're done responding to comments.

I don't think so, I see them as complementary. MinIO is great when you have downstream applications which speak the S3 API that need acceleration of that data. Regatta is designed for applications which speak file semantics (think, application logging, storing corpuses of training data, or state) that doesn't run on the S3 API. Regatta actually supports MinIO as an S3-compatible backend for your file system!

These distributed storage systems solve very similar problems, depending on how you use them. Our target customers aren't looking to deploy their own infrastructure, so having a "single-click" option without having to think about how much capacity they need is very valuable.

Great call out -- we'll get that done. Thanks!

This is a managed cloud service. If you're interested in using Regatta on-premises, I'd love to hear from you -- shoot me some mail at hleath [at] regattastorage.com

Because people are excited... All the positive comments didn't tip you off?

Hey there, thanks for the concern. There are a spectrum of teams out there. Some teams are totally comfortable building something like this and running their own storage infrastructure. Other teams want a fully managed solution to handle storage for them so that they can focus on building. I think it's great that we have a spectrum of products!

I bet this guy runs his own servers and databases in his basement too, because fk TCO amirite