Making Roomy More ATProto-Native

Last weekend I did some experimenting with Rivet for making my own PDS. I didn't get very far yet, but I did get some good perspective to add to some slow thoughts that we've been accumulating at @roomy.space .

To be clear about this whole post, it is largely speculative musing. We aren't going to put any of this into practice at least until after AtmosphereConf, and some of the ideas here haven't been strictly scrutinized by the team. But I think these are exciting thoughts for how Roomy might develop long term.

If we go back to some of the earliest thoughts along these lines, @meri.garden had the initial idea of using an MST for Roomy spaces instead of the append-only log that we are using right now.

That came up a while back and I wasn't so sure at the time, but since then we've gotten closer and closer to lining up with ATProto's design decisions.

First we realized that DID PLC was really good for allowing adversarial migration, and we used it for our streams to allow you to migrate your Roomy space to different hosting without the permission of your current host.

Then we decided that using the DRISL CBOR format that ATProto used was a good idea.

And if we were using DRISL, then with a little reconfiguring we could also have valid lexicons.

At this point, the only thing we aren't doing is making Roomy spaces legit ATProto repos. But what if we did....?

Early on in Roomy we decided not to build on top of ATProto directly, mostly because of privacy concerns. I won't go into the details here, but you can find more context in our blog post here:

Leaf 0.3 - The Server Behind Roomy

For the last couple months we've been iterating on Roomy with its brand-new architecture, and we're finally ready to talk in more detail about the not-so-secret sauce that will power Roomy moving forward.

At this point, though, we are finding ourselves agreeing more and more with the existing ATProto design decisions, and we even realized that there is probably some untapped local-first potential too.

We want to try to make Roomy spaces a special kind of community PDS repo. These repos will have built-in authorization that allows multiple people to write records to it, and they can also have custom logic around who can read specific records from it.

Instead of a personal PDS, it's a community PDS. Think something like a community forum. It intentionally acts as a shared place for the whole community to store stuff with rules that can be enforced by the moderators.

While Roomy will implement custom features around these repos for things like authorization & privacy, to the extent possible, we want them to be "normal" ATProto repos that other apps will be able to understand and support integrations with.

We're thinking of using UCANs as the underlying authorization primitive, and we're exited to start experimenting with it.

We're not 100% sure what this will look like just yet, but we're really liking the idea of having something more than just an "off-protocol" chat server. We already knew we'd be wanting to add XRPC endpoints for integration, but now we're thinking of going as all-in as we can while still adding the custom features we need.

And that's another thing, everybody who is doing anything with privacy on ATProto right now is rolling their own solution. We don't have a standard yet, so it'll be cool if we can at least add to the conversation with our own attempt at a solution.

We're not sure whether the work-in-progress "official" solution for privacy will be enough for our needs for Roomy or not. If it isn't, then we'll go right on ahead using our own solution as much as necessary. But we do want to align as much as as possible with whatever ATproto specs end up being standardized on by the community.

Essentially we've finally validated almost all of the existing ATProto specification by "re-inventing it" ourselves and then collapsing on what ATProto had standardized.

This may sound wasteful, but it was actually really important for us, because we, significantly, came into this with a goal of supporting local-first in the future, and we also had a completely different target use-case from big-world social media like Bluesky.

We needed privacy and small-world social networking. Bluesky had no support for one, and no examples of the other. So we had to deep dive into the design space and figure out what was essential for both a great user-experience, and extreme user agency.

By now, we think we understand the design requirements and how ATProto, while incomplete for our goals, doesn't conflict with them.

Rivet & the Actor Model

Along this journey of discovery we've also found out a bit more about what's necessary for reliably designing event-based systems like Roomy.

Roomy's current design works almost like mini firehoses for each chat space, with one AppView for each stream which is written entirely in SQL.

This has it's limitations. The design was inspired by a plan to be more local-first, and have heavier clients with simpler servers, but we have found the limits of such a design as we run into more need for server-side functions to provide the uncompromising UX that we are shooting for. To a certain extent, we are re-discovering ATProto's own journey of pragmatic decentralization.

To make the sytem more reliable, and more easily extensible with server features and other ATProto integrations, we are looking into using Rivet to build a server that will act as both a PDS, and a Roomy chat server.

The server will allow users to sign up for an account, in which case it will create them a normal ATProto PDS repo, as well as have support for creating Roomy community spaces, with our community repo functionality and all the features needed to power the Roomy app.

Rivet gives us a way to create something like a microservice swarm that is easily scalable with internal message queues and indexes that are almost like ATProto AppViews.

For example, we will create an actor that represents that community PDS, and different server features that we implement for Roomy will be able to subscribe to events on that repo and index them like an AppView, but without needing to leave our internal actor system.

Functionality such as our Discord bridge, which requires talking to external systems with inbound / outbound queues of messages fit really well with the features provided by Rivet. Other features like migrating from another PDS to Roomy's, would be a perfect fit for Rivet's durable workflows.

Rivet's design just matches so well with our needs. The actor model that Rivet implements is essentially the same thing that we alredy wanted to implement with our Leaf server, but we hadn't had the chance to mature it yet. The hope is that Rivet will provide us a solid base to build on, and that it will allow us to focus much more completely on Roomy. As far as I've seen they've done everything essentially the way I'd want it done, and the design preserves us the flexibility to do anything we should should need to do.

Incremental Implementation

The best news? We don't have to do this all in one big change. To reiterate, we aren't doing anything until after the AtmosphereConf. We just finished a nice stress test for Roomy and things are working. That's what's important right now.

But as we need to get Roomy more reliable, performant, and add more features like integrations and push notifications, there is a clear path for incrementally getting Roomy ready for that.

The rough plan looks something like the following:

First we can put our microservices on Rivet. The idea at this stage will be to get all of the small, simple stuff running on there as a first test and to get a feel for things.

Then we migrate the Leaf server to Rivet. We can do this without changing the way that Leaf works. The whole Leaf architecture can stay pretty much the same, except that modules will be implemented in Rivet instead of pure SQL. It will require minor changes to have the frontend use the Rivet client instead of the Leaf client. The goal here will be to leave everything as similar to it is already, just running on Rivet instead of our custom server.

Next we migrate our PDS to Rivet. At this point it's looking like it might be possible to take a good chunk of the reference PDS implementation and recombine it to work on Rivet. Getting our PDS running on Rivet will be a good test of Rivet and it will be a good preview of our future architecture plans of making Leaf more like an ATProto PDS. It also means that all of our infrastructure is running on Rivet and we don't have to separately manage the Roomy server and the PDS server. This is also good for self-hosting because now there's just one server to run.

Finally, with all of our infrastructure on Rivet we can start working on our plans to Make Roomy more ATProto native. Doing things like making Roomy spaces stored in an MST repo and trying to unify them as much as possible with the PDS implementation.

I'm really excited about this idea, because it would not only get us more integrated with the ATProto ecosystem, but also get us ready to scale, improve reliability, and enable us to iterate on features with more focus on Roomy itself.

Again, this is quite speculative. We're not sure whether things will pan out the way we're thinking about right now, and we've definitely got some technical details to nail down.

But that's OK! We'll just take things a step at a time and figure things out as we go. Having a rough plan ahead of us is exciting, though, and helps keep us focused as we push Roomy forward.