A Bit of Backstory
I started this post to write about how I'm using Quint as I work on the next iteration of , but I ended up thinking about why I even started looking into it in the first place. It's actually surprisingly deep so even though it's a bit of a weird spot for it, I'd like to share a bit of backstory.
A Crazy Year
The last year of working on Roomy has been a lot when it comes to architecture churn. Our passion for making software for even the least privileged use-cases led us down the path of local-first technology and resilient systems that seemed in conflict with the server-based software hosting that is common today.
In the end, we found that we did need to use quite powerful server features to get the uncompromising user experience that we were aiming for. But at the same time, we realized that, designed properly, having servers that optimize for user-experience doesn't mean we can't make tech that can work local-first, too. It will take us longer to get to that, but that's OK if we can satisfy more real-world needs today, and get our business going.
Anyway, we've been changing Roomy a lot over the last year as we worked through this, and it was something that was quite difficult for me to figure out how to handle.
I knew that I needed to make Roomy work. Not that I'm alone! There have been so many helpful people along Roomy's journey. , of course, , , , , and finally who joined me and Erlend as partners. Thank you all so much, and plenty of others who have joined us to chat and talk about things in Muni Town. 🙏
But I knew that with Roomy that we had to get something functional. It wasn't a side project that I could stall indefinitely for the sake of technical elegance or theoretical purity. The point was to actually help people.
So there were lots of times where I swallowed my gut feelings of "this isn't quite right" or "I don't like how complicated this is" or "this shouldn't be this hard". To be clear: that was absolutely the right thing to do.
To add to this, I also wasn't the only one working on the project. That was somewhat new to me, and I had to come to grips with the fact that I couldn't necessarily keep track of or take ownership of everything. I had to weigh the importance of things, and figure out how much attention to put where. This was also great for me to learn, and it is something I still need to get better at. But with all of the deeply fundamental revisions we had to make at the same time, it was very hard to get right. I often felt bad for our contributors who could have their stuff completely mangled as we made major change after major change.
Still, I don't regret any of the decisions we made. I think we needed this journey that we took. There was no better way to build the experience, character, and relationships that we needed to make Roomy what we want it to be. I also think there was wisdom in each rework. Things just haven't totally clicked into place yet.
Where We're at Now
Right now, we're running our own, custom, mostly local-first-capable system made up of a sophisticated web client with its own SQLite database and a simple server with just enough features to get by.
But the client is over-complicated and tricky keep working, the server isn't powerful enough to do the indexing that we need to make the client performant, and we're not really able to deliver on the local-first stuff right now because it adds a lot of extra UX requirements.
I got to the point again, where I felt like things were still wrong somehow. Much more right than previous iterations, for sure. But I could see the cracks in the decisions and assumptions that we had made and I could feel the pressure it was putting on us as things were more difficult to keep working than they should have been.
This got me thinking. A lot.
Roomy Will Keep Working
Before I continue, I want to point out that we are going to keep Roomy working. If any new version comes along, we will migrate the data from the old version. As work-in-progress as it is, people can use Roomy right now and we are going to keep it going, and we are going to improve the current version with important missing features such as privacy and push notifications. That's all on the roadmap already.
( 🙏 ) is also always quick to point out how what we've made is like actually doing its job and pretty well. It's a good thing that we've made, and we've put a lot of work into it. It would be a good thing to keep building on it.
And it's not that I disagree, but I do feel like we need things to be better still, and I was trying to figure out how to do that. So that's why I was doing lots of thinking.
Coding Should be Easy
I recently ran into this quote in a talk by Leslie Lamport, which I wholeheartedly agree with:
Coding should be the easiest, lest important part of programming.
If you're having trouble writing a piece of code, you're doing something wrong.
And what you're doing wrong is thinking at the code level, and trying to solve the problem, that you should be solving above the code level.
This is not to say that programming should be easy, but that writing the code should not be the hard part of it. The hard part of programming is designing solutions, and that is something that operates above the code at a more abstract level.
The issue is that in the Roomy app, we have a lot of code that we were having a hard time keeping working. Some of that is because the web platform can suck to work with a lot of the time. Not totally our fault there.
But I couldn't shake the feeling that our code didn't fit the problem. It was there, and it was doing things that we told it, but it wasn't the right shape for what we were realizing we needed to do. And it was slowing us down because for every change we made, we had to fix all the things we broke when we changed it.
I Had Gotten This Right Before
Before I started working on Roomy, I worked on some video game projects with . I contributed to the Jumpy game, then rewrote it to use the Bevy game engine, and then wrote my own game engine, Bones, from scratch, which finally satisfied our needs extremely well.
Game dev was hard too. We had some rather tricky needs not found anywhere else:
- 1.
Excellent network play based on rollback networking, requiring determinism and snapshot + restore
- 2.
Powerful modding features
- 3.
Rust-based engine, but with Lua or JS scripting
I went through many of the same phases that I experienced in Roomy while working on Bones & Jumpy, too.
There was the pain of putting up with existing tools that just weren't working for our needs. There was that scary feeling of uncertainty and almost guilt that came with thinking that somehow I needed to write my own game engine from scratch, because nothing else would cut it. And there was the slow grind while working on our own solution while I just hoped that I was making the right decision and not wasting our precious time on a fools errand.
But I also know what it was to succeed. Bones worked, really well. It was hard, but I found the abstractions we needed. I understood the problems that we were having, and I was able to model those problems in clean code that kept things simple, and helped us focus on making our game.
We didn't finish the game yet, because the movement in the open web, and our being a the mercy of Discord was just too important to ignore. But Bones worked, and my brothers are actually using it to make their first games as I write this! It's so cool!
I know what it feels like when the hard part was not writing the code, and I've been trying to get Roomy to that place for the past year. But have also been willing to let it go if that's not what was best for Roomy.
Feelings Aren't Everything, but They Are Something
Now feels like a good time to mention that this has been a lot about how things feel. And one thing I've learned about how things feel is that it often doesn't reflect how things are.
That's not to say feelings aren't worth something. The feeling of difficulty that came with using previous solutions is what drove me to make Bones in the first place. And now Bones is making my brothers' jobs easier, and hopefully enabling things that wouldn't be possible without it.
But I also know that things can feel like they are falling apart and sometimes you just need to stick with it and push through. Sometimes it's hard to judge which is which and finding balance can be difficult. Being honest and having a team where we trust each-other is huge. Thanks again, Erlend & Meri. 🫶
Also feelings can often encode things that you've learned from experience that you can't quite put your finger on. The "feel" of what I know to be right from years of coding is a large part of how I make coding decisions. I tend to learn more from doing and experiencing than I do just guessing.
Towards ATProto Native
Last month we had decided that we wanted to try to make Roomy as integrated with ATProto as we possibly could.
This gave me a good excuse to do some hard thinking at the drawing board. Not to completely redo what we already with Roomy, but to try and think about what could be better and how we could make things really reliable this time around. There was a lot that we got right in the last version, we just had to fix the things that kept tripping us up.
We needed our code to reflect the problems that we were solving with it. When the code is cleanly shaped like your problem, it's easy to modify as your needs naturally change or adapt. It's easier to add new features, because your foundations aren't built on false assumptions about what is important in your design.
This takes knowing your problem space, and that is something that is actually really hard to do, unless you already have tons of experience with it. This goes back to how we really needed this last year of rough rework and experimentation. We got vastly more familiar with our problem space.
I was doing all this thinking during AtmosphereConf while on-call, trying to keep Roomy working throughout the conference.1 Meri was at the conference and gave a talk, which was great!
After the conference, me and Meri had a catch-up call and discussed the stuff I thought we should change. Things like making the client simpler, removing our very cool, and original browser SQLite live query system 😔. We needed to move more processing to the server, remove barrier to development of server-side logic, make things more flexible and reliable.
It wasn't fun to think that we needed to make big changes, especially when we knew that we needed to keep the current version working, and improve it at the same time. But the consensus was that we probably did need to do it, so we would spend some time planning what the next Roomy might look like.
As steam was picking up for permissioned data in the Atmosphere, and our desire to somehow put Roomy on protocol, we knew that was the biggest blocker. So we started doing a survey of existing solutions for private data integrated with ATProto to get a deeper understanding, we think we have a pretty good plan now.
In-fact, just today I published the overview of the first new service we're going to implement:
The Actor Model
Anyway, while I was thinking about different ways to improve our server organization, I got pretty excited about Rivet:
One of the abstractions I was pretty sure we wanted to lean into was the Actor model. It has this idea of isolating mutable state to actors which communicate by sending messages and I think it would help fix a lot of the reliability and flexibility issues that we have today.
Rivet looked like it could do all of the complicated stuff for us regarding hosting scalable actors, and we would get to focus on writing them. But unfortunately, after getting into it, Rivet just doesn't seem stable enough for our needs.
Their actor system is too new. I really love the design of their API and the whole idea that they've got. But we ran into bugs that we couldn't fix quickly, and there were some rough edges. It just didn't feel ready for us to depend on as crucial infrastructure, as much as I loved the idea behind it.
So if not Rivet, what? I wanted to know that we could scale Roomy if we had to, but I didn't want to have to make a Rivet equivalent by ourselves. We don't have the time for that.
We heard about NATS working really well for the author of Chatto, a neat, self-hosted chat server that isn't ATProto integrated. I checked it out and it's really cool, but NATS only does messaging, not actors.
Still, NATS looks like one of our best options for a mature and scalable foundation for Roomy's server infrastructure.
If we wanted to do use something like actors, but we were to build on NATS, we would probably have to implement our own scheduler. Kind of like a container orchestrator, we'd want to be able to have the system figure out which server to run an actor on, move its data over there, and then start it up.
NATS could handle the messaging, but getting into distributed algorithms like we might need for actor scheduling can be really tricky. I wanted to make sure that we had a way to make Roomy reliable, even if we had to get into distributed programming.
Wrapping Up
Well, this is already a long post that didn't get to talking about formal methods like I meant to when I started writing. 😄 I'll get to writing about Quint & formal methods in the next post.
This has been a lump of thoughts and history, which I suppose doesn't really need a point per se, but there should be some takeaway from all that.
I think one of the things I'm trying to get out is that solving problems is difficult work, but the best way to do it is to actually go and do it. The challenging struggle that ensues is part of the process.
I also think I realized that a lot of my feelings about things were right to a certain extent, but that didn't mean I didn't have to spend the time uncomfortably working through things.
And there's doubtless more of that ahead.
But I think I'm starting to get more aware of this process, and I'm improving my ability find the right patterns quicker I think.
Making Roomy has been a lot harder than making Bones. It's funny because with games, there are a lot of challenging things, but you control the whole world. You have priority access to the player's computer. Yeah, things need to happen in under 16.6 milliseconds, but at least you don't have to deal with social coordination combined with application inter-operation and persistent data that must be replicated and somehow private.
But really this has been pretty great and amazing. We are going to figure this out. I feel like we've got a level of clarity that has been hard earned over the last year, and I'm pleased with all that we've learned.
Learning to write software is hard, but it's also good. PS that's basically life. 🙂
Well, that's been a lot of writing this week-end, hehe. Next up, formal methods & Quint!