How Omarchy Went From a Side Project to a $10 Million Foundation

Welcome back to another episode of the Not Brothers Podcast. It's been a minute since we've done one of these because we've been a little busy with a lot of things. today we're talking about a project that I've worked on for probably the better part of a year and a half. and has really gained a lot of momentum this week. Not that it didn't already have plenty of momentum, but it's Omarchy. It's a an Arch based Linux distribution that I've been working on. It was created originally by David Heinemeier Hansson over at 37signals. And we thought it appropriate to talk about that right now since we just did the launch of Omarchy Quattro, which is the probably the biggest release that we've had since the original creation of Omarchy in the first place. And really introduced some some new things like plugins, which app went absolutely fucking insane, to a lot of other things. And also this week, DHH announced the Omacom Foundation, which will kind of sit over top of Omarchy and help to drive forward the vision that you know we've kind of been working on in the background for some time. It'll help kind of ignite that and push it forward in a bigger way. So Mark doesn't use Omarchy yet. so we thought, you know, what more fun than to have a discussion about this from the perspective of, you know, I've been in this every day for, you know, since it came out. so it kind of is difficult for me to remember what it was like to use, you know, normal computers were like. And I still have to every so often, right? I'll pick up a MacBook and I'll I'll boot it up or I'll have to wipe it and it takes three hours. I'm like, what the fuck is going on here? and then you have windows like pop everywhere and it's a disorganized disaster. And I have a panic attack and I go back to Omarchy; everything's clean and beautiful and organized, and it just works. so that'd be fun to kind of compare those perspectives, maybe allow Mark to ask some questions from an outside perspective of somebody who hasn't been in the trenches, who doesn't use the operating system on a on a daily basis. just to provide for some perspective and yeah.

So let's start with the let's start with the basics, like How did you even get involved in this project? Like it was, you know, it's kind of a sort of a fringe subject, right? So the Linux user base is what something like 4% of the entirety of desktop computers, according to some basic research. And and then this

Yeah.

is a a fraction of that. So like how in the world did you first off even know it was becoming a thing? And then how did you get involved directly with this open source project?

So David had created kind of a he started tinkering in Linux sort of around the same time that I started tinkering in Linux in Linux. So like, you know, thinking about I've always gone to and from, right? Like I, you know, have had Linux VMs that I've carried around and I've, you know, even at one point ran my primary operating system was like a virtual machine was a virtual machine that I kept on an external hard drive. that I would just plug into my laptop or my desktop and it would kind of be transient. And I would just like full screen and, you know, I think at that point it was Manjaro was the distro that I was on. He created his setup called Omakub based on Ubuntu. and, you know, I tried that out. I've never personally been an Ubuntu fan for a number of reasons. So I, you know, tried that out. I was like, this is interesting. I've always appreciated, you know, David's ideas of how computers should work and that they should just It should be a positive interaction and the aesthetics are important. And you know, just the the general experience is good. Like Linux is not known for having a positive experience. It's usually like, here's a box of fucking Legos and I'm gonna dump on the ground. Good luck figuring out how to make them, put them together. And then people put them together and you have this like miscolored, not matching, ugly piece of shit that only the creator could love.

It's because developers are like, It works, what's the problem?

Yeah, and you're like I mean the problem is that it it nothing is cohesive. not only that, it took a lot of time. So he created Omakub and the idea was, you know, hey, this is my setup, the way that I like to use it, and I'm sharing it with everybody else. And that took off. A lot of people like that. I was like, This is really cool. I like a lot of the stuff that he did here. I just don't like Ubuntu. So I kind of started down the path. Of like how can I take Arch Linux, which is always like bleeding edge, rolling release distro, which is more my speed. And how can I take that

from beta.

and take all the cool stuff from Omakub and kind of put it in here? So I'd kind of started dabbling in that. That same literally that same weekend, David shared on X the Omarchy logo. And I was like, Holy shit, this motherfucker's playing with Arch. and I think it was the following week he released kind of version 0.1 of Omarchy. which was effectively the same kind of idea that I was working towards. Just he had done the same thing. I was like, this is fantastic. So I just kind of showed up to the party with all the stuff that I had built and and started just lobbing pull requests over at at the project. and eventually we connected and and you know, we're like, Hey, we want to make you know an actual ISO for this and do all these other things. He was like, You wanna work on this way? I was like, Yeah, fuck yeah, let's do it. And the rest is kind of history, right? We just kept plugging away and you know our ambitions got bigger and bigger. You know, what started as, you know, effectively like David's setup and dot files grew into something that was a lot more than that, and then grew into something that was a lot more than that. And before you knew it, we're you know, we're staring down the barrel of a you know, a true dis separate distro. still Arch based. It's still Arch at its core. but a completely separate distro that tons of people are using, you know, thousands, hundreds of thousands of people are using. so it's something that is still very opinionated intentionally, it's still you know, not built to accommodate everybody or everybody's whims, but is built to be something that everybody can use. and this idea that we're we've been adopting now that David's kind of coined as like the malleable computer, where the goal is that your operating system, you know, we kind of drop you off with a very opinionated, good set of defaults, but the tooling to be able to make it whatever you want to make it. And you don't have to stick with what we've come up with necessarily, but we do also have guardrails to kind of keep you in and keep you safe along the way.

Okay, so as a guy that doesn't use Omarchy, I I do use Windows and I do use macOS. I've never used a Linux instance. I've seen a Linux instance. I've seen your Omarchy setup and I don't live my life in terminal. Sell me on Omarchy. What makes it different and unique? Not just from other Linux based instances, but also from a Windows user or from a macOS user.

I think the biggest thing that you would notice right away is the tiling window management. tiling window management is a different paradigm than you know your typical computer where you just kind of drag windows everywhere and it's a goddamn disaster. with tiling window management, the whole idea is everything's a tile, right? So if I have one window open and I open another one, they split 50-50. If I open another one, It's half here, quarter here, quarter here. If I open another one, it's all it's four quarters potentially. or technically it would be like half quarter eighth eighth. or you can have a scrolling layout where it's like I, you know, I open this and then I open this, and then the next one's here, and the next one's here, and I can kind of traverse through however I want. That's the first thing anybody's gonna notice is that you don't drag windows around, you're not organizing windows, you know, the windows are just kind of all there. And you have the idea of workspaces like you have in macOS, where you may have different things on different workspaces and you sort of know where those are. The second thing would be it's very keyboard driven. So rather than you know, using your mouse to navigate between one window and another. you would use like super arrow key to kind of move between the those. Or instead of dragging one window from one side to the other, you would use like super shift arrow key to move one over the the other direction. Those are probably the two biggest differences that are acquired tastes. once you get those down and it comes it goes pretty quick.

Well, a as someone that actually I absolutely loathe the window management in macOS because

Or lack thereof.

lack thereof, right? For for an operating system that's supposed to be the simple like quote unquote simple for users, it makes it really freaking complicated real fast. that's like the window management sounds like something that would be freaking amazing. And you actually demonstrated this to me at one point where you you were like, Look, I have four windows up. Look what happens when I open a hundred terminal windows just for fun.

Yeah.

It it it just, you know, tied all together.

or if you have a bunch, you know, I can say like make this one full screen for now and it'll just kinda grow to full screen, but then you know, I can escape back out of that and it goes back to its place. Like there are a lot of advantages to that sort of setup. I t everybody uses it differently. I tend to have like, you know, workspace one is where my main browser lives, workspace five is where all of my, you know, Slack and Signal and and Discord and any communication things are all on workspace five on a scrolling layout. So I just kind of know like if I'm if I need to go to Slack, it's you know, it's super five, that'll pop me to that that workspace and I'm probably sitting on Slack if it's during the day.

The idea of workspaces to me is is always been fantastic in macOS, but because of the window management, it makes it borderline unusable. So I actually have defaulted to almost not use it. Right. So my my window management is essentially one workspace with a whole bunch of half screens and quarter screens that I kind of na navigate between.

Yeah, you four fingers swipe up on your computer, and I'm sure you like I would have an aneurysm.

Yeah, exactly.

the the other thing that like we've tried I tried it on macOS. So before even going down this route, I was like, Can we do tiling window management on macOS? And there are some solutions, like aerospace is the closest I ever got to it. And because Apple's Apple, everything's locked the fuck down. there are certain things you just can't change. Like the most annoying one by far is that when you swipe between workspaces, native workspaces, aerospace kind of creates these like pseudo workspaces, which works a little bit better, but also like doesn't it takes away certain pieces of of the whole workspace idea. The native workspaces, there's a hundred millisecond delay. Every time you change workspaces, where it basically screenshots the screen, slides it over, and then there's a delay before it releases it and gives it back to you. So if you just swipe, start typing, your input's lost. It doesn't go anywhere. And that drives me absolutely fucking insane, especially as somebody who switches Windows often. So that was a big thing, is like the speed, right? It's instant. We can I can hit, you know. Super one, copy something, super two, start typing immediately. And these screen transitions are not they're quick, they're not blocking me from doing actual work. And on macOS, like it doesn't sound like a lot of time, a hundred milliseconds isn't that big of a deal. But when you do that often, you start to feel that hitching every time you try to do something. It's like, you know, you just kind of like stumble a little bit.

Yeah, when you do a couple thousand times a day, it becomes something that's annoying. It's it's blocking you from just moving on.

Yeah, it's brutal. I think that the you know, speed is the natural one, right? we are kind of obsessive about speed for probably a lot of reasons, but You know, making sure that the operating system is incredibly performant, making sure that the installation is incredibly performant. Like I think the first version on kind of modern hardware and whatever we can install in like f it was like seven minutes or something like that, probably. the first version of the offline version. The very first version would, you know, it would probably take like fifteen or twenty minutes because it had to download everything from the internet. So it's kind of you're you're hamstrung by how fast can you download six gigs worth of shit. and then install it. Eventually we went to an offline ISO because we wanted to solve the connectivity issues, but also speed up the install. And I think that knocked it down to like seven minutes somewhere in there.

Which is still incredibly fast as compared to any other modern OS.

Yeah. then we knocked it down to sub five, right? You could get everything completely set up in less than five minutes. then we made it less than two. It's now less than one. I've

That's insane.

I've seen forty seconds.

That's insane. And that's that's without like ridiculous server grade hardware, right? This is just like installing it on the desktop.

Yeah, this is just on a laptop. This is on, you know, the a a typical consumer laptop. Like modern hardware, right? You're talking, you know, modern chips and everything. But you know, yeah, this isn't like an EPYC processor server or something like that. It's just a consumer grade desktop system. you can install the operating system and be ready to go in less than a minute, which is absolutely insane.

That's insane. Seven minutes is insane. Less than one minute

Maybe.

is like ridiculously insane.

The the crazy thing about it is I you know one of the things when I was up in Ohio this week, I went with my cousin and your other nephew. and we got him a new gaming computer. We picked up you know, a nice power spec system for him from Micro Center up there. And you know,

Happiest place on earth.

yeah, it is outside of Disney World, Micro Center is the happiest place on earth. he Brought that home. Obviously, it comes preloaded with Windows. We booted up. And it took almost an hour to finish installing Windows. I was like, what the fuck? We could have, I could have just wiped it and put Omarchy on it. Like I have it on a USB drive. I could have just wiped it and put Omarchy on it. Like and saved an hour of time. in that case we didn't because there we still have the Kernel level anti cheat problem that is being solved. Epic's committed to bringing their games to Linux, which is fantastic and I think will lead the way for other gaming studios. But we still have some studios, like Epic and Bungie and Riot who have kernel level anti cheat that is available for Linux, but they just have it disabled. So if you're running it on Linux, even if everything will run just fine, you're fucked. Because the kernel

Yeah.

anti cheat just says no. so hopefully we see that change, especially with you know, people like Valve bringing Steam and the Steam machines to market. surely they would love to see, you know, these AAA titles available on the Steam Machine. and they intend to invest, I'm sure, in that in that ecosystem. So it's a it's a matter of time before we solve the problem, I think. But anyways, we didn't do that. But it took an hour and I was like, this is insane. Because we can it takes less than a minute to install the damn thing. And then from there, it's faster at everything. There was a guy who just did a head-to-head benchmark. He took two of the exact same mini PCs, set them side by side, put Omarchy on one, Windows on the other one, and just did, you know, some standard Unigine benchmarks and things. And the Omarchy one beat it on everything. And that's Graphics. Typically Linux does worse in

Mm-hmm.

you know gaming and things like by a few frames. just based on the driver optimization and that kind of stuff. Obviously, Windows still gets a little more love. It beat on all of them. I did a test some time ago, I should do it again, where I took a was an Intel-series MacBook because we also, you know, it we can work on any of the what we call vintage MacBooks, the pre-M-series MacBooks. It'll also technically kind of work on M1 and M2, but there's a lot of hoops to jump through and it is something like that's something we seek we're seeking to make better and have better support for. M3 through M5 are like super locked down and you know. We need we need a good amount of work to figure out how to get those chips, get it to work on those chips because Apple has kind of locked the chips and basically said, Hey, go fuck yourselves. you can't use the hardware you bought for anything other than what we want you to. which is unfortunate. I think the the I took that MacBook, formatted it. It took three hours, by the way. and then pulled down, I think it was Open Street View. I pulled down their project and just ran their test suite. And then I wiped that system and I put Omarchy on it and I ran the exact same test suite. Same hardware, same everything. And I think it was like 50, it was twice as fast, if I remember right. It was an insane amount faster. which would be my cell, right? If you want an efficient operating system that works really well, that doesn't have planned obsolescence, that probably runs on a fucking potato. and provides a better user experience overall than anything you've ever used, that's Omarchy in a nutshell.

So so you guys you guys have taken this at this point from a side of desk project, kind of personal usage optimizations of a Linux instance based on Arch and have evolved it into effectively a full-blown operating system that has aspirations to be competitive with something like an an Ubuntu. So who who has started to take notice and get involved in the project, both from a contributor standpoint? And then let's talk a little bit about the foundation too.

Well, there are a lot of people who've taken notice. a lot of trolls.

I'm sure.

you know, it's it's to be expected with anything, right? Like anything it's kind of how you know that you're you're doing something right and that that you're doing something that is good is people start to hate it. so there's definitely

The more hate, the more right you are.

Pretty much, yeah. so there's certainly been There's certainly been some some folks who they don't like it, they don't they disagree with what we're doing, they think it's just a bunch of dot files. They're all wrong. I don't even spend time with it. Like I'll read I'll see some of them occasionally when I'm scrolling through my feed and I just kinda chuckle. Usually they're all uneducated viewpoints where, you know, they couldn't have been bothered to just understand anything, which is unfortunate. I wish they you know, wood. I think that, you know, with establishing the Omacom Foundation, which happened this week, you know, David kind of spear spearheaded that, pulled in a bunch of folks from the tech community who contributed, you know, kind of the the founding patrons there contribu all contributed a million bucks to the the deal, which establishes kind of the the the corporate entity that can help drive Omarchy forward. Contribute to projects that help that vision, but also allow us to do a lot more. It allows us to, you know, work with other companies and things where it's not just, you know, some individuals doing something and there's not really any way to to structure those deals or or make all that sort of stuff work. so I think that, you know, we've definitely gotten along the way, we've gotten a ton of people who've noticed. it's funny, I I mentioned in one of the chats the other day that if we get if we get Linus from Linus Tech Tips to convert over or just try Omarchy, we've successfully gotten pretty much my entire YouTube subscriptions list. last week was Network Chuck. Network Chuck made a fantastic video. on YouTube where he you know, he finally tried Omarchy and he was showing off all the new features and, you know, this is after the launch of Omarchy Quattro, it's got all that new stuff in there. with like the AI crash detection and the plug in system and, you know, all of this. And You know, I was like, damn, I've followed him for a long time. mainly as it relates to like home lab and that sort of stuff. But, you know, slowly but surely those those videos have popped up from a lot of those people. So that's been a really cool, you know, kind of thing to to see. I think that a lot of people have noticed. I mean, it's hard not to see it at this point. It's made a lot of noise and you know, really we're looking to capitalize on that now of you know, how do we take all of this momentum and ride that momentum with a lot of positivity with a lot of you know bringing new features, responding to the market itself, because you know, that has been insane. addressing needs for things like, you know, we had the plug-in system, which was kind of a reaction to the need for people to, you know, modify their systems, but also be in a way that we can we can push updates to the core without conflicting with customizations. People want. 'cause in the, you know, in today's age, right, the the vision of like the malleable computer is like, we have AI. And we have AI that can do really cool stuff. So you should just be able to sit down with Claude and be like, you know what would be really awesome if I had a widget at the top right that like, you know, I could monitor the cameras outside of my house on some specific system. And you can work with Claude to make that thing that is bespoke to you and be and really useful for you so that it makes your computing experience the way that you want it to be. we've also dedicated a ton of time to you know making AI kind of a first a first class citizen as it relates to Omarchy. That's probably something I should have mentioned earlier, where

Yeah, that's very unique.

you know, we really want it to be. Probably the like the agentic AI operating system, where an example I kind of mentioned i as kind of a first pass of this is you know, if there's if anything crashes in your system, there's a little notification that pops up and you can click on that and it'll if you have a default harness configured, it'll pass that over, and we have a skill that'll help diagnose that and basically say, like, hey. Figure out what went wrong here. Now, plenty of people had this, like, right? Windows/Microsoft has had that button forever. It never fucking does anything. this one passes it off to Claude or or Codex or, you know, a local LLM if that's what you have configured. so you can actually diagnose and fix your problems. And in some cases, you know, it'll in most cases that we've had, it'll actually find the problem and be like, yeah, this crashed because of this. I'll push this, I'll make this update or refresh this package or, you know, delete these temporary files, then you're good go. I think there are a lot of other ways that we can bring that in. The other one is Tobi pushed the the new like Quake Quake inspired UI, where you remember back in I think they did it in in some other games too, but in Quake you could hit The tilde button and it would slide down like the commands you could type.

Mm-hmm.

so that same thing. If you do like super tilde, or I guess it's technically not tilde, it's like super back tick. it'll slide down kind of a pane over top of what's on your screen with your default agent open.

Hm.

So if you just if if we're sitting here and we're like, you know, I need to do something and I just need to summon my agent real fast and and like start something and and it's not something that needs to be long tailed or whatever, I can I can just hit that shortcut, it slides in, it's just there. I do what I need to do and then hit it again and dismiss it.

It's nice that it's embedded into the OS so you don't have to go try to find, you know, your favorite things, actual desktop application. It's it's just hooked to a harness and you're good to go.

Yep. we come with first class support for like herder, which is like, you know, the harness to be able to run multiple agents in and and keep track of them and get notifications. So like, you know, there's a lot of thought put into, you know, how how are people working with AI agents today? How can we make that a core piece of the operating system so that it's a positive experience and we can maybe even push it further. In addition just using it for tooling that we're use we're using, right? We're getting, you know, pull requests with the plugin registry. You know, we launched plugins and in the course of a week have a thousand custom plugins, over a thousand custom plugins that people submitted. that's insane.

That's insane.

But it's also necessary that every one of those get reviewed. So we have kind of a combination of, you know, agentic review. processes that we can put things through and also or deterministic things processes that we can put things through and also agentic tests so that we can you know have AI agents help us find if you know somebody's trying to push something up that's that's malicious or seems out of bounds. because we really don't want users, you know, installing malware to the extent that we can

So so tell me what like what have you learned through this process? Because I mean, obviously you've been developing software for a long time, both for you know Oodle, which is our primary day job, you've contributed to other open source projects, but this is this is a a different level open source project in terms of your involvement and size and scale. So what have you learned in this journey so far?

A lot. You know, I've I've I have a a pretty wide experience of, you know, software and software development from kind of my past and you know, s quote unquote day job, right? along with kind of the business experience that comes along with starting and owning and and building a business for almost eighteen years now. Even beyond that, learning about There's some stuff that I've learned from a skills perspective that, you know, I've dabbled deeper in operating systems and the the computing level layers and even down to the Linux kernel. That was something I'd never done before. So, you know, seeing how all of that comes to fruition, and seeing, you know, how Linux kernel patches work, or how package managers Like how the package manager for Arch works or the orchestration that the Arch team has and how they manage, you know, all their processes. And there's been a lot of that stuff that has been really interesting and eye opening, both in a good and a bad way, right? I think when you're we don't know any better, you assume that things are generally more complicated than than you understand. there's a lot of stuff that I've found that It just really isn't as complicated as I always thought it was, or there's not as much formality in certain things as I would have thought there was. so those have all been, you know, some pretty interesting things. The most fun ones I think have come from just like, you know, being able to experiment kind of at that root layer. And that's a big change, I think, that like working with Arch. as the foundation brought to the table versus working with Ubuntu. where like it kind of feels like with Ubuntu there's only like a certain layer that you go to and then you're kind of you kind of stop. Whereas with Arch we kind of go all the way down to the hardware. even going so far as to when the new Dell XPS and with the Panther Lake chips shipped, you know, nothing worked. So we work directly with Dell and the Intel engineers to create the kernel patches and create our own kernel, like like maintain our own version of the Linux kernel that has patches necessary for the XPS laptops to work. so again, just that whole process and thinking about like, you know, having your own kernel is something that, you know, three or four years ago, I'd be like, there's there's no way. That's super hard. And now I'm like, yeah, sure. No big deal.

Times have changed, for sure.

Yeah. I mean we manage we manage our own we have our own package repos, we manage our own mirrors, like everything you could you know, at this point we own the whole system almost end to end. there's certainly, you know, a good amount of of things that we rely on the Arch team and Arch for still. you know, that's still the core the core seed that we kind of dr we branch off of. so the work that goes into that is huge. we still have you know all of the Arch packages that they build and then we you know have a separate mirror that we maintain in some ways that we sort of maintain a rolling release schedule, but in a in a more in a slower fashion. So like Arch Linux is a rolling release distro. that like it's bleeding edge all the time. Right. When somebody ships something new, it goes through a build process and then it's just out. The libraries move forward. There's no there's no maintaining legacy libraries. and that's why it has a tr it has a reputation for shit just breaking. Because it will, right? If if the core library for a while, the Bambu Studio app wouldn't work. Because the library, one of the libraries that it relies on, it updated to version eighteen or whatever the version was, doesn't matter. And the Bambu Lab application hadn't been repackaged to work with that version yet. So it would just fucking crash. It's, you know, we're on eighteen, it's looking for seventeen, and it crashes. And it was that way until, you know, the package maintainer fixed it. That's the reputation that that Arch has, which makes it a little difficult to build on. That's what's led to some of the, you know, the the fire drills that David and I have had to to fight over the past year. And also led to us creating sort of a a slower ring rolling release cycle is what I'd call it, where we have the idea that, you know, there's a stable there's a stable mirror and then there's the edge mirror. where the stable mirror is really cut at every release. So we know that it's gone through the test process. We know that everything works. We know that everything's compiled on the same libraries. And and you're kind of like freezing time or a copy of the the the package repos at at that time. And that is really good for like ninety-nine percent of users. Because, you know, certain packages we still we put them on what we call a fast ring, where like, you know, Claude and Codex and these things that need to move quickly and are not library dependent. We'll put them on the fast ring so they elevate to stable a little bit faster. But, you know, your core pieces that make your operating system, they really don't need to move until we release another release, which is usually every, you know, three to four weeks. Or if you're crazy, you can operate on the edge ring and some shit might break sometimes. But you know, that has been a learning along the way is that like, you know, while rolling release gives you cutting edge tech to work with all the time, that cutting edge tech, like they call it bleeding edge for a reason. like it at some point it's gonna bite you in the ass. and figuring out how When you're introducing, you know, we have people who are using Linux for the first time ever. So if their system crashes, they're not 100% sure what to do with it. They're not, these aren't people who've rolled their own arch systems for years. so we've had to learn like how do we how do we create something that takes all of the good that comes along with this, but adds maybe a layer of insulation on top of it. So that you have this positive and reliable computing experience and every time you update your system, it doesn't just fucking break.

Yeah, so so tell me about some of those learnings and and give me a story around an emergency maybe that has happened because to your point, you're you're helping people roll in some cases probably their primary new operating system. And so if they install a new soft software update, they expect to be able to work the next day.

Yeah. I would I can give you two. Two that stick out in my mind in a big, big way. and it's funny how often this has happened, right? Like we we've had a number of times where we cut a release, like a major release. And in Omarchy, if we cut a new release, you know, four point one comes out, you get a little icon that tells you you need to update. So Obviously, like there's going to be an influx of people updating during that time period. we've had a comical number of times where before we established the stable ring and the stable mirrors, we would cut a release. We'd go, you know, we'd high five and go to bed. Be like, all right, that's awesome. Like everything's good. We tested it, everything's awesome. We go to bed and wake up the next day. And some vendor of some library that we rely on pushed a release right after we cut

Gosh.

our release. So we invite everybody to update their systems. And now they're getting something we didn't test against that specifically breaks their system. UWSM was one of them. They released a package in the middle of the night, Eastern time. it was like 3 a.m. or something crazy. And that package, the net of it was that your system wouldn't boot. It would just black screen. So David and I cut a release the night before. We're like, all right, this is this is great. Talk to you tomorrow. And I wake up and I'm like, my fucking God, what is going on? And I I think we even comically created like a like a a badge that said like I survived the the UWSM outage of of twenty twenty five. But like that was crazy. the other one that I remember very vividly was the the bootloader that we use is I think it's actually pronounced Lemone is w is how it's pronounced. They like a whole pronunciation section, but I've always called Lemine. so you know, old habits die hard. That there was a specific order that it used to load the config files in. So we positioned ours at the end. So that if you happen to have any of the other ones through like Arch install or whatever, ours would take precedent and it was it was positioned to be like the highest priority. in a release. not indicating a breaking change. It was just a a point release. The that got reversed. So what was previously the highest priority is now the lowest priority. and at the time, you know, we used the default Arch install flow. So it also dropped a Limine config during its process. And then we would kind of harvest that and drop ours. And didn't we didn't clean it up. Like we should have, we didn't. and it That led to a whole bunch of people. Again, we cut a release. I was on my way to a wedding in the middle of nowhere. I had like no I had literally no service. and when I got there, luckily they had Wi-Fi and I was able to connect. But I was like texting with David about like what needed to happen and what was going on, and then cut a guide from there of like how to fix your system if you were impacted by it. Because we cut a release, invited a bunch of people to update their system. They updated their system. The priority changed from the bootloader and completely fucked everybody's system. I think that was the one that I went in. I was like, please don't update your system until we drop until we until we drop a fix for this, don't update your system. Like edit, undo the the release. that might be the one where we were like, okay, we need to really make this stable thing happen. because this is insane.

Lesson learned.

Like we we can't we've done a good job of it. We joked for a while that you know we have kind of a 24 hour service because in a lot of cases he, you know, I'm a night owl. David gets up a little earlier in the morning, not crazierly, but he's like a normal person. I'm a I'm a bit of a night owl. and often he would be in Copenhagen and I would be here in in Florida. So when he's waking up, we would have like a couple hours of overlap in the morning when he was getting up and I was getting ready to go to bed. And then, you know, we'd have a chunk during the day and then we'd kind of overlap at the end of the day where he would go to bed and then, you know, I've still got six or sea six or seven hours. And then so we'd have that kind of handoff conversation of like, all right, here's here's where I left off, talk to you in a little bit. but eventually, yeah, that twenty four hour support ran cycle got a little old. where we were like, okay, we need to stop dealing with fire drills so that we can really focus up here. And you know, to I guess our credit since we launched that, everything's stabilized substantially. update cycles are stable. We don't have those, you know, panicky moments where you know someone we we push an update and then one of the one of the libraries pushes an update and then now things aren't working as intended. we're able to hold some stuff back if we don't want to progress forward. it's given us a lot of control over it. In a way that we didn't before. But it also means that like we have to control a lot more. We have to manage, you know, our own package mirror and basically three iterations of it, because we also maintain an RC version. we have our own our own package manager that you know we rely on some packages from Arch User Repository or that are not part of, you know, Arch core. So Arch builds, you know, all of their packages that are in the main mirrors. but the Arch User Repository is really just a d instruction set of how to build the package, which takes a lot longer to install, but also, you know, you're susceptible to a lot of different things. So we take all the ones that we use in the o in Omarchy and we build them so that they're they're pre-built. You you're just installing the built assets so that we know what was built, we know when it was built, we know that it's signed off on, that it was built in a good environment. and it's just faster. Because you don't have to rebuild it on your system, especially for people that have, you know, slower systems. you know, if it takes 10 minutes for it to build on my system, which is a 9950 X3D, well that 1.4 gigahertz Intel Celeron is gonna have one hell of a time trying to build that thing. so if we can just pre-build it. It just installs and it's literally just I mean it's effectively a an IO operation at that point.

So we're we're approaching kind of the the end of of time here. I have one final question for you. How does this impact your day job? Meaning

Mm.

what can clients at at Oodle benefit from from your involvement in in a project like an Omarchy?

I think You know, there's a lot of push and pull that happens from that perspective. Sometimes it impacts my day job in that like there's some shit going on and I'm like, well, we'll get to that later. so that's not necessarily great. you know, a lot of times there there's a ton of the stuff that I've learned from, you know, being this deep into this into the technology stack or interacting with you know, AI in the way that we're using it, dealing with things at the scale that we're having to deal with them. it all translate over because we've learned I've taken learnings from here and applied them to how we manage projects, how we manage teams, how we interact with clients, the technologies that we're using. You know, they've all kind of been battle tested in an arena where you, you know, have a lot more volume and opinions. The other thing is it it it has given is definitely given me access and introductions to a ton of really, really smart people. and I've gotten the chance to work on a lot of really cool projects or or at least give opinions on things that people are working on. so that general awareness of like what's what's out there, what's going on, who who is being successful, how are we being how are they being successful? that lens I feel like I have a better understanding of where things are at and you know where things are headed. sometimes it's it's hard, especially in you know my world, to not feel like you know, like AI is everywhere and everybody's it's like what everybody's talking about all the time. So you know a big thing that we've talked about amongst our team is it's really easy to feel like you're falling behind. and, you know, have this sort of like major FOMO issue if like you don't have agents running twenty four hours a day and you know, you don't you haven't heard of some framework or you haven't used this model or you don't have access to this model yet. And just the reality of like backing up and realizing that there's a lot of people that don't have any they like they don't have any experience with AI or they're or they're they're still at the training wheels level. it's not quite as dire as we would have it make it out to be. And sometimes just having agents running for the for the sake of having agents running isn't beneficial. so you know, being able to kind of check yourself a little bit.

I'll take it one step further, especially in the agency world. I think they're the vast majority of agencies that think they're good at technology are like six months behind where where we are from a technology standpoint because of your involvement in projects like an Omarchy, where we're able to push the boundaries, where we're able to interact with best in class engineers that are working on some of the the the most prestigious technologies and tech and tech companies in the world where you're able to learn from those things and bring those things to to our clients and bring it to our own in internal operations. So

Yeah. I mean it it also works well with, you know, how we've always done things. Like, you know, we're not afraid to tackle you know, things that maybe are a little left of center. Because we understand the underlying technology, right? It's it's one thing to be able to, you know, use a tool or build something on top of a tool as long as everything fits nicely and neatly. It's different when everything doesn't fit perfectly and you go, Well, we gotta rip the rip the hood off of this thing and figure out how it works and how we're gonna make these two talk to each other and speak the same language and you know, but also do it, you know, securely and and in ways that are performant and friendly for users and you know all the the different things you have to consider. It's just kind of second nature where, you know, we don't have an option but to solve the problem. you know, it's like I I mentioned the plugins issue. There are a thousand submissions in a week. And we're like, well we can't just publish all these 'cause we don't know what's in here. And we can't just not publish any of them. So we have to find a solution. And you know, failure kind of isn't an option. We have to find a solution. And we would we did. We found a good solution that works really well. We've got another solution even in the works that'll be even better. so you know, even that sort of attitude and being able to work on that, I think, is hugely beneficial. and for me just a good reminder that gets reinforced amongst our team often of like there's always a solution.

So there you have it. Most agency owners and founders play golf on the weekend, maybe tennis, maybe watch some sports. Ryan builds whole ass operating systems for fun.

I mean it's fun.

Yeah.

Well, till next time.

next