Episode 16 - AI Loops...This is the way

Welcome back to another episode of Not Brothers Podcast where today we're talking about loops. You can't go anywhere on the internet right now without Anthropic beating you over the head with the idea that you're doing AI wrong and you should be using loops. But what the fuck are loops and how do they work and how can you use them in real terms? And that's what we're gonna talk about. Probably some other stuff, but mainly that.

We were talking before this that loops aren't exactly a new idea, but what we are finding is that they're more effective than what maybe most people realize, especially in how, you know, modern AI works best. You know, you were saying in the in the before times we we used to give a you know, an an agent a prompt and you'd wait for it to respond back and then you would correct it and nudge sort of along the way. And that's That evolution has changed pretty rapidly this year of, well, actually the best way to get what you're trying to to achieve is to give a set of success success criteria in the same way that you would a capable human to say, this is what the end result should look like. Now please find a path to get there.

Yeah. It's interesting 'cause we it's not terribly dissimilar to like how we've always worked, but I think that the AI workflows that we've had to date have kind of trained us to get out of this habit, right? Where It used to be that you would spend a large portion of on any sort of like technical project, you'd spend a large portion of that time coming up with the the requirements and vetting those requirements and breaking that down into user stories and you know, playing planning poker and all this other bullshit that we used to do. and then you would take all of that and a team would execute that. And the value was seen, there was a lot of value placed on that upfront activity. Because that's what saved you all the time and made sure that you, you know, you weren't expending resources and I think that on things that you should be doing. And I think that the the before times work loop of you know with AI, we've sort of gotten in this habit, or a lot of people have at least, of st starting down a path and saying, you know, create this thing. And you look at it and you're like, okay, that's the I have the frame. It's like, okay, now add this piece to the frame. Now add this piece to the frame. Now change this. I don't like this. And rather than doing the upfront work to try to figure out, you know, what am I trying to construct here? You're sort of constructing it one piece at a time and then and working through those flows as you go. And there can be some advantage to that for exploratory efforts or just you know, other things, but by and large, you're really benefited to kind of think through those on the upfront for the same reasons now. I think in addition, we see that you know the kind of the first step in this sort of like looping direction was in a big way I think was open claw, right? Open claw largely was just a it's just a loop, right? All it's doing is it's a it's a cron job that just runs every every so often to trigger the agent to do things. And then you were encouraged to set up additional sort of heartbeat and loop activities to run. And that would just trigger agents on a periodic basis to perform some set of tasks and and whatever. Right. I think going a step further, the the model makers have sort of baked that into the core of the system. So looking at GPT-5.5 specifically, the prompting criteria directly from OpenAI, right? OpenAI, the creators of that model, are telling us that when you're working with GPT-5.5, it's a bad idea. To harp on, you know, what you don't want and exactly how you want it to do something. You're it's really designed and works best if you just give it its goal and you tell it, you know, here's the success criteria, here's your win conditions, and it will achieve those win conditions by whatever means it needs to. And I think that's that's showing sort of that optimization, right? GPT-5.3, I think it was. When you plugged it into even something like OpenClaw, it was infuriating. And working with it directly was infuriating as well because it would just stop. It didn't it didn't want to keep going ever. And it would just stop and say, Are you sure it seems you want me to create this feature? Do you want me to go ahead and create that feature? And you're like, For fuck's sake, I just told you to create the feature. Yes, please do. GBG55 is completely opposite. I had one that I was telling you about ahead of time that you know, I'd spent some time yesterday on a on a Project that we have and a pretty large feature that needs some refinements to sort of outline, you know, the set of win conditions, right? The same feedback that I would give to a human if I knew a human was writing all of that code. I mapped all that out with detail and screenshots and all those sorts of things. And then I was like, you know what? Let's see, let's see what what comes of this. And I just plopped it into in this case, codex. And said, hey, take a look at this fizzy card and execute all the work that's there. And I walked away. And it churned for about four hours. And it did that because I also paired it up with a review buddy, it was a separate agent, that the requirements were you have to meet all the winning conditions that I've outlined, but the review agent has to give you the go-ahead. So every time, and I can see it kind of looking back through the work chain. The main agent would the builder agent would come through and say, I'm done. And then Mammoth would look at it, the review agent, and say, No, you're not. You missed this. This is wrong. This doesn't meet the win conditions. And then it would loop back around. And it did that for four hours where I didn't have to touch anything. And without sending a single follow-up command, without changing, you know, I haven't refined anything at all. My look at it is it's about ninety-five percent of what was asked for, which is honestly pretty fantastic that it's able to do that and saved a it was a lot faster and more efficient than if I had sat there and tried to get to that same end state one chunk at a time by dictating to the AI directly, hey, here's what I want, change this stuff. Instead by documenting it and and then placing it into a loop to execute and finish out. I feel like the work that got done was perceivably faster because I didn't have to do anything for that four hours. I just turned it on, went and walked my dogs, had dinner, played some video games, and eventually it kind of popped up and was like, Hey, I'm done. so it's perceivably faster. and the work is I think higher quality work than I would have gotten otherwise.

So let's talk about that for a little bit because for a long time, and maybe it's still this way, but for a long time, the idea of chunks of work was much more favorable to get better outcomes from AI. And so you would take something that was a small piece of work that might be, I don't know, one of ten steps that you may end up walking through and you would want to achieve the first step before moving on to the second step. So like how do loops change that in in today's world and is that only applicable for models like Codex five five and or I'm sorry, GPT five five and a lot of the other things that are more frontier based as opposed to some of the more what do wanna call them less tokenized or less less robust models.

Yeah. I I mean I think it's still good practice. It I think ultimately what we're finding here, right, is like we're we're we're converging back to just good practices, right? And what's good for what's good for human development, what hat what has been good for human development for years, surprisingly, is good for for AI augmented development, agentic development, right? It turns out when you create a detailed set of of specifications and win conditions and you clarify your assumptions that performing that work is able to be done faster, more efficiently, and to a higher degree of quality. And I think that's that's ultimately what I'm seeing here. I think the same can be true of chunking work. Right. It's one thing to come in and say, hey, I want this gigantic thing that's actually like seven features baked into one. it's a totally different thing to establish, like, all right, here's here's the broad vision. We've broken it down into these seven chunks. And here's sort of the spec and the win conditions and all of those things for each of these chunks. And now we're going to execute each of those chunks in a loop. So I'll take the first chunk, execute that one. Maybe take a maybe review it, maybe don't. Maybe, maybe you just, you know, chunk through all of them. But the scope of what has to be considered at that point in time is smaller. And I think the smaller the the smaller and tighter the scope is, whether you're working with a human or you're working with an agent, I think the rules are the same. You'll get better outputs if you can keep that scope tight and and keep it clean and keep it clear. Otherwise it's it it introduces potential for nuance and it introduces potential for for I think missed or assumed conditions. One of the biggest challenges I think even with the feature that we're talking about is that it's fucking big. It's really big. And there's a lot of intricate pieces to it that are difficult for somebody who's coming into it blind to maybe understand. And and there's even some like nuance To some of the pieces that maybe isn't readily apparent to everybody on the surface. I was having a conversation with somebody on our team about it yesterday, and I was like, you know, the version that we have today, I look at it, it looks like 90% correct to me. You look at it and it's 10% correct. Clearly, there's something that you see that I don't see. And there's which is the nuance of of the equation. It, you know, after some Pulling on some threads, you're like, okay, I yeah, I guess I do see that. I didn't that wasn't readily apparent to me. And if it's not readily apparent to somebody like me, it's a good fucking chance it's not readily apparent to the agent either. And when we're using a genetic workflows, they're more likely to just kind of take the completely ignorant approach and ignore a whole bunch of shit than even your average human would. Like a lot of times I think with requirements, we can also get used to that. Right. We can get used to like, hey, I know I'm working with XYZ person. They're really smart. They're really proactive. So I don't really have to be that good about my requirements. I can kind of like I can get by with just like the basic overview and I can know that, you know, Terry on our team is this way. Like he's gonna read those requirements, he's gonna play 20 questions with me, he's gonna fill in the blanks or at least do his best to. There are other people I've worked with that will not. And I think you can a pretty much assume that most agents are probably not going to. They're just gonna like, Well, you didn't define it, so fuck it.

Even if you did define it, I think one of the more important pieces of this that we should dig into is pairing the the creator agent with a review agent or skill or skills or something as a reviewer is maybe the most important piece of the loop process because otherwise that agent is going to tunnel through brick walls in whatever way it sees fit and will probably miss a whole bunch of the requirements because it's skipping steps it's not even realizing it's skipping. And so the review agent plays QA and like, hey, you're not done. You you said you're done. You're not. And that's that's why in your example, that agent ran for four hours because you had a a specific set of criteria that that likely continued to loop and say, mm-hmm. Mm-hmm, almost still not good enough. You miss this too. Hey, you also miss this.

I think that's the importance of d defining your win conditions, right? I can't tell the loop when to stop if I don't have wind conditions. And so it's one thing to create the criteria and you know, here's what I want. But you do need those specific, like, what are the wind conditions? When can I call this work done? It's like it has to have this, it has to do this, it has to be this. You know, there are specifics associated with that. There's also the stuff that sometimes it's out of scope specifically. And you're saying, like, hey, we don't need to worry about this. We don't you know, one of the things that I have to put in a lot of mine are like, don't worry about legacy migrations, right? Because Codex in sp in particular fucking loves. Anytime that it writes something new, it'll be like, you know, writing legacy shims and stuff to make sure that old anybody using the old code can still get by. And I'm like, dude, we just wrote this sex six minutes ago. We don't need legacy, we don't need to maintain the legacy code we wrote six minutes ago and has never seen the light of fucking day. Delete it and let's move on. And and those are things that like, you know, it'll do that. And then the review agent will catch that stuff because the review agent's coming behind it and saying, sort of with fresh eyes and you know, a prompted or trained pessimistic approach to say, like, all right, assume that all this is wrong. And here's what was supposed to be done. is it done? And you know, you will see for a lot of it, like there are constant issues that it'll find. And some of them can be major critical issues, some of them are nuanced issues, some of them are just silly issues. But the point is it's catching them, right? And those are all issues that don't ever have to see, I never have to look at them, right? Instead of me prompting and saying, you know, update this page so that it has these columns, and then it updating them and putting the columns in, but doing it in a stupid way, and then being like, okay, we'll group them by this. I never saw the first iteration, the second iteration, or the fifth iteration. I just saw the final one. And the final one is correct. So I think that's the that's definitely the power. Now it it again it changes the equation. The the value equation comes goes from the outputs and what you're producing with AI and the act of sitting there and typing to really sort of the knowledge working aspect of it. And I think this is something that we've talked about for a long time. Other people have theorized about is that I think the value we look at it especially things that are are being impacted most by AI. Development and writing code is an easy one to pick on. There are plenty of people who Dario from Anthropic continually says, like, we're not ever gonna need developers eventually. I still believe that that's bullshit. And he's wrong. Now, he maybe has to say that because his share price depends on that. Right? Whether he believes it or not, I don't know. If he does, he's an idiot. But the the value in the part that's been difficult about about development and developing software for years has never been writing the code. That's the easy part. The the hard part, the fun part, is figuring it out. Solving the problems and figuring out how to solve problems that have never been solved before is inherently difficult and challenging and forward thinking. And what we've found in AI agents and the way that they're trained and they exist is they are they're data driven, right? But data driven models are inherently backwards thinking. They can only look at things that have been and existed. They can't look at anything that has never existed. And that's largely the things that take the most time, right? Thinking around the corners, doing the thought exercises like we did yesterday. I did the same thought exercise and drew out four different architectures for the same problem with four different people yesterday. And eventually settled on one that's that I think I believe is really solid and it's the right approach and for all the reasons that have been described. But had I not done that, there are 12 permutations of of ways we could have solved that problem, all of which would have been problematic down downstream. But we caught the problems because we spent time to sit and think, okay, well, what if we if we do it that way? What happens in this scenario? Like, well, I didn't think about that. What happens in this scenario? Like, well, it all fucking falls apart in that scenario. Like, okay, well, well then we would have to have a way to shim for that. And through sort of that whole process, netted out, and like, all right, well, that's a bad solution. Looks like a good solution on the surface is probably one that an AI agent would just immediately jump to.

I think you're pulling on a thread that permeates well beyond AI. And I'll I'll I'll circle to non-AI and circle back. Like I think it's a human condition that we value tangible deliverables far more so than we do knowledge-based deliverables or knowledge-based activity. Right. So I I was thinking we were talking before the show of of examples of even my personal life, right? Where I'm involved in several things from, you know, some some short term some some long term rental projects where we had some repairs to do and the knowledge of the repairs versus the value of the of how to do the repairs. I think it's a fundamental human condition that we overvalue the tangible and way undervalue the thought exercises that go into the why. And you circle that aro around to exactly what we're talking about here with requirements building, requirements, gathering. It's it's actually less important what actually gets written down than the 30 things that got thrown away in the first place. So the the the what got thrown away is the most important thing is the exercises, not what made it to the final requirement set necessarily.

Yeah. I think you're right. And we've seen that before, right? There's the old the old adage, there's multiple versions of it, basically, but like the of like the boiler repairman that somebody hires someone because their boiler's not working, and he comes in and you know, whacks it in on the side with a hammer and sends an invoice for two thousand dollars. And they say, this is absurd. Like it took you 30 seconds to solve that problem. Send me an itemized invoice. It sends an itemized invoice for $100 for the for the trip fee and $1,900 of 30 years of experience of knowing where to hit the pro where to hit the boiler. And you're like, It's a little tongue in cheek, but it is it is the reality, right? Like I've been in this position plenty of times where there's a particular activity that I can do because I have twenty plus years of experience doing that a thousand times, but I can do it really quickly. And oftentimes that translates to people devaluing the the the thing because they say, Well, you can do it really quickly, it should be really cheap. Like, actually. If I can do it really quickly, it should be more fucking expensive. Because now I'm saving you time and I'm delivering you a quality product. Right? It's all three of the of the triangle. Or two of the triangles.

It's it's the reason why even in our our marketing business we've we've gone away from hours based billing. We don't even do tracking time anymore for that reason. It's it's essentially value based billing across the board. This thing is worth in s in some cases we win, in some cases we lose. we hope we win more than we lose is sort of the the equation there.

Well and what we the reason we did that, right, in the first place is we found that as a technology based business, right, we consider ourselves a tech a technology based business that just happens to do marketing. And as a tech business, we spend time, energy, and money investing in tooling and building products and education and enabling our team to do things better and faster. But the old school models of fee for you know fee for time is it directly incentivizes us to be as slow and stupid as possible without getting in trouble. And that doesn't work, right? That's at that's at odds with our core values, and at odds with what I think makes our clients most successful. So we found that you know a better way to to to do that is sort of the the value based billing model that we do have where it costs what it costs. If it takes us ten minutes because we have a tool that we've invested a million dollars into over a few years, that's fine. And if it takes us three weeks to do because we underestimate it, that's fine too. and we'll we'll deal with those as we need to deal with them. And like that has largely worked out pretty well. But again, it places the the value equation on the knowledge and the tooling and how to interact with it and understanding how to use the use it and execute it effectively. I think that's the that's the shift that we're seeing sort of with with loops is like, well, now the value isn't just me sitting in front of the computer typing and generating the output. The value is actually me sitting and thinking and figuring out what the totality of the thing that I'm asking for is. and I think for people who have been using AI today, it's kind of a little counterproductive. It feels it feels a little bit difficult to switch to that model because it doesn't feel like it's probably not as satisfying, right? It it's not as it's not as sad it's really satisfying to get that dopamine hit when you're like, all right, make me a dashboard. And in three minutes, you have a dashboard. And then you're like, move this over. And you get those immediate feedback loops and immediate sort of gratification versus I'm gonna spend a day figuring out a requirement spec and win conditions so that I can one-shot this.

And there's probably some value in the iteration of of even that, right? So like AI can still be your best brainstorming buddy. So if you're at a loss for like, I don't even know where to start to create a dashboard, just give me something to react to. That can still be a wonderful place to start. And then you can use that to back into your actual requirements that you're gonna you're gonna lose for your loop use for your loop process. So I don't I don't think there's I don't think it's one or the other. I do think there's still value in the

Yeah, you can still do some prototyping little pieces. You'll still have pieces that like, you know, even in the one that I was talking about, like there's some small nitpicky stuff. It's like, hey, we need to change how this form works. We need to move this, clean this up. You know, those are gonna be part of that in the process anyways. But the bulk of the work was done without me having to babysit it. And I think that's fantastic. I think the you know, there are definitely tools and you can use AI to help with writing some of those specs, so long as we don't fall in the trap of not reading the fucking specs, which is one that we've experienced where you know we use the tooling to say, like, hey, we've we've made all of these things, and you're like, Did you read them? 'Cause if you read them, you'll see that there's some stuff missing. and then you just have like You it's kinda garbage in, garbage out at that point where if the spec that went in wasn't proper because we didn't take the time to r read and revise and re and and make sure that it was truly correct, the output is also gonna be incorrect because it's gonna f it's gonna follow the input directly. so that's another like little potential pitfall. But kinda I think we beat the the idea of loops and specifically in in the dev world to death. I think the more interesting thing is like where else can we see this paradigm play out? And one of that you were explaining to me is like how you've been using sort of the same idea in a different way, with regards to some more creative activities, specifically with some social content that we've been producing again, 'cause i it it can be a very laborious task to sort of brainstorm, write, edit, and do all of that. And that's something that I think has been able to help with a lot, at least to getting sort of the bulk of that flow done.

Yeah, and and just like in the development flow, it it's not without error, it's not without it its own warts, but there's been a a shift, I think, with the ability for AI agents to do pretty good work even in the creative space, not just writing. It's always been half-decent at writing, but in the visual representation. So creating cards for social posts as an example, or creating consistently themed things for various pieces of content, including this podcast for the shorts and clips that people are seeing. we've developed flows that allow agents to be able to take the recordings that we have, slice them into different pieces and parts, knows what our writing style is and our speaking style is based on brand guidelines, knows the visual representation of how that should take shape in the form of card posts or shorts that end up on our website and and and and With that, there becomes a flow that used to take a half a day to produce 15-ish pieces of content that goes across all social channels. So now it takes about an hour to do that because there's still a manual piece to some of those. But that loop process is very similar in just a different context. So rather than just having rather than just saying go build stuff, I've created tools in the form of skills or agents to do a review. So the review cycle is still the most critical part of the success criteria is on the on Instagram, make sure that nothing is below the whatever, the last 50 pixels. Because if it is, there's going to be an overlay that that goes over that piece of content. And so the review agent or the review skill allows it to to look at that in context and say this meets the success criteria or it doesn't. And if it doesn't, it iterates until it it does. And so that's just one small example in in the social space. Of how you can do something similar with loops.

Yeah, and I think the the benefit is it catches a lot of stuff before you have to. The reality is we're still looking at at least today. we're still looking at everything that goes out the door, we're still making manual refinements, we're still adjusting where necessary, but it gets us to that ninety percent spot with a pretty high degree of accuracy. That also is based on the inputs too, right? The inputs are high quality inputs of, you know, hey, what are we what's our goal? What are we talking about? What are we doing? and in some case, right, like with the podcast one, it's a l somewhat easier because you have the whole fucking transcript. So the transcript is sort of a self documenting, like, hey, what's the purpose of this and and what am I doing with it to try to carve up and figure out how to discuss w what's being promo like if you're posting a clip Understanding the context of what was being discussed is pretty easy because it has the whole transcript there. with other forms of content that we use sort of similar loops on, it's a little more nuanced. and having a review agent that'll come in and try to understand that nuance or at least or call out the fact that like this feels wrong and I'm not sure why. is helpful.

When you're calling out an interesting point, because there there are things that we do in our business that we normally wouldn't do like a video recording for, like, you know, going into a meeting or whatever, that we've intentionally started doing that so that we could replicate some of the loops that we just described for social content. Because it already has that that transcription as essentially the first version of requirements, that level of understanding is already there. So whether you need the video or not, you can throw that away. Use the transcript as as a a way to say, like, okay, well now we can use this. Now we understand tone of voice because we know that we know the speakers. We know who was speaking and in what tonality. And we can take that and carve it up into blog posts or carve it up into a newsletter piece or any other piece piece of content that you might be able to do that isn't just video oriented, but because you have the video, it it allows you to do so much more in a just a a different loop.

Yep. So I mean I think ultimately sort of wrapping putting a nice little neat bow on top of this this whole topic. I think if you find yourself still using AI n ninety percent by typing into a a harness of some kind and then sitting there staring at it until it finishes and gives you something back. It's important to understand that there's another way. And depending on the work that you're doing, it might encourage you to give it a try. And that is producing sort of a a a contract or a set of criteria first, passing that in, pairing your agent with a review agent, and sort of a set of instructions that says, like, hey, just loop on this until you're finished. And see what you get out of that, because my guess is that it'll probably free up some mental cycles and time that you normally are spending just staring at the screen and the quality of what comes out of it and the the ability to get to the end state will probably happen faster. and that's true of of larger development projects and development features, creative idea idea ideation. social post creation, cre you know, any of those anything you could use it for. the other thing is I would encourage people to just set up small loops. Like start, don't try to set up crazy fully automated workflows. Like I think that's, you know, when you go on things like X and other w otherwise like everybody's showing you their fifty agent setup that, you know, is life changing and whatever. And, you know, I'm still very adamant that all those motherfuckers are accomplishing is a lot of maintenance tasks. They're not actually accomplishing much work. the ones that I've seen accomplishing the most work and being the most efficient are the ones with like one or two agents. and a few loop sets or something that are are running. So, you know, simple simple things like we have a loop that runs every week that posts our standup notes. We have a loop that runs consistently that will transcribe in a better form and post meeting notes for any meeting that it's that's shared with it. we have a simple loop one that I've experimented with with on some repositories is just rebasing pull requests. So that, you know, eventually when I get to reviewing that pull request, if it's fallen out of if there's a merge conflict, I'm gonna have to rebase the fucking thing anyways. So this has changed it for me and then I don't have to I don't have to think about that. It's just already rebased by the time I get to it. So starting with, you know, simple workflow loops like that and then working up the sophistication chain, I think, is the right approach to something like this.

Till next time.