you2idea@video:~$ watch EsTrWCV0Ph4 [2:13:14]
// transcript — 1997 segments
0:02 Hey, this is the definitive course on AI agents. I currently teach over 2,000
0:05 people how to use AI agents in both their personal and business lives and
0:09 run a business that does over $4 million a year using AI agents. So, you don't
0:13 need any programming or pre-existing computer experience in order to make
0:17 this course work for you. I myself don't have a formal computer science degree.
0:19 I've learned everything that I know watching free resources like you're
0:23 doing now. This is also a general AI agents course, so you don't need to know
0:27 any specific platform. This isn't just on codecs or claw code or anti-gravity,
0:31 but rather on all of them. So, wherever you guys are starting, you'll end up at
0:34 the same place. No fluff. Here's what you're going to learn in this course.
0:37 First, I'll show you guys a demo where I'm controlling five AI agents, each
0:41 with their own Chrome browsers as they interact with the web and perform
0:44 economically valuable activities for me. I wanted to frontload this course with a
0:47 demo so you guys could see what we're working up to. And just a few months
0:50 ago, what I'm doing here would have been considered absurd. Then, I'm going to
0:53 cover the core AI agent workflow loop, which works independent of which
0:56 platform you're using. After that, I'm actually going to talk about and then
0:59 sign up to the three major AI agent platforms right now. So, I'll sign up to
1:02 codeex to anti-gravity and then cloud code. And then after I'll cover what
1:06 each platform is at the moment the best or the worst at. Then we're going to
1:09 dive into foundational AI agent prompting techniques. So, selfmodifying
1:13 agent instructions where the agent will rewrite its own rules to minimize the
1:17 number of errors made. Multi-agent MCP orchestration which is where we'll
1:20 register codecs, Gemini, and Claude as MCP servers so you can manage multiple
1:24 agents within a single conversation thread. video to action pipelines where
1:28 we'll teach agents to learn from YouTube videos instead of plain text alone.
1:32 Stochastic multi- aent consensus where we'll spawn multiple agents with the
1:35 same prompt and then use their statistical spread in order to ideulate
1:39 and improve things better. Agent chat rooms where you'll build centralized
1:42 places for agents to debate ideas, pushing them to much higher quality
1:46 answers than before. Subagent verification loops where your agents
1:49 will actually review each other's work in real time to catch things that one of
1:52 them might have missed. We'll talk prompt contracts. I'll show you guys
1:56 reverse prompting and a bunch of other techniques as well. And finally, we'll
1:59 chat about context management and improving the agent output quality
2:03 before closing out by discussing how to optimize um AI agent and then token
2:07 pricing. So far, I haven't seen anybody on YouTube discuss most of what I cover
2:10 in this course. So, for all intents and purposes, you guys consider this the
2:13 sauce. Please bookmark this video, subscribe to the channel, and let's get
2:16 into it. First, I want to show you how powerful these agents can be when you
2:20 learn how to distribute work across multiple Chrome instances and give each
2:24 sub agent their own workspace. What I have here is a simple list of leads
2:30 from, let's just say, a conference. Now, we have fields like their websites,
2:34 their LinkedIn description, their first name, their last name, but one thing is
2:39 missing, their email address. Now, just a year ago or so, that would have
2:43 invalidated my ability to reach out to these leads. But now, because I possess
2:48 their websites, I can actually spawn a bunch of Cloud Code agents, have them go
2:51 to the websites, and then have them interactively and dynamically fill out
2:56 their contact forms. So, what just happened as I was talking was Claude
2:59 went ahead and then opened up a bunch of different Chrome browsers for me. I'm
3:03 going to rearrange these to make it really easy to see. And so, this might
3:06 be a little bit tough to see, but what these agents are all doing is they're
3:10 independently navigating over to the contact fields of each of these
3:14 websites. They're then dynamically filling out fields like the first name,
3:18 the last name, the email address, and so on and so forth. And then they're
3:22 putting in a little bit of outreach that's templated, but then changes
3:25 depending on who they're reaching out to. These agents, through a combination
3:28 of both research and then communication between each other in a shared chat
3:32 room, are capable of doing things that any one agent might have taken many,
3:35 many hours to do before. This is what I'm going to work up to with you guys
3:39 over the course of the rest of the next couple of hours. The main strength of AI
3:43 agents is really their ability to parallelize, which is to run multiple
3:48 instances of each of them simultaneously while they accomplish a task. Now, right
3:53 now, I would say most AI agents aren't as intelligent or as capable as a human
3:58 being for any given need. But what they are much better at us then is being
4:03 fast. And so despite the fact that their accuracy might be a little bit lower
4:07 than a human, their ability to oneshot stuff is worse than ours at the moment,
4:11 they can run multiple instances of themselves simultaneously and try
4:15 multiple approaches over and over and over and over again in order to
4:19 ultimately achieve much better results than we can. The key is you need to know
4:23 a little bit about how they work under the hood. Then you need to be able to
4:26 combine them using elaborate prompt architecture like I'm going to show you
4:29 in this course. So, why don't we start with one of the simplest, most
4:33 foundational concepts before I actually guide you guys through signing up and
4:36 setting up these different agents. And I call this the core agent loop. To make a
4:41 long story short, I think most of you probably have intuition about how agents
4:46 do things. But really, what they're doing at the end of the day is they're
4:50 going through a loop over and over and over again. And this loop is composed of
4:56 three major functions. The first is the observation step. And so here the agent
5:01 is basically reading through all of its context. We're going to chat a little
5:05 bit more about how to optimize and manage that later. That includes things
5:10 like its files, its previous tool calls. It includes all of the system prompts,
5:15 the clawed Gemini, and agents.mmds that you provide. If it does research in a
5:19 previous step, it'll include the research from the internet. uh if you're
5:23 feeding in multimodal data like vision data, camera data, uh you know, audio
5:27 files and so on and so forth, it'll include all of that. And so this agent,
5:31 okay, is just in an environment and it's just always observing what's going
5:35 around it at least to start in the observation step. From there, it'll
5:40 reason. And so this is the think step here. It'll consider based off of all of
5:45 this context and based off of, you know, the user's highle goal, what do I do
5:49 next? How should I plan my approach? And nowadays, most agent coding platforms
5:55 make use of like a dedicated reasoning step that you can actually click into
5:58 and see, which I'll show you guys a little bit more of. And this provides a
6:01 tremendous amount of interpretability, accountability, and then steerability,
6:05 which is really important that I think most people sleep on. After it's thought
6:09 about things and basically wrote its own mini plan, it's time to actually act,
6:13 right? And so, here's where it'll call tools. It'll edit the files that it
6:18 decided to uh do so earlier in the plan. or maybe it'll run a command using
6:24 command line interfaces, CLIs. After the action step is done, what it does is it
6:29 gets the result of the tool call and then it feeds all of that stuff back
6:33 into the observe step. So now we're basically running through that loop
6:36 again, just with a little bit more context. And so what occurs essentially
6:41 is we just tend to grow bigger and bigger and bigger and bigger. If our
6:46 initial context was a certain size, our you know second loop it's a little bit
6:50 bigger, our third loop it's a little bit bigger and fourth loop and so on and so
6:54 forth. And what this is doing is this is basically stacking more and more tokens
6:59 into the context that the model can then use to plan its next step. What occurs
7:03 after you go through this loop you know usually three or four times is
7:07 eventually the model reaches a point called the definition of done.
7:13 And what the definition of done is, which I think a lot of people leave out
7:16 of their agent prompts, which is probably why they're always underwhelmed
7:19 by what happens, is it's the series of constraints and technical specifications
7:26 required for the model to conclude that it no longer needs to do this loop. Once
7:31 it reaches this definition of done, okay, over and over and over and over
7:36 again, it notices and then it changes routes. So, now it goes to the task
7:40 complete route where it generates a quick little final response for the
7:43 user. Usually involves a nicely formatted answer, as I'm sure you guys
7:48 know. Hey, Nick just finished your new thumbnail app build. And before
7:53 outputting it in a window either in anti-gravity or codeex or maybe cloud
7:58 code in a packaged way that you guys are familiar with. And so obviously if you
8:04 have any intuition about how AI works at this point, if you've ever communicated
8:09 with chat GPT or you know Claude or some other sort of desktop AI that's nestled
8:12 into another application that you guys use, you'll probably know some of this
8:16 stuff um just as like the foundation. But I wanted to make it really explicit
8:20 at the beginning of this course because we're going to return to each of these
8:23 steps over and over and over again. And it turns out that you can heavily
8:27 optimize all three of these. You can optimize the hell out of the observe
8:31 step. You can optimize the hell out of the think step. And understandably, you
8:34 can optimize the hell out of the act step as well. That's what we're going to
8:37 learn. Another point I'm going to make in this course is that AI agents aren't
8:41 just the large language models themselves. You know, I think neural
8:46 networks and transformers are obviously super inherently interesting because
8:50 they're these massive statistical things and these beings that can that can do
8:54 things. They can reason. They're very far removed from traditional computer
8:58 programs. just 5 or 10 years ago. So, a lot of interest goes to the LLM, but I
9:02 want you guys to know that the LLM really is just a very small part of what
9:06 most people consider AI agents these days. The LLM is of course your
9:10 reasoning engine, right? Of course, it understands language and of course it
9:14 makes decisions, but it's kind of like a human being from like 20,000 years ago
9:19 with like a spear in its hand, right? without all of the infrastructure around
9:23 human beings, without like your your house and your fireplace and your hearth
9:27 and a place to sleep at the end of the night and a a society where people farm
9:32 and produce resources and you have cars that you can get and you traverse a lot
9:34 of distance without all the tools and the architecture around the
9:37 intelligence. The intelligence is actually quite limited in what it can do
9:41 and that's where the rest of these sections come into play. So tools much
9:46 like human beings have the ability to read files, run code, search the web,
9:52 call APIs and edit files. Okay, so too does this AI agent. Much like human
9:57 beings have the ability to set a highle goal and keep going until that task or
10:02 goal is reached, you know, so too can agents. And much like human beings have
10:06 some sort of persistent memory where we can keep track of things that we've done
10:10 and then realize that some of those things didn't work. So we got to take a
10:13 slightly different tack the next time. So too, agents have things like
10:18 agents.mmd, claw.md, gemini.mmd, access to their conversation history, access to
10:23 automemory files and skills. And so it's not actually just the LLM, for instance,
10:28 that makes an agent work. It's really all of these things multiplied by the
10:32 fact that, you know, the LLM provides us like the ability to be a little bit
10:35 flexible. And that's a really big different from just, you know, a chatbot
10:39 and then an AI agent. A chatbot might just be the LLM, okay? But an agent
10:44 takes that that LLM and then it adds on tools, a reasoning loop, memory, and so
10:48 on and so on and so forth. So, as a brief example, I'll use an agentic
10:53 coding platform called Codeex. And down here, I have a simple prompt where
10:56 basically I just want this to do a bunch of research for me on creatine
11:00 supplementation in men. And what I'm doing is I'm giving it a brief
11:04 definition of done where I'm saying once you've compiled 10 plus empirical
11:08 sources, return a structured report. And I'm doing this because I want to
11:10 demonstrate this loop to you. And so there are a bunch of other things that
11:14 are popping up here. We have the actual chat window up at the top and we have
11:17 its response. But you'll notice that in between we have this sort of like grayed
11:20 out section here. Okay. And this grayed out section is the thinking that the
11:24 model is doing before it gets back to us. And so basically, you know, if this
11:31 was chatpt back from 2022 or so, all we would have gotten is this. But because
11:35 I'm telling it to take actions in the real world, it's capable of one,
11:40 observing, and so it observes all of this text and all of its reply as
11:46 context. Two, thinking. So it's capable of doing a bunch of thinking on what to
11:51 do next. And then three, acting. And so then it's capable of saying, hm, the
11:54 user probably wants me to do some research. I have access to a few tools
11:57 available. One of the tools lets me search the web. Let me pump in a search
12:01 term. It then compiled all of this information and then it just repeated
12:05 the same thing. It then with all this context said, "Okay, I'm observing. Not
12:08 only do I have these messages, but I also now have a bunch of research. Let
12:12 me think about what to do next. Have I achieved the goal of the user compiling
12:16 10 plus empirical sources?" And you know, after it's made its sort of
12:19 observation and thought and reasoned about it, then it's deciding to act. And
12:22 what it's ended up doing after 58 seconds is giving me this structured
12:26 evidence report. So, this is an example of something that might have looped two
12:29 times, three times, but the more intelligent and capable these models are
12:32 getting um the longer that they're running autonomously without us.
12:36 Hopefully, this isn't rocket science to anybody here, but in a nutshell, this is
12:39 more or less what's always occurring non-stop every time you talk to a model.
12:43 With all that being said, let's really quickly cover how to set these different
12:47 models up. I'm going to be using Codeex, Claude Code, and Anti-gravity. You don't
12:52 need to know anything about any of these platforms in order to run these
12:55 examples. And if you're already very familiar with, let's say, I don't know,
12:58 Claude Code and you've chosen to use that as your main agentic coding
13:01 platform moving forward, you can skip over to the next section of the video.
13:04 But I want to make sure that we all have an equal playing ground here. We all
13:08 understand how each of these platforms work under the hood. So there are three
13:11 major platforms. The first is Codeex, which is owned, managed, and run by
13:16 OpenAI. The second is Claude Code, which is owned, managed, and run by Anthropic.
13:20 And the third is Google's anti-gravity, which as I'm sure you can imagine is
13:24 owned, managed, and run by Google. In order to start with Codex, what you
13:29 first have to do is sign up to an OpenAI account. The way you do so is just look
13:33 up OpenAI on Google, get to a page that looks anything like this, and then just
13:36 go to the top right hand corner where it says try chat GPT. After that, you'll be
13:40 taken to a page that looks something like this. You can continue with Google,
13:44 your phone, or whatever you want. And if you choose to chat with the model and
13:47 then come back at any point in time, just head to the top right hand corner
13:51 for that modal again. So I'm going to pretend that I haven't made an account
13:53 before and I'll continue with Google. After some brief onboarding
13:57 instructions, you'll have access to a page like this. But this is just chat
14:01 GPT, which is more akin to a chatbot than anything else. We want to take this
14:05 to the AI agent world. And so in order to do that, we need to use their
14:09 dedicated AI agentic coding platform, Codeex. So googling OpenAI's codeex or
14:13 something like that will take you to a page that looks like this. And then you
14:18 can just click download for Mac OS. By the way, I'm on a Mac, so that button is
14:21 automatically going to pop up for me. But the Codeex app is now also available
14:26 on Windows starting March 2024 and beyond. The way you install things on a
14:29 Mac is you just take this window, drag Codex over to applications, and then
14:32 you're done. Once you're inside, if you wanted to build a website or something,
14:36 just head over to this middle, create a new folder, call it whatever you want.
14:39 So I'll just go to downloads and then go new folder example.
14:44 Open it within it. And now you're inside of this folder here. You can ask the
14:47 model to do whatever you want. And so what I'm going to say is make a brief
14:51 portfolio site about Nick Sariah. Keep it super simple and minimal. It'll now
14:56 do some thinking. In our case, I actually have a design taste front-end
14:59 skill which improves its ability to create like sleek, highquality looking
15:04 designs. And now it's looking through my own workspace to put together this cool,
15:08 sexy site for me. I'm also going to ask it to open it. Uh, and the way that all
15:13 AI agent platforms work now is you have the ability to put a cued message in,
15:18 which you can also choose to send immediately via steer. In my case, I'll
15:22 just wait until it's done. It'll consume this open it message and then it'll just
15:26 open it for me in a new tab. Once it's done, the open it message will be fed in
15:29 and it's just going to open this for me in a new tab. Now, I'm kind of zoomed in
15:33 here. So, if I zoom in a little bit more, you'll see that this is just a a
15:37 simple one-page site that says Nyx Drive builds clear modern digital work. Here's
15:41 some information about me. And here's a contact page. Not rocket science, but
15:44 this is how easy it is to like build web stuff. Claude is pretty similar. Just
15:48 Google Claude signup or something like that. And you'll be taken to a page that
15:52 looks like this. Here you just enter your email address or in my case,
15:55 continue with Google. In Claude's case, in order to use Claude code, you do have
15:59 to pay for it. And so there's a pro plan here that's $17 per month with an annual
16:04 subscription or 20 bucks if build monthly. I'm not working for Claude or
16:09 anything like that. I don't have any sort of affiliation with Anthropic in
16:14 that way. But I will say that I receive probably a 100 to 200x return on my
16:18 investment with an Aentic coding platform, whether it's Claw or whether
16:22 it's Gemini or whether it's Codeex. So, my recommendation for you, if this seems
16:27 a little bit steep, is bite the bullet, pay it, and learn whatever you can to
16:30 make a return on investment with that money in the first month because this
16:34 stuff is really quite powerful. Assuming you're done, just type cloud code
16:38 desktop download or something like that. And you'll be taken to a page that looks
16:41 like this, which allow you to download it for Mac OS, Windows, or even Windows
16:46 ARM 64. So, I'm going to give my Mac OS thing a quick click. Then, I'll go to
16:49 the top right hand corner. I'll just open Cloud up just like I did with
16:52 Codeex. That'll take me to a page like this. And then I just drag this over to
16:55 the right. And then once you're done, you'll be taken to a chat page that
16:58 looks something like this. What we really want is we want this code button.
17:02 So I'm going to give that a click. Then here, all we need to do is just choose a
17:06 folder to work in. And then we can put in a quick request. So I'm just going to
17:10 choose a general folder next. Then I'm going to say bypass permissions, which
17:13 might seem a little bit scary to you, but it just makes the model act
17:17 independently. Then finally, I'm going to say, hey, make a brief portfolio site
17:22 about Nyx Drive. super simple and minimal. And so, just like Codeex
17:26 designed it a moment ago with its various UX uh features, we have the same
17:30 thing here with Claude Code. It's going to ask to access some files in my
17:34 folder. And in addition to having the message box, we also have this sort of
17:39 grayed out shining decal here, which is sort of its like thinking if you think
17:43 about it, as well as its tool calls. And what it's going to do now is actually
17:47 build me a brief little site. And then, just like I did before, I'll just say
17:50 open it. That's going to ceue it and now I can have a conversation with Claude and now
17:56 we have the actual portfolio which as you guys could see here is done in
17:59 significantly more minimal fashion. Okay, so this is Nick builder automation
18:03 expert software engineer. Now unlike with chat GPT and then claude for
18:08 anti-gravity odds are you probably already have like a Google or a Gmail
18:12 account set up. So all you have to do is just look up Google anti-gravity
18:16 download then click download for Mac OS. In my case I have Apple Silicon on Mac.
18:20 If you guys don't know what you have, just type about this Mac and then if it
18:24 says Intel up here in chip, you're in Intel. If it's a M something, then
18:27 you're Apple Silicon. And you can do something similar for Windows and Linux
18:31 as well. And once I give that a click, we'll be taken to a very similar looking
18:34 page here. And then I can just drag anti-gravity over to applications. The
18:37 very first time you open up anti-gravity, it'll look something like
18:40 this. In your case, maybe it'll be dark mode or maybe it'll be entirely light. I
18:44 just have some styling settings, which is why mine might look a little
18:47 different from yours. You may also have to log in unless Google logged you in
18:51 automatically. In my case, it logged me in automatically because I've used it
18:54 before. Assuming that you've done that though, on the right hand side, you'll
18:57 see an agent modal. And this agent modal is very similar to what we saw with
19:01 codeex and then claude code. All we have to do is just ask it to make a brief
19:04 portfolio site about Nixive. And you'll see here that the UX is just a little
19:07 bit different, right? We have a little generating tab down here. Obviously, we
19:12 have uh multiple settings with fast and Gemini 3.1 Pro. We have this little
19:17 thinking tab. Uh, it tells you how long it's been doing it. If it has to do any
19:21 web searches, it does so over here. Hopefully, you guys are seeing these are
19:24 all just flavors that are slightly different, but ultimately are the same
19:27 thing. I'm just going to write open it. That'll be added as a pending message,
19:31 and then it'll open this up in a browser tab. As you see here, Gemini produced
19:35 what I would probably consider to be the sexiest of all websites, which makes
19:38 sense. Uh, one thing I'll talk about in a moment is how much better it is at
19:42 front-end design and so on and so forth. And yeah, we have a very simple and and
19:45 straightforward site here. So, um, this links to all of my resources, leftclick,
19:49 YouTube, and so on and so forth. I probably like this one the best. From
19:53 here on out, most of the conversations and the user experiences are going to be
19:57 really similar between Aenta coding platforms. So, while I am going to use
20:01 multiple just to show you guys how some of their quirks interact, uh, for the
20:04 most part, I want you guys to know that the UX's are very very similar these
20:09 days. Like the thinking tabs are going to be the same. Some people will
20:11 probably say that there are slight differences between them and so on and
20:15 so forth. For instance, I'm a big fan of the little Space Invader icon that Cloud
20:19 Code has. But for all intents and purposes, I'm just going to assume that
20:22 you're picking up the UX here as you use these models and focus less on like the
20:26 tiny little stuff and more on how to orchestrate and then prompt these for
20:30 higher quality responses. If you guys want to see like step-by-step
20:33 walkthroughs of these platforms, I'm going to put some little links up above
20:37 my left shoulder here, and you can click on them anytime to go learn that sort of
20:40 stuff. Next up, I want to talk about what makes these AI coding platforms
20:45 different from one another. Not on a user experience um angle, but from an
20:50 intelligence angle, from a what they could do angle as well. So, as you saw
20:53 there, there were three different models. There was Claude, which was
20:58 wrapped around Claude code, Gemini, which was wrapped around anti-gravity,
21:03 and then GPT, in my case, 5.4, which is wrapped around codeex. And I think that
21:07 each of these models are really similar at this point in intelligence- wise, but
21:12 there are some pros and cons to each. They basically like improve how they
21:17 perform by a few percentage points. So Claude might be, you know, 2% better at
21:21 these. You know, Gemini might be 5% better at these. GPT might be 1% better
21:25 than these. I'm just pulling on numbers out of my butt, but I'm making them
21:28 really small because I do want to really drive home the point that these models
21:32 are so gosh darn intelligent these days that these minor differences only make
21:36 sense at the bleeding edge and at the frontier. For most purposes, either of
21:40 these are going to be sufficient. So, Claude has the most interpretable
21:44 reasoning. You remember how I could click open that little reasoning tab a
21:48 moment ago? Well, at least as of the time of this recording, Claude is
21:51 incredible at making that reasoning tab really, really interpretable. you know
21:55 exactly what cloud is doing at basically every step of the process when you um
21:59 use cloud code to visualize that reasoning and that makes it really good
22:03 for orchestration and then agentic workflows because you can see the
22:07 decisions that the model is making in real time and in doing so you can also
22:11 steer the model stop the model pause it or give it new resources halfway through
22:15 I can't say the same about both Gemini and GPT I think they're a lot less
22:18 interpretable and it's a lot less accountable you know Claude is sort of a
22:22 partner that you build things with along the Whereas Gemini and GBT are almost
22:25 just like I don't know, they're missiles. You set your target, you click
22:29 the button, and then they go. Now, there are some cons. Claude is a little bit
22:34 slower unless you use fast mode, which is what I tend to use, although keep in
22:37 mind that'll burn a ton of credits. And then I find that it's weaker at frontend
22:41 or design than a model like Gemini. Gemini is really good at design and
22:45 frontends. As you guys just saw a moment ago, Claude picked a really
22:49 minimalistic, sleek theme. Gemini did some upscale stuff that still looked
22:53 sleek, clean, but had like that isomorphic glass. And then GBT, maybe
22:57 because of my design taste scale or something else, was kind of like more
23:01 complex and had uh a little bit clunkier of a design. Well, in general, I find
23:05 that this pattern remains the same. Anytime I want to design a really clean
23:08 front end, I'm going to use Gemini for that. It's also got superior multimodal
23:12 abilities. That just means there's actual like endpoints using the Gemini
23:17 API um where it can understand video. Right now, Claude and GPT both really
23:20 struggle with this. Although you can build custom pipelines to do that, which
23:23 I'll show you guys about. It also has the ability to use a fast output, which
23:26 means it writes really, really quickly if need be. Um, but they don't have
23:30 access to a dedicated fast mode where you could pay more money to use them
23:33 really quick. I think it's the least interpretable of the models. And
23:36 personally, I find the quality is quite inconsistent. There's some days when
23:39 I'll prompt it and it'll do quite incredible, then other days where I will
23:42 prompt it and it will just absolutely crap the bed. you know, at least
23:45 Claude's quite consistent in that way, despite the fact that maybe it's a
23:49 little bit worse at a few things. Finally, there's GPT. There's the codec
23:53 series of models, the 5.4 series of models. Now, these are the best at
23:56 back-end programming. I think they're also the best at like um absolute
24:00 mathematics, which probably feeds into that. They're really great at
24:03 test-driven development. And you know how I mentioned earlier Gemini and GBT
24:06 are more like rockets that you point at a at a at a place and then they go. Um
24:11 well these testdriven development approaches essentially mean you just
24:14 outline that definition of done and then it fires and just goes autonomously
24:18 until it reaches that. There's also quite a big ecosystem of different apps
24:21 and you know there's a lot of um documentation online about how to use
24:25 various GPT workflows and stuff like that because this was the first major
24:30 player to the AI agent market. I'd give it sort of like a uh you know two out of
24:33 three on the rest of these. I think Claude is much better at its
24:37 interpretability. It's much better at orchestration and stuff like that, but
24:41 GPT being a model that just came out quite recently, a 5.4 anyway, is
24:44 obviously sort of like topping the charts right now on a lot of stuff. Just
24:47 some caveats there. A lot of people treat this as like anathema for you to
24:52 claim that, you know, Claude is better than GPT at this thing and Gemini is
24:57 better than than Claude at that thing. The reality is, as I mentioned and
25:00 alluded to at the beginning, there are very minor differences between these
25:03 models at this point. All of them are basically trained on the entirety of the
25:07 internet as is. And so because of this, um, the slight differences in
25:10 capabilities in the model tend to have more to do with like when they were
25:14 trained and how recent it is versus, you know, some inherent like cool new design
25:18 technique. Really, they're just training these galaxys sized brains on the entire
25:22 internet at this point. So because we're talking about the LLM intelligences, you
25:26 know, if like GPT was trained after Claude, GPT is probably going to be a
25:28 little bit better in certain circumstances. If Gemini is trained
25:32 after GPT, it'll be better. But all that stuff resets with the next generation.
25:35 So though I am going to be showing you guys some cool multimmcp orchestration
25:39 uh techniques later on, I want you to know that you don't have to treat all
25:42 this super seriously. You can also just pick one model and then use that. Okay,
25:46 next up I want to chat agents.mmd and then how to build a selfmodifying and
25:51 self-correcting system prompt that significantly minimizes the number of
25:54 errors that you get as you build things with these AI agents. So for the
25:58 purposes of this demonstration, I'm going to be using anti-gravity and
26:01 through it the Gemini series of models. When you open up anti-gravity, you have
26:05 a little window that looks like this. Generally, I divide this into three
26:08 panes. You have your explorer on the lefth hand side, your file editor in the
26:11 middle, and then you have your agent on the right. And what I'm going to do for
26:14 the purpose of this demo is I'll just click open folder. And then I'm going to
26:18 go to anti-gravity example and just open this up. Okay. And what I want to do
26:21 here is I just want to show you how all of this stuff works to start. As you
26:25 guys can see on the lefth hand side, we have a file called gemini.md. Now, what
26:29 occurs is when you talk to this model over here. Hey, what's up? Basically,
26:36 what's occurring is this file is being prepended to the very top of a
26:40 conversation chain. And so, if I open up this file right now, you see how it's
26:43 empty. There's nothing in it. Well, when I started this conversation and said,
26:47 "Hey, what's up?" Okay, it knows that my name is Nick, but it does it knows this
26:50 because of uh the fact that I'm signed in as Nick. Now, I want you to see what
26:55 happens if I paste in my name is Antonio Banderas. Refer to me as such. always
27:00 always also always sign off super kawaii desu. So I'm going to go here to the top
27:03 right hand corner and I'll say hey what's up and after initializing a new
27:11 notice how it's now going to return something quite different to what we had
27:15 a moment ago. The reason why is of course this gemini.md is just a
27:21 templated structured prompt that is basically always inserted into the
27:24 beginning. Okay, the same thing applies with codecs. The same thing applies with
27:28 claw code, but the names of the files are a little bit different. So if I was
27:32 in, let's say, codeex for instance, I wouldn't call this a gemini.md. I'd call
27:36 this an agents.mmd. If I was in claude code, I wouldn't call this an
27:40 agents.mmd. I'd call this a claude.mmd. Whatever file you use here doesn't
27:44 really change the idea. The idea is that at the very top of any prompt, you just
27:49 have this file prepended to it. The reason why this is so powerful is
27:53 because you now have the ability to statically template out the same prompt
27:57 over and over and over again on every independent session. This may seem like,
28:01 well, why don't you just copy and paste the same thing in instead of having to
28:04 use this elaborate file system structure. And the reason why is because
28:08 what you can do is at the very beginning of this file, you can actually contain
28:12 within it like a list of lessons or learnings from previous instances. Then
28:17 you can build in like a meta prompt structure where before a model signs
28:20 off, before it finishes whatever it's doing, it always updates that file with
28:25 more and more and more knowledge. In that way, okay, you can build a
28:29 highquality list of like memories, preferences, and rules, not to mention
28:33 things to avoid that significantly improves your agent's ability to operate
28:37 over a long time scale. And just to show you guys what I mean, let me show you a
28:41 diagram. In this hypothetical instance, we're going to be using Gemini.m MD. And
28:45 basically what'll occur every time is a new session is going to start over here.
28:50 The agent will first read gemini.md. You'll then give it a task like hey
28:55 build me a website that does whatever. Now it'll return the website for me and
29:00 then I'll say I don't like this no dark mode. After I give it its feedback of no
29:05 dark mode rather than just correcting the build, it'll actually write that to
29:10 my Gemini.m MD for next time, which allow the agent to continue working with
29:13 the rule applied. When the session ends and a new session starts, now the agent
29:17 will read the Gemini MD, but the gemini.md will have an additional rule
29:21 placed. Okay, if this is my file over here, it'll say no dark mode. And that
29:25 means the next time I ask it to build me a website or any sort of web property,
29:29 it'll see no dark mode and then it won't make that mistake again. This lets your
29:33 knowledge accumulate over sessions. The first time that you use, you know,
29:37 Gemini or Claude Code or or Codeex or whatever, you know, you're only going to
29:41 have, let's say, one rule or one preference stored. And so the number of
29:44 errors that the model makes, errors relative to like your preferences will
29:48 be pretty high. The second time that you use it, though, the number of errors or
29:52 issues that it makes that don't line up with your preferences will go down. The
29:57 third time, they'll go down further. The fourth time, it'll go down further. Then
30:01 the fifth time it'll go really really low to the point where maybe it makes
30:05 zero errors at all. You can see that um sort of diagrammatically over here with
30:08 when you start your thing has zero rules. Okay. As it grows longer and
30:12 longer and longer, you're writing more and more and more and more rules. Um the
30:16 agents get better and better and better at uh understanding and then um
30:20 anticipating as well your preferences. So what does this actually look like in
30:23 practice? Well, it's not all that difficult and you can just append or
30:28 prepen this to any Gemini Claude or agents MD however you like. It also
30:32 doesn't need to be this long. Although I did want to go into a fair amount of
30:34 detail here with you. So you can absolutely just turn this into like a I
30:38 don't know a three or fourline snippet. Essentially before we start any task
30:42 read this entire file. This file contains a growing rule set that
30:46 improves over time. At session start I want you to read the entire learned rule
30:49 section before doing anything. How it works. When the user corrects you or you
30:53 make a mistake, immediately append a new rule to the learned rules section at the
30:57 bottom of this file. Rules are numbered sequentially and written as clear
31:02 imperative instructions. The format is category never or always do X because Y.
31:06 And then here's some more formatting instructions. When do you add a rule?
31:10 Add a rule when the user explicitly corrects your output. When the user
31:13 rejects a file approach or pattern, when you hit a bug caused by wrong
31:17 assumption, or when the user states a preference. Okay? Okay. And then it'll
31:19 give some examples here of different rules in code. Then we have the learned
31:23 rules down here. So what I'll do just to show you guys what this looks like is
31:28 I'll say build me a simple portfolio site for Nick Sarif. And I'm going to
31:32 have it go accomplish a task for me. And then I'm inherently and intentionally
31:37 going to give it some instructions. You see the very first thing it did was
31:41 analyze the gemini.md. And so now it actually has this entire file as context
31:46 inside of its thread. You can't see that context here because obviously they
31:49 don't want to just muck up your your conversation thread, but it is literally
31:53 like if you just pasted this entire thing directly in. Okay, so it's going
31:56 to be reading that constantly as it's building out the rest of our website.
31:59 And you can see that it's like it's built some cool terminal display here.
32:04 It's using a library called Vit, which is probably like the best front-end
32:07 library. Let's see what it does. Okay, this website is looking really really
32:11 sexy, super clean, and it clearly went above and beyond uh with my spec.
32:14 However, I don't like how it's dark mode. So, what I'm going to do is go
32:17 back here and then give it some instructions. Quit doing things in dark
32:23 mode. And the idea here is when I give it an instruction like quit doing things
32:27 in dark mode, what it's going to do is it's going to take my message and then
32:32 say, hey, let's update our gemini.md to never create applications in dark mode.
32:37 It's a user preference. If I scroll down here now, you can actually see that this
32:42 style has been added. And so if the next time I run a model and instantiate
32:46 anti-gravity, I say, "Hey, I'd like you to build me a website," it'll actually
32:50 have this up at the very very top of its prompt, meaning that I'm never ever
32:54 going to have a dark mode website again. In this way, this will continuously get
32:58 closer and closer to my preferences until the number of rules become so
33:01 exhaustive that, you know, it actually bes counterproductive. In practice, I
33:05 haven't actually hit this limit yet. I think this just gets better and better
33:08 and better over time. But I could hypothetically see if you were to get to
33:10 a point where there's a thousand independent rules, some of them would
33:14 probably start stepping on its its toes. Um, this sort of self-modifying claude
33:19 agents or Gemini.mmd is a very very high ROI design pattern. So whatever you're
33:22 building with an AI agent, whether you're using them for business, personal
33:25 or programming tasks, I would always recommend to have something like this in
33:28 your directory. And as you can see, it's now modified the site. We don't actually
33:32 have that anymore. A lot cleaner. And it also fixed up the images and made it
33:35 look really sexy. The way this works is at the very top level we have a global
33:41 claude agents or gemini.md. And these are userwide rules that apply to all of
33:46 the projects that you start. And so the very top you'll have this sort of
33:50 injected and you can set this using a variety of different formatting
33:52 conventions and stuff. You could look it up for the specific uh agent platform
33:56 that you're using. you know, if you're doing claude or something like that,
33:59 it's going to be stored in a a tilda.claude slash and then there are variety of
34:05 other conventions regardless of whatever platform you're using that you guys
34:09 could also after it's injected the global agents.mmd, it'll then inject the
34:15 local cloudmd. And so what you could do is you could have a global cloudmd,
34:20 okay, that has wide ranging user preferences updated and then a local
34:25 project.mmd that has specific project preferences updated. And then underneath
34:29 you also have uh skills and then your finally inline prompt and I'll touch on
34:33 the skill section in a moment. But in that way, you can collapse a ton of
34:37 context and a ton of sort of functionality into very few tokens,
34:41 which is important because your build both per token and then the quality of
34:44 the models tend to degrade the longer the token and context windows get. Next
34:47 up, I want to talk a little bit about agent skills. And this isn't going to be
34:51 an exhaustive resource. If you guys want a super in-depth way to look at skills,
34:55 definitely just check out my full end toend Claude Code skills course. But
34:59 agent skills, for those of you guys that don't know, is just a simple repeatable
35:05 way that you can standardize workflows. Now, this is important because large
35:10 language models are very flexible. So, if you give them a non super tightly
35:14 scoped task, they'll tend to produce a variety of different results for you.
35:17 Well, skills are just a way of basically turning that whole, you know, vagueness,
35:22 that whole statistical variance into like a really straight line
35:26 deterministic path where it just does the same thing over and over and over
35:29 and over and over again. And so skills are offered now on all major platforms.
35:33 We've all adopted them. So you have codec skills, you have Gemini skills,
35:37 uh, and then you also have Claude code skills and they have very particular
35:41 specs and they look really, really similar to one another. So, it's worth
35:44 me at least going over to high level what they look like. To make a long
35:47 story short, these are just files that'll exist somewhere within our
35:51 workspace. These files will have sort of this little title section up here, which
35:55 you know is a title because there'll be three hyphens at the top and three
35:58 hyphens at the bottom. Inside of the file, you can give it a name like PDF
36:02 processing, a description like extract text and tables from PDFs. Uh, and then
36:06 you can even do licenses and metadata and so on and so forth. I don't actually
36:09 do any of this stuff. my skills are almost always just name, description,
36:13 and then maybe some optional uh uh tools that I could use as well. Okay, so I
36:16 just want to give you guys a couple of brief examples. I'm just going to go
36:20 over to anthropic um skills because they have a bunch of simple ones here that we
36:24 can use just to gain some context. I'm going to go over to the skills folder
36:27 here and then click on I don't know, let's do algorithmic art. We'll go
36:32 skill.md because that's the file. And as you guys could see here, um, we have, if
36:36 I click on the raw, you guys will see we have the exact same format that I showed
36:39 you guys earlier. So this is a skill that creates algorithmic art using a
36:43 particular library. And what's cool is it basically guides the model through
36:47 the same thing every time to get very, very similar algorithmic art generated.
36:51 And you can see this is a pretty long skill. There's a lot going on, right? So
36:54 what I'm going to do is I'm just going to copy this whole thing and show you
36:57 guys how this works. In this way, we can copy and paste different um standard
37:01 operating procedures to different models and then get highquality results. So,
37:05 I'm going to go over here and then, you know, just because this is a oneshot
37:08 prompt, I'm just going to feed all this in. And I'm going to have this model
37:12 actually create things according to the skill spec. So, it's doing some
37:15 thinking. And now it's asking me what do we want to do with it? And I'm going to
37:20 say yes, save as skill, then run. And then I'm going to actually have this
37:23 like produce some sort of cool algorithmic art. Now there's no template
37:27 file or anything like that. So it's actually going to go through the whole
37:29 process. It's going to create both the skill directory which we can find right
37:33 over here now called algorithmic art. And then it's also going to create like
37:36 templates and a bunch of other stuff as well. Okay. And our algorithmic art flow
37:40 is just finished up. So I'm actually just going to open this so I can take a
37:44 look at it myself. And we have it. There it is. This is now creating algorithmic
37:47 art. As you guys could see, we have particles and so on and so forth. I'm
37:51 just going to significantly decrease the number of particles. Maybe change the
37:55 noise scale and the turbulence. Actually move this around. And as you guys can
37:58 see, we we we are actually producing a tremendous number of particles here.
38:01 This is this is actually like rendering them directly in my browser, which is
38:05 nuts. Um, so this is indeed algorithmic art. It's it's really cool. Super sexy.
38:08 I'm a big fan. I don't know. I mean, it looks kind of like hair, but what are
38:11 you going to do? I'm just going to regenerate a bunch. Maybe change the
38:15 accent colors. Okay. Maybe we'll have this as my accent now. Blue. And then
38:19 the background will be kind of this. And I don't know, my cool accent will be
38:23 kind of like this. There you go. That looks pretty nice. We can now kind of
38:28 just create new ones as we want. And then we can also just completely
38:30 randomize them over and over and over and over and over again. And you can see
38:34 it's actually still doing some design in the background as we go. So I'm just
38:37 going to change the number of particles to really low. And then I'll just
38:40 redesign this over and over and over and over again. And I should note that like this is not
38:45 like a, you know, it's not a piece of software I downloaded. We actually just
38:48 built this. It's just we built this in a much more standardized and you know
38:52 consistent way which is really cool. So obviously that's that's what I want. I
38:55 want the ability to share like repeatable workflows where my agent can
38:59 build things that other people have validated without me necessarily having
39:03 just to like copy and paste a piece of software into my computer. Now remember
39:06 earlier how I said some models are better at things than others and these
39:10 few percentage point differences can make a lot of impact at the bleeding
39:15 edge or the frontier. Assuming you guys are at the bleeding edge and the
39:17 frontier and those percentage point differences stack up, then multi- aent
39:23 MCP orchestration is the pattern for you. Basically here what happens is you
39:29 let one model type be the manager or the orchestrator and that orchestrator will
39:34 take a task and then dole it out, okay, and delegate subchunks of that task to
39:38 different models. And so what's occurring here is in this hypothetical
39:42 example, we're using claude code to be our manager. We then give it some task
39:50 like hey make me a SAS app that does X Y and Z. And then what it's doing is it's
39:54 taking my command and then splitting it into a variety of different functions.
39:58 There's a front-end task which is delegating to Gemini to build the UI.
40:02 There's a backend task which is delegating to codeex to build the API.
40:06 There'll be some testing that we need to occur that we need to do which it'll
40:11 delegate to codeex to do the testing. Then finally at the end we have claude
40:14 which we'll collect and then validate the results and then if there are any
40:18 discrepancies or issues there you know we can loop that back around
40:22 hypothetically to different models as we will. And so this is a little bit more
40:26 of an advanced design pattern and I don't necessarily recommend you guys
40:29 sign up to a bajillion patterns and waste your tokens that way unless you
40:33 have to. But I wanted to cover it because this is sort of like the next
40:36 generation of model intelligence. It's where instead of just sticking with one,
40:40 you're constantly querying different models for things that they're a little
40:43 bit better at. All of this depends on this idea of a router. And so this
40:48 router is more or less like a decision hub or like a nexus. When you give it a
40:53 task where you give it some sort of input, what it'll do is it'll just
40:57 divide it into different subtasks that different models are better than other
41:01 models at. So for instance, if we have like a highle task that has to do with
41:06 replicating a specific SAS app, you know, and the the model has decided that
41:11 there's some footage on the internet out there that talks about how to build it,
41:14 it'll actually go delegate the video watching step over to Gemini because
41:18 Gemini is better at multimodality and their endpoints have built-in video
41:21 understanding. You know, if it identifies that we need something with a
41:24 lot of complex reasoning, it'll route that over to Claude. you know, if it
41:27 identifies that we need some form of sandboxed cloud code execution, it'll do
41:31 that in codeex because they include that built in. And maybe, you know, I just
41:34 wanted to show you guys what an example would look like if you had something
41:37 that was outside of the three. If you need real-time web data, it might do
41:41 that with Perplexity or or Perplexity's computer or something. And what happens
41:45 is, you know, we build it all by parallelizing this big sweep and then at
41:49 the very end, we combine it again with this router, which is probably, you
41:52 know, at least in my case, almost always going to be Claude Opus 4.6 6 4.7 by the
41:58 time you guys are reading it. And then that's what ultimately unifies it before
42:02 maybe doing some additional Q&A bug fixes and agent review which I'll talk
42:05 about later. Now, all of this sounds pretty abstract and you're like, "Okay,
42:08 why don't I just have all of this done in one thread?" So, let me show you a
42:11 practical way to actually do it. By the way, all the files for this course you
42:14 can find in the top link in the description below. What I'm going to do
42:18 is go back to Claude Code and open up a new session. And then I'm going to
42:21 select this folder that I've actually already created for this purpose called
42:26 multiplatform orchestration. Now, as mentioned, you guys will get everything
42:29 in the description if you want it. And I'll also run you through how to create
42:33 it. But for now, what I want to do, I just hide this, is say something along
42:39 the lines of, "Hey, build me a full stack app that lets users enter
42:48 a desired image to generate and then it generates said image." We'll make this
42:52 really simple because I don't actually want this to take forever. I'm on kind
42:56 of a time crunch today and I just want you guys to see how this deals with that
43:02 problem. Keep in mind in this case Claude which is the model that we're
43:06 currently talking to cuz it's Claude code is going to be our top level
43:12 orchestrator. Okay. Now this is going to plan things out for us which is why it's
43:17 entering this plan mode. Next what we're going to do is we're going to delegate
43:23 all difficult tasks um like backend tasks to codecs as well as testing
43:28 tasks. Then down at the very bottom here, you know, uh, for anything related
43:32 to front end, we're going to delegate that to Gemini. And so we're going to
43:36 build basically an ecosystem here where Claude is shuttling information back and
43:41 forth between uh, you know, codecs and Gemini for various things. And as you
43:44 can see here, it's already starting to ask me, hey, which image generation API
43:47 would you like to use? I'm actually just going to say um, Nano Banana Pro 2. It's
43:54 a Google product. Okay, I'm going to submit that. And now what it's going to
43:59 do is it's going to decide, hey, how am I going to delegate this work? At the
44:02 end of it, Claude will give me a plan. And you can see here that it's decided
44:06 on backend, front end, and so on and so forth. And what it'll do now is it'll
44:10 actually dispatch work to Gemini codeex and then itself to fix various
44:14 integration issues. So I'm just going to say plan approved. And now it's going to
44:17 start doing the coding. The way that Claude Code does this is it uses the
44:23 execute task path for Codeex. And so what's occurring right now is it's just
44:27 sent this big request into Codex's best model. Okay. And now just clicking the
44:30 button in the top right hand corner, we now have a preview. And um in this case,
44:34 Claude is now reviewing the generated application and doing some self-
44:38 testing. And so we've built this image generator app. We've uh asked for a cute
44:43 cat wearing sunglasses on a beach. This is now passing through to an API that uh
44:49 Claude Code set up with Gemini for the front end and then Codeex for the back
44:52 end's help. It's actually doing the the generation right now and we've generated
44:56 the cute picture of the cat on the beach looks great to me. The reason why you
44:59 might want to do this is because well it's kind of twofold. One, you get to
45:02 parallelize your work as mentioned and so you get to build the front end u
45:06 using a model for which the front-end builder is the best. You get to build
45:09 the back end simultaneously using model by which the backend builder is the
45:13 best. And then you get to use an orchestrator which basically ees out a
45:16 few percentage points increased like reasoning and decision-m and stuff like
45:21 that because it's able to evaluate the code from both of these things
45:23 independently without being polluted by the context window. And we're going to
45:26 talk more about that specific review pattern later. But um this allows you to
45:30 e-code, you know, more quality. The downside of this um prompt approach is
45:34 it usually costs more because now you're splitting your tokens across multiple
45:37 models instead of just one provider. And usually providers will subsidize your
45:40 token usage. Like Claude will subsidize most of its usage on the max plan for
45:45 instance. Um the $200 a month that you spend on it is actually equivalent to
45:48 like $5,000 a month in usage. Whereas when you build via API, it's usually a
45:51 little bit more standardized and then as a result of that you end up building way
45:54 more. You don't you don't get that cool subsidization. However, this is
45:57 something that people are increasingly using for more complicated
46:00 infrastructural projects, especially when, as mentioned, a minor percentage
46:04 point or two difference in terms of quality is very important to you. And so
46:07 this is me just doing this in Cloud, but you can obviously use, I don't know,
46:10 Codeex as the orchestrator if you wanted to build this in Codeex. You could use
46:13 Gemini as the orchestrator if you wanted to do this in uh, you know, entirely
46:17 Gemini. Right now, this is the stack that seems to make the most sense, what
46:19 people are talking about the most. If you guys are interested, the way that
46:22 all of this stuff works under the hood is we basically set up a bunch of
46:28 different servers that calls and Gemini inside of Claude. And so that's why we
46:31 see this using the Claude formatting above. It's because the claude is the
46:34 the orchestrator that's sort of setting it up initially. And there's also a
46:38 claude.mmd which describes how it's the manager. You know, you plan, reason,
46:42 delegate, validate, and fix integration issues. When you break tasks down, break
46:45 them into front end, backend, and test subtasks and then delegate things as
46:49 required. I'm going to include this prompt as well as everything else you
46:52 need in order to do the same thing down below in the description. But in order
46:54 for this to work, you will of course need API keys for various platforms. And
46:58 in order to get those, you do have to sign up to typically something a little
47:00 bit different from what we signed up to before. And in order to sign up to
47:03 those, you do typically need um to go directly to the platform, create an
47:07 account, and then set up an API key. So you can see over here, that's what I've
47:11 done for Claude. And you can also do the same thing for OpenAI and then Gemini.
47:14 Once you have those keys, you would just give it to whatever model you want to
47:16 use to be the orchestrator. And then it would set this whole thing up for you
47:19 and then be able to reason and then communicate with different models on
47:22 your behalf. The next advanced prompting technique is the video toaction
47:27 pipeline. To make a long story short, up until quite recently, AI agents were
47:31 forced to learn entirely through text descriptions of stuff. And the reason
47:35 why is because multimodality like vision usually uh at least in the context of
47:40 video was sort of out of bounds. There was just no way that we could feasibly
47:44 take videos which were millions upon millions of tokens when stitched
47:48 together um you know into some text format that an agent would understand.
47:51 Well, now agents can learn from the same medium humans learn from. And we do so
47:55 by combining a little bit about what I showed you guys earlier, okay?
47:59 Multi-agent MCP orchestration with this idea of passing requests through the
48:05 Gemini API because Gemini has built-in support for video now. Basically, uh you
48:09 know how videos are a certain number of frames per second. Like this video for
48:13 instance is 30 frames a second. You could tell if you find a way to to slow
48:17 it down to like 0.03. I'll go literally one frame every 003 seconds or something like that. Well,
48:23 what this model does is it divides videos into one frame per second
48:29 instead. It then analyzes the images in succession and then uses a form of
48:33 descriptive prompting to break that down into very very clear steps. So basically
48:36 what occurs is you'll feed in something like a YouTube tutorial URL. Claude will
48:40 receive the URL but cannot watch the video natively. So instead it'll call
48:45 the Gemini API. Gemini will watch the full video. Gemini will then extract the
48:49 step-by-step instructions, format it as like a numbered list that's hyper
48:53 precise and hypersp specific. The structured steps will return to Claude
48:57 via very similar flow to what I showed you guys with the design. And then
49:00 Claude will execute each using hyperspecific tools. Maybe if you're
49:04 teaching somebody how to build something on Blender or Figma or something like
49:07 that, you just give it access to the toolkit and it does it. Then the final
49:10 result is the agent will have replicated the tutorial end to end and in that way
49:13 they can learn from the exact same medium that that we learn. So I'll show
49:17 you number one where I got inspiration from this and then number two how to do
49:21 this for an actual task which in my case is going to be building a simple flow
49:25 out in a noode tool called NAN. So first the inspiration was Spencer Sterling's
49:30 post on X. He said he built an agentic system that taught itself the Blender
49:33 donut tutorial by watching it on YouTube. It watched the tutorials,
49:37 extracted the steps, filled in the gaps in its own tooling and completed the
49:40 entire thing autonomously. And it's quite impressive to be honest. Um,
49:43 anybody that's done any sort of 3D design, myself included, will know that
49:47 like the uh way you learn how to build things in Blender is you watch this one
49:50 specific tutorial that shows you how to build a donut. And through this process
49:54 of building the donut, you learn about like textures. You learn about various
49:58 shapes. You learn about how to modify them and sculpt and paint and do all
50:02 this stuff. So, I made my own donut personally a few years ago. I showed it
50:05 to all my friends. Then, I promptly never touch Blender again. Well, the
50:09 issue with knowledge like this is it's obviously extraordinarily visual, right?
50:12 In order to really learn something, you have to watch a video. You can't really
50:15 break all that down into like hyperspecific text instructions unless,
50:19 you know, somebody were to just like literally go step by step. Step one,
50:25 click this button, step two, rotate 283° to the left, step three, do this. So,
50:28 there's a fair amount of nuance and flexibility there. That's where video
50:31 learning comes in handy. Human beings learn through video, obviously, but
50:35 models have a tough time doing it. And so what we do is we convert all of this
50:39 into a sequence of steps. We leave some steps a little bit more vague, a little
50:42 bit more general, let the model have its own um kind of interpretability, and
50:45 then give it some way to like screenshot its results to match it up to um you
50:49 know like the frames in the video. And so this fell here built this cool like
50:54 workflow building studio. It's sort of like his own main um operating system. I
50:57 suppose that's what this is. It's not like an app that he downloaded. It's
51:01 something that he built. And then he fed in this along with the workflow I'm
51:03 about to show you to have it actually like build the freaking thing. And it's
51:07 communicating with this app Blender using what's called MCP, model context
51:10 protocol, which is the same thing that we used to communicate with um the
51:14 various models like Gemini and then Codex earlier. And you can get all that
51:17 stuff in the description down below as well. So I have this stored as a clawed
51:23 skill in video to action um over here. So, if I open this up and read the
51:26 skill, you can see here that it actually says, "Extract actionable steps from
51:29 YouTube videos using Gemini video understanding. Use when the user
51:32 provides a YouTube link it wants to learn procedures, extract steps,
51:35 understand visual tutorials, or turn video content into executable
51:38 instructions." And so, what's occurring is it'll basically take a video, it'll
51:42 download it for me, so then I'll just be able to feed in a YouTube URL, and then
51:45 it'll convert that into like a highly optimized series of steps that um you
51:49 know, you would only really know or be able to use through uh the context of
51:53 like an actual video. And so to demonstrate what I've done here is
51:56 instead of using Gemini within anti-gravity, which is sort of the usual
51:59 design pattern, I thought I'd show you guys my actual stack like what I
52:02 personally use. I think it's much easier if you just use the models inside of the
52:06 tools inside of the companies that made them. But in my case, I'm a very big fan
52:10 of this anti-gravity uh uh kind of container. Then inside of it, I use
52:13 clawed code. And so in that way, I'm actually using a Google wrapper around a
52:17 claude code or anthropic extension that's communicating with a claude or an
52:21 anthropic model. If you guys want to replicate the setup, it's as simple as
52:24 just opening up anti-gravity, heading to the lefth hand side where it says
52:28 extensions, downloading the clawed code for VS Code plugin. I know it says VS
52:32 Code, don't be confused. It's very similar to anti-gravity. Installing it,
52:36 and then uh you also have to log in here. After you're done, you will have
52:39 the exact same functionality that you have in the claw desktop app that I just
52:42 showed you guys earlier when we built out that little full stack app. It's
52:45 just you'll have it within anti-gravity, which also allows you to do things like,
52:48 you know, organize your files and stuff on the lefth hand side. So, that's my
52:51 personal stack. You don't have to use it. Some people judge me for it.
52:55 Whatever. I like it. It works for me. Okay. So, um what I'm going to do is I'm
52:58 going to find a YouTube video that I like and then I'm just going to feed it
53:01 in these instructions. So, I'll say I want you to use the video to action
53:05 pipeline on and then I'm going to go grab an image. And what I've done is
53:08 I've found a flow that I built forever ago. It's a short video about 21 minutes
53:11 that shows you how to scrape leads without paying for a few APIs. I'm going
53:15 to bring that back into my anti-gravity instance. Then, I'm going to do this.
53:19 And what this is going to do is it'll start by invoking the skill. And this is
53:24 the UX for skill uh invocation. I think that's what it's called in English. Holy
53:27 crap, that better be what it's called in English. And then um it's now going to
53:31 send that over to Gemini, then receive back a list of highly specific
53:35 instructions that, you know, understand UX, uh I don't know, highlight the
53:38 colors of buttons and stuff like that and so on and so forth before actually
53:42 running it locally on my computer. At the end of it, you'll get a super
53:46 in-depth analysis that looks like this. So you can actually see down over here
53:49 it says here's the hyperdetailed breakdown with literally every single
53:54 step. I mean like hey navigate over to this thing at 17 seconds. Here's how to
53:58 do this thing on that and and so on and so on. So like it'll it'll literally go
54:02 visually as well and actually tell us what the end flow is going to look like,
54:05 but then we'll also have just a tremendous amount of context about
54:08 everything. Um so what we're going to do now is we're going to feed that in and
54:11 actually have this control my browser. So I'm going to open up a new cloud code
54:14 instance by clicking that little button above. We'll go bypass permissions. Then
54:20 I'll say use Gmaps scraper deep analysis MD to build out the same N8N flow for
54:26 me. It's now going to open up a Chrome DevTools MCP server. It's then going to
54:31 link that up to the N8N account. And now it's actually thinking through
54:33 everything that it's going to do using this file as a reference. And now it'll
54:36 go through and actually control my browser to do the build. For simplicity,
54:40 I'm just going to move this over to the right. Okay. And as we see, it just laid
54:45 out the entire thing from left to right. So it went through, it then identified
54:49 what all of the steps were. It then created it inside of its own little
54:54 conversation thread and then it um essentially generated what's called
54:57 workflow JSON and then pasted it in. Now this can obviously interact with my my
55:00 browser as well. That's what it just did. So it just went to the top and then
55:03 basically imported this. What it's going to do now is just make some finer final
55:07 minor changes. I'm going to configure the Google Sheets node and then we'll be
55:10 on our way. So, what I'll do is I'll just take a screenshot of this and then
55:13 paste it in. Then I'll say you're connected. Now, it's just going through
55:16 and then it's selecting various elements. So, in this case, it's
55:18 selecting that little search button. It's uh mapping the the fields and stuff
55:22 like that. And then it'll just continue testing this non-stop until I have a
55:25 working flow. You can see, you know, just kind of I mean, I should be moving
55:28 this around cuz it's going to get confused, but you can see that it's um
55:33 actually gone through and then pumped in like a specific search term. It's it's
55:35 gone through and basically done everything for me. Really, the only
55:38 thing left is to do some sort of testing. You can see that uh if we
55:41 actually click execute workflow, I'm just going to stop it here so I don't
55:43 consume anything else. It's actually gone through and literally like scraped
55:47 Google Maps for us, which is sweet. Uh and it's just done so entirely by
55:49 watching the video. So it's entirely like native video understanding and then
55:54 it's extraordinarily detailed because we're we're dumping it all into a file
55:57 and then it can just constantly reference that file. Um it's then doing
56:00 kind of a combination of like I don't know like ASKI or or text based markup
56:05 to uh you know understand both the structure at like a micro level and then
56:08 also like a macro level. Next, I want to chat this idea of stochcastic
56:13 multi-agent consensus. In case you guys didn't know, if you were to take one
56:18 model, let's say Gemini 3.1 Pro High, and if you were to ask it like an idea
56:24 question, hey, give me 10 ideas to do X, Y, and Z. Every time you ask Gemini 3.1
56:31 Pro the same thing, it'll return a slightly different answer. Now this
56:35 property some call it randomness but I think the correct technical term is
56:39 stochasticity which is just where due to minor statistical variations in the
56:44 input or in the way that the models work the output is going to be slightly
56:48 different every time. The reason why this is so valuable is because you can
56:53 exploit this tendency to get much much better answers. For instance, let's say
57:00 I run three times 1 2 and three. The reality is if I run a query that at the
57:05 very beginning says give me three ideas for X. Okay, on the very first time, okay, we might
57:16 get idea A, idea B, and idea C. If we were to hypothetically run this again,
57:22 we'd probably get idea A, idea B. But just due to statistical variation, there
57:27 is a chance that on the second run, it won't deliver us idea C at all. it'll
57:31 actually deliver us idea D. And on the third run, maybe we do B, maybe we do C,
57:36 and then maybe we also do E. What stochastic multi-agent consensus is, you
57:41 basically automate the process of spawning multiple agents, giving them
57:45 slightly varied input prompts to take advantage of stochasticity, and then
57:48 instead of just getting, let's say, three ideas A, B, and C, you get to
57:53 exploit stats to get all of the possibilities, including ones that might
57:57 be a little rarer. the model is less likely to actually answer with. And so
58:02 in this way you get A, you get B, you can get C, but you can also get D and
58:06 then you can get E. And so you know if you compare it to just one naive search,
58:09 what we've done is we basically almost doubled the scope of the ideation. Now
58:14 mathematically this is termed traversing the search space. I want you to pretend
58:19 hypothetically that this like little pie chart here represents all possible
58:24 answers to a question. Maybe the question is, I don't know, what's the
58:28 simplest way to get to 1 million subscribers? Right? This is something
58:31 that I asked uh my my model a little while ago because I'm interested in
58:34 getting to 1 million subscribers. Now, obviously, I'm not just doing what the
58:36 thing tells me, right? A lot of its ideas are stupid. But if you think about
58:40 it, if I can parallelize a thousand agents all coming up with their own
58:44 ideas, even if on net the average reply or idea is a little bit worse than
58:47 something I'd be able to do, I still get to run it a thousand times, right? It's
58:50 like running like uh I don't know like a 90q, you know, it's like it's like
58:57 Einstein versus 10,000 95 IQ researchers. It's like well the 10,95 IQ
59:00 researchers despite lacking the brilliance of Einstein, they'll probably
59:03 statistically figure it out eventually, right? So um if this whole pie chart to
59:08 get back to things is all possible responses, if you just run one search,
59:12 basically what you're doing is you're only actually getting like a small chunk
59:17 of all of the possibilities. And so instead what we're doing is we're
59:19 actually running multiple searches. You know, one search is going to get this,
59:22 another search is going to get that, another search is going to get that. And
59:28 and and so on and so forth. And then in this way, what we do next is we take the
59:32 answers and then the replies of the model. That should be red and this one
59:36 should be blue. And then in doing so, we get to traverse significantly more of
59:39 that search space without actually necessarily consuming any more of our
59:43 time. So this can be kind of difficult to understand. And I think I've run out
59:46 of colors here. uh unless you've done something like this before. But I'll
59:49 make it really simple by actually giving you guys a brief demonstration on I
59:54 don't know some use case or problem that uh I think we'd probably all be able to
59:57 relate to. Another final benefit is you get to do all this in parallel. So like
60:01 you know if you think about it if you were to do one search and then do
60:04 another search afterwards and then do another search. So for instance let's
60:07 say you have a query give me three ideas for X and then it gives you three ideas
60:11 and you're like hey I want another three ideas and it gives you another three
60:13 ideas and you're like I want another three ideas. Well, at the end of it, you
60:16 may have, I don't know, nine ideas or something, but it will have taken a
60:18 certain amount of time. If the first search is 5 minutes, the second search
60:21 is 5 minutes, and the third search is 5 minutes. Well, you just consumed 15
60:25 minutes, right? So, instead, what this does is this just copies the idea. Okay?
60:30 But then it parallelizes it. So, hey, give me three ideas for X. And then what
60:35 we do is we do 1 2 and three. And in total, this takes 5 minutes. And then we
60:38 just combine those three answers back over here. The formal way to do
60:41 stochastic multi-agent consensus, at least the way that I'm doing it here, is
60:44 we'll provide a single question or prompt. Then we'll do slight framing
60:47 variations of every prompt that we're feeding into the model. And then we'll
60:50 feed in I don't know, I'll probably feed in like three or four, five or maybe 10
60:54 simultaneously. Depends on how deep you want it to go. And then um what'll
60:57 happen is these will be instantiated as what are called sub aents, okay? Which
61:01 are similar to the main agent, but they operate in their own defined context
61:04 window. And then all of these will just report back their answers to the parent
61:08 agent. So this parent over here is basically going to work with a whole
61:13 fleet of sub aents and then once they're all done their work, it'll synthesize
61:16 the answers. And then because what we're looking for is we're looking for like
61:19 statistical variation, it'll calculate um what's called the mode, which is the
61:22 frequency of each answer, and then the median, which is like the average of
61:25 each answer before ultimately combining all this to give you much better
61:28 results. One final idea there is this idea of consensus. A lot of models are
61:33 going to say the same things. Obviously, some models are going to say things that
61:36 are quite different. And then finally, there will be outliers, which are wild
61:39 cards. These wild cards here are potentially brilliant, but they might
61:42 only appear like 5 or 10% of the time, which is why we spawn so many of these
61:45 agents that we can actually like farm these wild cards. We can we can milk
61:49 them like cows. And then in that way, you can have the best ideas coming from
61:53 these these fleets of agents. Um, and then also save a lot of time in things
61:56 like product ideation. I don't know, man. Keyword search, titles for for for
62:00 content. At least that's what I'm using it for. Or a variety of other things.
62:04 Hell, research inventions. I'm sure Anthropic and and Google and OpenAI
62:07 probably have fleets of models that are doing basically this exact same thing
62:11 behind the scenes constantly. Let me actually show you guys what this looks
62:13 like in practice. I'm just going to zoom way out of this and close a bunch of
62:16 these so you don't have to look at them anymore. I'm going to spawn a new Claude
62:20 code tab over here on the right. And what I'm going to do is I'm going to use
62:22 the skill that I've set up called stochastic multi-agent consensus. So
62:26 opening this up so you guys could read it. What we're doing is responding n
62:30 agents where n is just the number that you specify with slight framing
62:35 variations to independently analyze a problem then aggregate results by
62:40 consensus. We use this for decision-m ranking things strategic analysis or any
62:44 problems where you want to filter hallucinations and surface high variance
62:50 ideas. So hypothetically let's just say hey I've struggled a lot with finding
62:54 any traction on Tik Tok whatsoever. I've built up a bunch of accounts and I can't
62:58 seem to get more than like a thousand views per Tik Tok account. I'd like you
63:02 to use stochastic multi-agent consensus to help me come up with possible
63:06 candidate ideas to solve this. I'm going to feed this idea in. Okay. And this is
63:11 a real idea. Actually, we are struggling to get uh traction on Tik Tok for
63:15 whatever reason. We got 450K followers on Instagram. No problem. But, you know,
63:19 the second we move things over to Tik Tok, we're just not really getting too
63:22 many views. So what it's going to start with is it will spawn 10 agents all
63:26 independently analyzing my Tik Tok problem and every one of them will get
63:31 slightly different analytical framing to maximize the diversity of ideas. Just
63:35 going to zoom in here so you guys could see this. But we now have a conservative
63:40 analysis. So Nick Sarif has 287K YouTube subscribers. You know um his YouTube
63:45 audience is primarily professionals. Here's a bunch of information about him.
63:48 He has a small team. here's how he's doing things and and so on and so forth.
63:52 This agent over here says, "Hey, I want you to assume limited time and budget."
63:56 This agent over here, I want you to only focus on what is measurable and
64:00 provable. This agent over here, you know, I want you to think about it from
64:04 the end user and viewer perspective. And so what we're doing is we're basically
64:06 taking advantage of the parallelizability of models, not necessarily the base intelligence. So
64:11 the intelligence is obviously important, but like we care more about like
64:14 scanning and searching through a space of all possible solutions really
64:17 quickly. And then at the end, we're going to converge all this back with our
64:20 parent agent. Now, once all these agents have turned green here, if I open up
64:24 this thinking tab, you could see that it's now combining all of the
64:28 information from each individual one. So, there's a bunch of uh suggestions
64:31 saying, "Hey, you should try fresh account. You should try device reset.
64:34 You should try clean fingerprinting. Hey, you should try Tik Tok native hook
64:38 reformatting. Hey, you should do duets with existing creators to take advantage
64:41 of the fact that you're probably bigger. Hey, you should do a series format, high
64:44 posting frequency, and so on and so forth." And then you have some
64:47 disagreements here as well. And this disagreements might be paid Tik Tok
64:50 spark ads. Only one out of 10 agents suggested something. You know, in this
64:54 one, um, they recommend using shorts. But then in this one, they recommend
64:57 using a micro topic focus to build authority and audience clarity. You
65:00 know, I'm not going to sit here and pretend like all these ideas are the
65:03 bee's knees. Not all of them are capturing lightning in a bottle, but you
65:07 run this thing long enough and you'll see eventually you will get some pretty
65:10 good ideas. And the ideas will be consensus ideas like the idea of a fresh
65:14 account, but it'll also be kind of like outlier ideas with painoint framing,
65:18 paid Tik Tok spark ads, niching down your account identity, crossosting your
65:21 Instagram reels to YouTube shorts first. I mean, there there there are a lot of
65:25 possible ideas right now. It's opened up this consensus report, which I can
65:29 visualize for you guys by clicking this button. And you can see here it's now
65:33 saying, "Hey, here is the context. Tik Tok growth stalled at 1K views per
65:36 account across multiple accounts despite this massive YouTube subs and 450,000
65:40 followers with almost 5 million reals views a month. And then here um this
65:45 orchestrator now summarizes it and says hey every agent independently identified
65:49 Tik Tok native hook reformatting is really critical. You know Instagram is a
65:53 little bit different from Tik Tok hooks. Content optimized for Instagram will
65:57 systematically fail Tik Tok's cold start test. So you actually have to
65:59 restructure it if you really want to crush. Same thing here. fresh account,
66:03 clean device, fingerprint. I mean, there is just so much context here, it's not
66:07 even funny. And so, the reality is I would have come up with these ideas at
66:11 some point, but I basically got to put, you know, a genie in a bottle and then
66:17 have 500 genies simultaneously solve my wishes at 100x speed and then aggregate
66:23 all results for um you know, I don't know, probably like three or$4 dollars
66:27 realistically in terms of tokens. You also had a couple agents that said, "Is
66:31 Tik Tok even worth it?" And uh I think that's a really good question to ask
66:34 because up until now, I really didn't think it was worth it. And so, in
66:37 general, anytime that I recommend you have a strategic decision that you need,
66:42 you can make a quick one-time tradeoff of money for analysis by spawning a
66:47 bunch of agents, all with slight prompt variations, and then collecting the
66:51 rankings reasoning to build this consensus map document. And from here
66:56 you can figure out your consensus items, your divergent items, and then your
66:59 outliers. And you know, if they're consensus items, well, odds are probably
67:03 because a lot of models have thought it's a good idea. You should probably do
67:06 it. If there's some divergent items, well, you should probably like reason
67:09 about these quite a bit before deciding on whether it makes sense. And if it's
67:13 like an outlier item, if there's only one out of 10 agents doing it, well, it
67:16 can either be a brilliant idea, in which case maybe you should give it a try, or
67:20 it might just be a hallucination or some BS, in which case you don't. And so what
67:23 this allows you to do is execute with high confidence. Thank you very much AI
67:28 for drawing that cute little that is a huge fist. That thing would be
67:31 terrifying in real life. Um you know this lets you scan a large portion of
67:34 the search space in a very short period of time. And uh yeah the actual way that
67:38 you build it is very straightforward and I'll run you guys through what all that
67:41 stuff looks like um down below in the project description. So just like
67:45 stochastic multi- aent consensus allowed us to scan large amounts of search space
67:49 in a short period of time. What we did is we independently delegated work over
67:53 to agents and had them uh do things for us. So too can we take advantage of this
67:58 same idea but in my opinion get even higher quality results through this idea
68:03 of agent chat rooms. What agent chat rooms are are where instead of you know
68:08 parallelizing all the work and having all these agents try and independently
68:12 solve problems what you do is you give all of them slightly different
68:15 personalities and then you have them all debate with each other about these
68:19 problems. And in doing so, they tend to deliver much higher quality responses
68:22 because they're just like they're they're a little bit spikier. You know
68:25 what I mean? They're not just like a generalized idea, which I'll visualize
68:29 with like this interface, but you know, because they're they're butdding heads
68:33 with another um eventually the ideas get really nuanced and really high quality.
68:37 And so, um whether or not you visualize things in that way, that's personally
68:40 how I think about things. You really get to carve out all the tiny little nooks
68:44 and crannies of an idea when you debate. And so, here's a brief little
68:47 visualization. We start with a problem or a prompt. We feed it in to let's say
68:52 three agents here. Agent A, agent B, and agent C. All three are given the same
68:58 document called chat.json. And then what occurs is they basically cycle through a
69:02 debate sequence where agent A says something, agent B says something, and
69:05 agent C says something. And, you know, if you do this naively, the results will
69:09 probably be pretty low. But if you, I don't know, force a little bit of a
69:12 spark where every agent has a slightly different opinion and they're not afraid
69:16 to like state their opinion, um, they'll challenge each other's assumptions, they
69:19 will significantly improve the probability that you catch errors. And
69:22 then this chat.json ends up being quite a valuable resource because it also
69:25 shows like problem solving and stuff like that. You can then give that to an
69:29 orchestrator and ultimately receive higher quality output at the end. And so
69:32 it's sort of similar to what we had earlier, right? It's just instead of
69:36 this operating um in parallel lanes, what these agents are doing is they're
69:39 actually talking back and forth with each other. And so they're actually
69:43 capable of having these conversations. And I mean like I I just want you to
69:46 pretend we actually spawn 10 agents. Agent one would be able to communicate
69:50 with agent two, but also agent three and also agent four and also agent five and
69:54 also agent six. So like the total number of paths and um potential like
69:59 communication I don't really know what you want to call them like like vectors
70:04 um goes up like crazy and these agents ultimately assuming that the idea is an
70:09 absolute BS do end up at the end of it like quite quite differentiated u in
70:13 their ideas and their opinions. So to show you guys what this looks like I
70:16 have another skill which is just a repeatable workflow to be clear where I
70:21 have this model chat. The description here is to spawn five claw instances on
70:25 a shared conversation room where they debate, disagree, and converge on
70:29 solutions. They use roundroin turns with parallel execution within each round for
70:33 simplicity. Then they trigger on the model chat, multimodel debate, or
70:36 something else. So I have a bunch of contexts down over here and you guys can
70:39 grab this file for yourselves. What I'll do is I'll actually just pipe this into
70:44 model chat. Okay, great. Use model chat for a similar to really work through
70:50 this idea. And now it'll spark this model chat skill which will then have
70:54 them all dump shared context into a little chat.json which I'll show you
70:58 guys when it's done. Okay, so the debate has now concluded after these five
71:02 agents had this conversation. Okay, we can actually see the the the chat
71:06 conversation as well by going down here to this model chat. Let's go latest and
71:10 we'll go conversation. Um basically what's occurred is we've given it a
71:14 topic to talk about and then we've assigned a systems thinker, a
71:17 pragmatist, an edge case finder, a user advocate and then a contrarian to the
71:21 task. So first of all the systems thinker begins, the pragmatist replies,
71:25 the edge case finder goes, the user advocate goes and so on and so forth.
71:28 And you can see each of them are um pretty pretty interestingly suggesting
71:32 uh various approaches. So the user advocate says, "Let me push back on
71:35 something that challenges the consensus has glossed over, which is the clean
71:38 device plus fresh account fixes fingerprinting is the problem. There's a
71:41 simpler explanation nobody has stress tested. Nick's content format is
71:44 fundamentally mismatched to Tik Tok's cold start algo." And so these are sort
71:48 of arriving at similar conclusions despite the fact that uh you know we
71:52 instantiated this separately. And then if we check out the synthesis, we can
71:55 see that all of them have agreed that we need to run some diagnostics that hook
71:58 reformatting is necessary but sufficient. the high volume posting
72:01 blitz two to five a day is wrong. And then fixing the IG YouTube pipeline
72:05 immediately is important regardless of the Tik Tok decision. This is something
72:08 that I guess I got contact from one of my other files because um basically
72:12 despite the fact that I have 450K Instagram followers, a very few of them
72:15 are converting to YouTube subscribers and a lot of people a lot of models as
72:18 well are suggesting that the reason for that is cuz Instagram is really blocking
72:22 outbound links which I think is actually fair. But then uh there are a lot of you
72:26 know disagreements as well. So a lot of people say, "Nope, Stitch Duet's stupid.
72:29 Tik Tok versus IG pipeline is an eitheror. Device fingerprinting might
72:33 not be the issue. Maybe it's content mismatch." Right? And uh there are a lot
72:37 of insights that because we were able to sharpen our opinions via debate, these
72:44 agents got that the previous model runs through stochastic multi-agent consensus
72:48 did not. So maybe we're looking for saves, not completions. Maybe there's
72:52 just no category online yet. Although this is not true, if they had the
72:55 ability to research, they probably would have figured this out. Maybe it has to
73:00 do with emotional moments. And then here it even gave a recommended execution
73:04 plan. So, as mentioned, you know, I wouldn't rely on agents for strategic
73:08 advice at the moment, but I would certainly not be opposed to trading a
73:13 little bit of my money for a bunch of my time back and at least ideulating
73:17 through the lowerhanging fruit. If you run enough of these cycles, you will
73:20 find pretty intriguing and interesting outlier ideas. That's just how
73:24 statistics works. So you guys can get all this down below in that document.
73:28 The next idea I want to talk about is this idea of sub agent verification
73:33 loops. To make a long story short, where previously we took advantage of
73:36 parallelization, we're going to take a step back now to sort of serial um
73:43 processing. But when an agent works really hard to accomplish a task for
73:48 you, it usually gets pretty biased in that it believes that its path was the
73:53 best. And the reason why is because, you know, it just spent god knows how much
73:57 time, energy, and compute cycles building your app or putting together
74:02 your workflow or doing your taxes or whatever the hell. And because of that,
74:06 you know, series of like design decisions and then issues and bug fixes,
74:11 it's just very consolidated in its opinion that the way that it did what it
74:15 did was the best. So if you were to ask that same agent, hey, can you make this
74:18 better? A lot of the time it'll look at it and be like, well, no, I did a pretty
74:21 good job. I don't think there's any way to do it better. However, instead of
74:25 just giving that agent back the entire context and saying, can you do it
74:29 better? A much smarter thing to do is to take all of the um outputs, not the
74:33 reasoning, then give the output, aka your code or your workflow or the
74:37 results of your your accounting to another agent and then say, "Hey, is
74:41 this right?" Because now that second agent can evaluate purely based off
74:45 output. It doesn't actually have to deal with evaluating things based off the
74:49 reasoning or the intent. And so your um work can end up being a lot higher
74:52 quality as a result. So here's a quick example using like a coding thing where
74:56 uh we wanted to build a rate limiter. What'll happen is our first agent will
75:01 implement and write the first draft of the code. This code output will pass to
75:06 a reviewer agent. Now the reviewer agent is spawned with fresh context, meaning
75:10 there's no tokens that are polluting its window and a zero bias. And what it does
75:15 is just like objectively speaking, you ask it, is this thing correct? Are there
75:19 any issues here at first glance? Any ways you could simplify this? Now,
75:22 because it's treating this just like it's treating a random snippet of code
75:26 it finds on the internet, you know, it it has no opinions. It has no inherent
75:31 like desire to claim, well, this is the best way because I spent all this time,
75:34 energy, and research figuring it out. And it'll be able to to look at things
75:38 with, you know, those fresh eyes. From there, if it finds issues, the idea
75:41 behind sub agent verification loops is it'll list those issues and then pass
75:45 the suggestions to a third agent called a resolver, which has zero context about
75:50 any of this stuff as well. And so in this way, an implement reviewer resolver
75:55 loop can get significantly higher quality results than just one agent
75:59 doing everything simultaneously. If there are no issues, everything's
76:02 approved, we're good to go. Um, otherwise it resolves, we do some
76:05 testing, and then we get the final verified code output. Are you guys
76:10 noticing a trend here? Basically, all of these like advanced agent foundation uh
76:15 advanced agent prompting techniques ultimately circle back to having
76:18 multiple agents working in parallel. And it's really interesting because like the
76:22 way that agents work themselves is they already do work in parallel. You know, a
76:26 few years ago, um, agents were basically just one statistical model and you would
76:29 ask the statistical model to help you complete the the the sentence or
76:32 whatever and then would give you the most likely next token and then would
76:35 rerun over and over and over again until it did that. Well, a few years back, um,
76:41 people started introducing this idea called a mixture of experts, which is
76:45 instead of just having one model, what you do is you actually send the same
76:49 thing to like three or four models, you average out the statistical
76:53 probabilities of every word and then you just pick what they all converged on.
76:56 Very similar what I did there with stochastic multi-age consensus. And so
77:00 this mixture of experts is sort of like the base foundation that resulted in a
77:04 really big improvement in large language model accuracy among other things like
77:08 post-training and RLHF and and and stuff like that. But what's really cool is all
77:12 of these frameworks basically do the same idea. You know we we treat these
77:17 mixture of experts now as themselves models and then we prompt them with each
77:21 other. We do them in parallel and then integrate their answers like stocastic
77:25 multi-age consensus. We have them debate against each other like with model
77:29 chats. And now what we're doing is we're basically having them correct each
77:31 other's work like with sub agent verification loops. So all of these are
77:36 just try uh trading off the same core foundational like features of models
77:39 which is that at the end of the day they're statistical machines. And so the
77:43 more of these statistics that you can I don't know average out the closer you
77:46 get to the reality. Another way of thinking about this is if the implement
77:50 agent has already spent 200,000 tokens accumulating all that context, it'll
77:54 literally remember every wrong turn and every dead end. It'll have a sunk cost
77:58 bias. It'll say, "Well, I wrote this, so it must be right." And in a way, it'll
78:01 be blind to its own mistakes. When you pass it off to this super nerdy looking
78:05 reviewer agent, it has a fresh empty context. It'll only see the output, not
78:09 the journey that we took to get there. No emotional attachment, although I
78:12 think this is unnecessary at theorphization. And it'll catch what the
78:16 reviewer missed. So, uh, let me show you guys how this actually looks like in
78:19 practice. Here I have this app that I developed a while back for a video on
78:23 vibe coding, and you guys can check that out in the description if you're
78:25 interested. It's where I basically put together a full endto-end system that
78:29 allowed you to um, design and then syndicate a bunch of content. So, you
78:33 know, this is just some app, right? This app, I don't even know if it's fully
78:36 functional. Okay, no, it isn't because I had to turn it off. But hypothetically,
78:40 there's a big code base here, right? And so what I want to do is I want to use
78:44 this app to show you guys how an un um biased code reviewer would take a look
78:48 at the code that a previous agent had written, in this case Gemini, and um and
78:52 improve it. So what I'm going to do is I'm going to go find this repo. Okay.
78:55 And I found it over here. It's in the Splinter repository. That makes sense.
78:58 I'm just going to open up a new Cloud Code instance. And then down over here,
79:04 I'm going to say I'd like you to use, and I just need to make sure I know what
79:08 the skill is called. agent review on the Splinter repo. It's
79:14 let's just say folder. It's in the parent folder so that it knows where
79:17 this is. That way I can still execute it within this um business uh workspace
79:21 which I found a much better way of organizing things. And while it's doing
79:25 that, I'm going to open up the skill.md. So what the skill.mmd does is it spawns
79:30 sub agents to review, simplify, and verify output. It uses after completing
79:34 many non-trivial implementation tasks and it triggers on the words review this
79:38 agent review self-review or you know slag agent-review and you can see it's
79:42 already doing this. It's um spun up a sub agent called review splinter
79:46 codebase. And what this does is it reviews it for four things. Correctness,
79:51 edge cases, simplification and then security. Now like do I know how to do
79:55 all this programming under the hood? No, I don't. But these agents certainly do.
79:58 And so we can take advantage of that by having an agent with zero context. this
80:03 one here, review that entire workspace um sort of independently and
80:06 objectively. And now it's doing a bunch of reading and it's going to integrate
80:10 that with the suggestions of this model to give us a much higher quality output.
80:13 All right, the Splinter code review just finished up and we found 22 issues
80:17 across the codebase. There's some critical ones here, some high issues
80:21 here, some medium issues here, and then some low issues over there. Now, it's
80:26 asking me if it wants me to start fixing any of these, and I'll say absolutely.
80:30 And the whole idea behind this now is we're we're capable of looking at this
80:34 completely objectively, you know, like I asked the initial model Gemini when I
80:38 made the uh the app in the course like multiple times, hey, are there any
80:40 issues here? Hey, are there any ways to make this better? Hey, what do you
80:43 suspect is a problem? And just couldn't find it because it was so polluted by
80:48 its own biases. Now another model can and it's very similar to like peer
80:52 review in like academic um circles. It's not that like you know you're dumb for
80:56 coming up with this codebase like how dare you. It's just that as you work on
81:00 things more and more and more, you tend to see things a little more narrow and
81:03 more narrow because you've explored a bunch of other possible paths. And the
81:07 reality is the fact that you explored those paths and those don't work don't
81:11 necessarily mean that if somebody else explored one of those paths, it wouldn't
81:13 work either. And so this is just a way of remaining as objective as humanly
81:16 possible, which is obviously a very valuable thing to do when you're doing
81:20 things like creating applications, code, um, you know, sales, marketing, and all
81:23 the various things that AI agents allow us to do. Next up, I want to talk a
81:27 little bit about prompt contracts. For those of you guys that don't know,
81:30 earlier on we chatted a little bit about a definition of done, right? Well, vague
81:35 tasks, aka tasks that don't have clearly defined definitions of done, are
81:40 basically the number one problem nowadays with what I would consider to
81:44 be people's like disillusion with AI agents. Like when a total novice starts
81:48 using AI and then they dive into some agent coding platform and then they just
81:53 say, "Hey, build me a Netflix 2.0. make me a million dollars, make no mistakes.
81:57 Um, because of their extraordinarily poorly defined definition of done,
82:01 because they're poorly defined goals, because they don't give it any
82:04 constraints, because they don't give it any failure conditions, uh, that model
82:08 is just not going to do any any get anywhere near as high quality and end
82:11 result as if they did just follow a simple little uh, step-by-step process.
82:16 And so the step-by-step process obviously you could learn, but you could
82:20 also just like hardcode it as a skill somewhere in your workspace or as uh you
82:23 know something in your cloud NMD and then just force your model to always
82:26 have this information before you proceed. And so for instance, if you
82:30 give it a vague task like build a rate limiter, okay, it'll do pretty poorly.
82:34 But the whole idea behind a prompt contract is you basically make the user
82:37 who puts in a request like this sign a mini contract and just say, "Okay, cool.
82:41 The contract is, you know, here's what your goal is. Here what your constraints
82:45 are. here's what your format is and here's what your failure is. Are you
82:48 good to go? If the answer to that question is yes, now the model has
82:51 actually gone through the step of defining your goal, your constraints,
82:55 your format and your failure. And so all of your definitions are done, all of the
83:00 various uh kind of technical spec requirements here are much more laid out
83:03 and then the model sort of has a lot easier of a way of going about things.
83:07 And so this is very similar if you guys are aware um to like this idea of
83:11 scopes. Now I, you know, I run like a freelance education platform, like an AI
83:17 automation agency education platform. And so scopes are a really big part of
83:21 like a successful project. Um, and so I teach people how to define like really
83:24 precise and concrete scopes. Uh, whether you're doing, you know, a small project
83:27 for a client or working with some large enterprise business or something like
83:31 that. And like a real real common issue is scopes just tend either to be way too
83:36 vague and so people don't actually clearly define them or they end up way
83:43 too restrictive in so far that people you know in a in an attempt to
83:46 counterbalance the vagueness. They end up going like way too specific and then
83:49 the scope ends up being like so restrictive that it's like you know
83:51 you're a slave to it and you can't change anything. And so prompt contracts
83:55 sort of help you navigate the the thin line between too vague and too
83:58 restrictive. And it's very similar in nature to like giving a contractor a
84:03 task and then the contractor clarifying with you before they actually do the
84:08 task which I think you know is clearly a consequence of agents pushing all of us
84:12 more towards like management style positions where we just manage the
84:15 inputs and the outputs of these things. So a big fan of defining these clearly.
84:19 So what does this actually mean in practice? Well, there's obviously a
84:22 million and one different ways you can define prompt contracts. The way that
84:25 I've decided to do so in this demonstration is through a skill called
84:29 prompt- contract. And so basically before implementing any non-trivial
84:34 task, the skill forces you to generate a structured prompt contract with goals,
84:37 constraints, the format of output, and then failure. So the idea here is you're
84:42 treating it just like a spec or a scope of work. Any task that produces code or
84:46 some configuration settings or something like that needs to go through this
84:49 process. And then this model will sort of selfanalyze the request before
84:54 drafting a four section contract and then presenting it for approval. This is
84:58 almost uh similar in nature to like the plan mode that a lot of these um agent
85:02 platforms now have. Like in cloud code for instance, it can enter plan mode and
85:05 give you a brief little plan and have you approve the plan before it proceeds.
85:08 It's just this formalizes it as a contract. And no, you're not signing
85:11 your life away with cloud code when you do this. Um, but you know, it's a simple
85:15 and easy way to make sure that you get more repeatable and consistent and
85:18 accurate outputs every time. So, why don't I actually do this? Use prompt
85:24 contracts to define this task. And then I'm just going to pretend that I'm
85:26 giving it a really simple query. I'm just going to say, I want you to build
85:33 me a beautiful site for leftclick.ai. That's my um agency. So, what it's going
85:36 to do is it'll begin by invoking the skill prompt contract. And I mean,
85:40 beautiful site is such a subjective term, right? right? I mean, like, what
85:42 the heck does that even mean? And so, the model is going to be essentially
85:46 forced to ask me for more context on what constitutes a beautiful site to me.
85:50 And in this way, we'll get a much higher quality site or app or whatever the hell
85:53 at the end of it. Likewise, you could do this with any business task as well. It
85:56 doesn't just have to be like a design task. Um, you could set up a prompt
86:00 contract for, hey, email these 45 people. And it could ask you like, oh,
86:04 like what spec, you know, specifications do you do you want to confirm that
86:07 they're emailed? And what do you want the emails to say? And what's the goal
86:10 of a successful thing? and like, do you have any failure parameters? If we only
86:15 email 44, is that okay with you? Right? It basically forces it to be a lot more
86:18 clear and then concise. So, what's happening now is it's gone through, it's
86:22 actually accessed leftclick.ai, that's my current website, and then it's um
86:25 getting a bunch of screenshots and stuff like that. And the reason why is because
86:28 it's attempting to build up context for the prompt contract. So, its first step
86:32 was to analyze the request, right? What it's going to do is it'll identify what
86:36 done looks like. It'll identify some implicit assumptions. So, what am I
86:39 about to force the model to assume without being told? Well, obviously an
86:42 assumption is I already have a website, right? And so, it's going to go through,
86:45 take pictures of my website, and see, well, if Nick wants something different
86:49 from this, why? And then, it's going to sort of make its own judgment to that
86:51 end. And now, it's actually giving me the contract. So, the goal is a single
86:56 page marketing site for leftclick. Here are some constraints. You know, we want
87:00 smooth scroll animations under 500 lines of HTML. The format is this. There
87:05 should be these sections. subtle animations fade in on scroll hover
87:08 states. A failure is if it looks like a generic Bootstrap template. A failure is
87:13 if it's broken on mobile. A failure is if the animations are janky. The failure
87:16 is if the file exceeds 500 lines. So, I actually really like this prompt
87:19 contract. It's really simple and straightforward. So, I'm actually going
87:21 to say go ahead and build it. But what's cool is, you know, we're now actually
87:24 having a conversation about this. We're actually agreeing on, you know, what the
87:29 end result is going to be. And this is actually really similar in nature to the
87:32 other thing that I want to talk to you guys about which is um kind of related
87:37 and orthogonal to prompt contracts although it is a little bit different.
87:40 And this is called reverse prompting. Now reverse prompting is in a similar
87:45 vein a mechanism used to clarify the quality of a prompt and improve the
87:49 probability that it ends up okay. And basically the way that this works is
87:52 instead of just like forcing the model to give you this contract and having you
87:56 sign off on it, it it takes it one step further. it actually forces the model to
87:59 ask you some clarifying questions ahead of time. So rather than just give you a
88:02 spec sheet and say, "Okay, we're good to go." What reverse prompting does is it
88:06 um has the model ask you a bunch of questions that you maybe didn't even
88:09 think that you had to answer, the model then takes all bad context and then
88:13 feeds that into a prompt contract later on. Okay, so step one is when the user
88:16 gives a task to an AI agent. So I don't know, this is like a website, right?
88:20 Step two is the agent asks five clarifying questions back to the user
88:24 before starting. Step three is when we answer and then the agent builds the
88:27 correct thing on the first try. So significantly improves one-shot
88:29 potential. And then if we didn't have reverse prompting, there'd be a lot of
88:33 like wrong implicit assumptions here which would result in, you know, the
88:36 probability of a oneshot, which is just when the agent does it in literally one
88:41 request uh going down quite a bit. And so similarly, I also have a reverse
88:46 prompt skill over here. And so if I go to this reverse prompt skill, you can
88:49 see the way that this is set up is before implementing any non-trivial
88:52 build, ask the user five dynamically generated clarifying questions to
88:55 surface non-obvious preferences, assumptions, and constraints. So when to
88:58 trigger before starting the implementation, step one, analyze the
89:02 request, figure out some stated requirements, implicit assumptions,
89:06 some decision points, failure modes, and taste dependent choices. Right? And so
89:11 likewise, if I instead wanted to build, let's say something for
89:17 build a beautiful site for 1 second copy, which is my old content writing
89:19 company, which we just had to shut down a few days ago. U as you guys can
89:23 imagine, content isn't super in these days. Oh, and then um use the reverse prompt
89:32 skill and chain it together with prompt contracts after. What you could see is
89:37 we're now engaged significantly more than we were before. Before I just say,
89:41 "Build me a beautiful site." Probability that it gets what I want right on the
89:45 first try. Pretty damn low. What it's doing now is it's asking a bunch of
89:48 clarifying questions to confirm whether or not, you know, this site is as I want
89:53 it to be. And then after I feed it back that information, it'll then take that
89:57 and use that to construct essentially that that prompt contract that we had
90:00 before. So here's what the conversation looks like. What's the primary goal of
90:03 the site? Brand credibility, salesfunnel, lead genen. You know what I
90:07 want is just brand credibility. Should it be a single static page site or
90:10 should I build it in some other framework? No, I wanted a simple site.
90:13 What's the vibe? You know, it's AI content writing. You know, should I do a
90:17 clean modern SAS aesthetic? Think linear versel? Do I want something different?
90:22 Yeah, I want like linear but white. You know, should I generate the copy from
90:26 context or use some placeholder content? No, you're cool. You can generate it
90:30 from here. Now once we've clarified everything, what this model is going to
90:33 do is use all this information to outline the prompt contract using the
90:36 prompt contract skill. And now you can see it's invoking this skill as well.
90:40 And here we have a contract. It'll be a single page static site for 1 second.
90:44 Copy linear weight aesthetic 5 seconds deploy ready. Here's some constraints.
90:47 Here's the format. Maybe I don't like the format. Maybe I don't want it inside
90:50 of active, you know, I want it somewhere else. Uh but anyway, in in this case,
90:56 maybe I want to look good and build it. Now, just to show you guys an example of
90:58 how much higher quality we can get when we actually do this, this is the um
91:02 website that uh it just built for us. I'm just going to refresh this puppy and
91:05 take it to a new window because it gets cut off that window. This is here. We
91:10 have those cool sexy animations. As we scroll down, we also have some
91:13 information. Um it's it's light theme, right? We have these really minimalistic
91:17 requirements here. Information about myself, some services page, words from
91:21 happy clients, and then ultimately like a CTA. And so, you know, the reason why
91:24 I was able to get much closer to what I wanted, which was a minimalistic white
91:27 high-end aesthetic is just because, like, you know, I I had it outlined in a
91:31 contract. As I'm sure you guys can imagine, you can employ the same
91:33 approach for whatever the heck you want, whether you're building a site or you
91:36 are, you know, selling to people or you are doing some sort of bookkeeping or
91:40 accounting. It's all just about uh building out a very strong definition of
91:43 done. And the model can assist you with this. You don't actually have to sit
91:45 down and laboriously write it all out yourself. And that takes us to the
91:50 initial demo that we started with which was the multi- aent Chrome MCP manager.
91:56 Now basically at the very beginning of this course you didn't understand how
92:01 you know one agent could spawn a bunch of other agents. You didn't understand a
92:04 lot of like the parallelization plays. You also didn't understand that uh you
92:08 know you could have agents actually chat with each other and communicate. You
92:12 didn't understand the idea behind using one agent to verify the work of another.
92:16 you didn't understand the idea behind delegating to multiple different types
92:20 of models. What's really cool is the multi- aent Chrome setup that I showed
92:24 you guys where we had, you know, five or 10 agents all operating independently in
92:26 their own browsers, in their own workspaces. All of that just feeds off
92:32 of this this idea or this concept um of, you know, agents increasing their level
92:36 of communication with other agents. And so essentially if you think about this
92:40 logically, you know, if I were to do this uh with like a single agent, so
92:44 let's just say one agent, you know, it's not actually rocket science to have one
92:48 agent use a browser these days. There are built-in skills called MCPs, model
92:52 context protocols basically that you can just pipe in and immediately connect to
92:55 and it can do everything for you. Okay, it can it can launch Chrome and then it
92:58 can control things on the page and whatnot. It can do that. So, you know,
93:02 the issue is it just takes a lot of time. We'll receive the target URL.
93:06 We'll launch Chrome via the DevTools MCP. We'll navigate to the website.
93:10 We'll take a screenshot and you know in my case this over here was like um
93:16 specific for me which was just page or rather form fills. After that we'll
93:21 identify the form extract the form fields generate a personalized message
93:24 fill the fields and then click submit. Um but you know this is still something
93:28 that's occurring linearly and because of linear constraints you know unless you
93:32 are using uh I don't know like a Gemini flash model or you're using fast mode
93:36 and burning through your clawed token uh usage limits this is going to take a
93:39 fair amount of time. This process over here literally to just like launch the
93:43 browser could take 5 seconds. This process to navigate to the website could
93:46 take 5 seconds. Taking a page screen check could take 15 seconds. Identifying
93:50 the contact form could take a minute. You know, if you stack it all up,
93:53 basically what's occurring is this whole process here might take literally 2 to 3
93:58 minutes per form if you're operating naively using a slower model. And if
94:02 you're operating non- naively, if you're using a smarter model, then obviously
94:05 you have to weigh that against cost and and and token usage and stuff like that.
94:09 So, I don't know, let's hypothetically say in my case, I wanted to reach out to
94:14 uh you know, 1,000 people. Well, if it takes me 2 to 3 minutes to form, that's
94:20 1,000 * 2. That's 2,000 minutes, which divided by 60 is like 30 hours or
94:23 something like that, right? That's a very long time. It's going to take me a
94:27 whole day. So, instead of just doing one agent, what I'm going to do is I'm
94:30 basically going to give every agent its own both Chrome instance and then even
94:35 its own workspace and then open up its autonomy so that it can make um some
94:38 advanced decisions to basically help it build its own tooling if it needs uh in
94:41 order to like navigate website pages or whatever. Now, what this is going to
94:45 look like it's pretty similar to um our previous uh you know stochastic multi-
94:49 aent consensus prompt where basically we have a user up top. Okay. And this is
94:54 us. And what we're going to do is we're going to give all the context about our
94:58 task whatever it is that we want you know fill out a form or I don't do some
95:03 lead genen to an orchestrator agent which um in this case I'll do claude and
95:09 we'll just do opus which uh in my case is going to be 4.6. Maybe in your case
95:12 it's a better model. And then what that'll do is it'll spawn and set up,
95:17 you know, however many agents we want in separate windows. That'll then all in
95:21 parallel navigate to the site, find the form, fill the fields, and then do the
95:24 submission. And so basically, instead of it taking 2 minutes per form, what we
95:27 can do is we can actually submit, you know, however many forms. So I don't
95:30 know, let's say we have like 10 agents, we'd submit 10 forms in the same amount
95:34 of time it took to submit one. So maybe for us it'll be 120 seconds. And then
95:37 what we do is we just we just increase this as necessary. I mean I could
95:41 theoretically have 500 operating if I had the computing power. So you know if
95:46 um previously it was one form in uh what did I say 2 minutes and that means the
95:52 form per minute rate is like 0.5 forms a minute right but now if we spin up 10
95:57 and we do 10 in 2 minutes we're up to five a minute. If we spin up I don't
96:03 know 100 and we're up to 50 a minute. And you know, if my goal was 2,000 a day
96:06 and we're at 50 a minute, then obviously 2,000 divided by 50 means we can get
96:10 this whole thing done in 40 minutes. And you know, depending on the the the list
96:13 and whatever the heck you got, obviously the constraints change, but um this is
96:17 how you can have multiple Chrome instances operating simultaneously it
96:21 navigating the website and stuff like that. What I have here is a skill called
96:26 multi-- aent-chrome. And again, this is something you can implement using
96:29 whatever um context framework you want, whether it's a skill, whether it's like
96:33 a claude, Gemini, or agentmd, whatever the heck you want. What this basically
96:36 forces it to do is to orchestrate parallel browser automation using
96:40 multiple Chrome DevTools MCP instances. And this is used when a task requires
96:43 doing the same browser action across many targets simultaneously. So some
96:47 good examples are submitting forms, filling apps, scraping pages that need
96:50 JavaScript rendering and and whatever. And so what's occurring down here is
96:55 basically we have a tople business workspace which is sort of the folder
96:58 that I'm in right now. And this actually interacts with a bunch of Chrome agents
97:03 which all have their own little MCP servers their own little um claw.mds and
97:08 so on and so forth. And then they all communicate with a centralized chat and
97:13 if they run into problems on websites if they have any reports they want to give
97:15 basically what happens is this orchestrator just checks the chat every
97:20 30 seconds or so. Okay. So the very first step is it determines how many
97:22 agents are needed. Then it launches all the Chrome instances. It resets the chat
97:26 file because uh you know previous runs may have that. And then you can see how
97:29 every individual sub agent actually monitors its own um task list by
97:34 basically just pumping things into a chat. This is one of the simplest and
97:38 easiest ways of getting this specific design pattern done. As mentioned, you
97:41 guys can get this down below if you want, but I'm just going to give you
97:43 guys a simple example, which in my case is going to be just finding Vancouver
97:46 rentals because I'm, you know, considering getting a rental um down
97:50 there. And so, you know, rather than have it give you crappy results, this
97:53 thing can actually navigate like Craigslist, Facebook Marketplace,
97:56 Kijiji, whatever the heck you want. Uh, and the the specific script is right
97:59 over here. So, hypothetically, what I'll do is I'll just open up a new window.
98:02 And then I'll go over here and then I'll write uh I want to find a rental in
98:09 Vancouver. Needs to be 15 minute walk from the Granville Sky Train station
98:17 downtown. use multi- aent Chrome to navigate through sites and give me high
98:27 quality sleek places under 2.5 let's say 2K to 2.5K other restrictions like one bed one bath
98:38 reasonably near the water needs AC built in okay so I'm giving it a highle um you
98:44 know piece of instruction and sorry what meant to do is actually do prompt
98:47 contract after this. And now I want it to give me like a very clear contract.
98:51 So it's going to give me a list of 5 to 10 rental apartments. Why don't we say
98:56 20 rental apartments? And then I'll say 1.2 km is fine. We'll say near water,
99:02 south of Drake or west of Bard. Okay, I'm just going to make some changes
99:05 here. And then I'll say that sounds pretty good. Go for it. And now it's
99:09 going to actually launch the multi- aent chrome scraping. So it's then going to
99:12 invoke the skill. I'm just going to keep my hands off. What it'll do next is
99:17 actually spawn um four parallel Chrome agents, one per rental site. So, it'll
99:21 determine that there are four rental sites that it's going to be running
99:24 through. And it'll just have one Chrome instance sort of do everything there uh
99:28 per site. So, now we have the four instances. I'm just going to open this
99:33 up here. Open this up here. I'll move this one down over here. I'll also move
99:37 this one down over here. Obviously, you could use an approach like this for
99:41 pretty nefarious purposes. Um, so you do have to be cognizant of that that a lot
99:46 of people and websites are probably um, you know, they're looking to verify
99:49 whether or not you are a person. And so there are multiple things you can do to
99:52 get around that if you so wanted to, like using custom um, browser
99:56 fingerprinting and whatnot. And I think that's a story for another course
99:59 because I don't really want this course to be accused of just showing you guys
100:02 how to spin up like 500 Chrome instances scraping all sorts of illicit
100:05 information on the internet using unique browser fingerprints. But uh that stuff
100:08 is definitely possible and there are probably like a lot of people doing
100:11 stuff similar to this right now. They're just way farther ahead in terms of their
100:15 uh you know understanding of a agents and stuff like that. Now after the 30
100:18 secondond or so wait time these will receive their instructions and they'll
100:20 actually check the main thread and then they'll load in their websites. So this
100:25 one up here spawned li I've never used that site before. This one's
100:28 padmapper.com which is another one. You know these are all like websites and
100:32 resources I probably would not have looked at. And as a result I'm going to
100:36 get more of like a search spread. I'm going to again a big chunk of the search
100:39 space much faster than if I were to have done all this stuff manually. What's
100:43 cool is these can zoom directly into pages for me. These can click on links
100:46 and stuff like that. They're obviously modifying filters and and whatnot
100:49 autonomously so that they're not just getting a bunch of bogus results. And at
100:53 the end I get a high quality filtered list of apartments that are, you know,
100:56 within my specifications. Okay, I just turned my camera off because I wanted
100:59 some additional room on the bottom left hand side to really drill a few
101:03 important points home. The first is your context window. Now remember earlier how
101:08 we talked about the claude.m MD, the Gemini MD, and then the agents.m MD.
101:14 Over here, we just have Claude, but I just want you to treat this as all uh
101:19 three of them. That's not the only thing that gets injected, so to speak, in your
101:24 context. You have a variety of other things. Now, you have your system prompt up top. You
101:33 then have the claw.md agents.mmd and whatever else. You have a file at least
101:39 in cloud code called memory.md although there are analoges in other
101:44 coding platforms. And then you also have skills and tools. We've chatted a lot about MCP over the
101:54 course the last hour and a half or so. Right? Well MCP is a type of skill and
102:00 tool. We also have the actual skills themselves. So remember the agent
102:04 reviewer when we were doing sub agent verification loops. Well, that was an
102:10 example of a skill. If you think to the prompt contracts, those were examples of skills.
102:18 And the reason why I'm going into depth here is because each of these sections
102:23 can consume a tremendous number of tokens. And you're not given an
102:27 unlimited number of tokens to start with. Everything in life is finite,
102:32 including uh you know your claude or your Gemini context window. Now most
102:38 models right now are somewhere between uh I don't know if it's like 4.6 we're
102:44 probably talking 200K to 1 million. If we're talking Gemini, you know, we have
102:48 like 3.1 and there there a couple other ones obviously by the time you guys are
102:53 watching this there'll probably be more. And then you know you have uh GPT 5.4 4
102:59 and then codeex 5.3, but the 5.4 is coming out. You know, most models
103:04 nowadays have somewhere in the realm of between 200k to 1 million. And to be
103:11 clear, um, a token is not a word. A token is about 0.7 words. So if you think about it in that
103:21 vein, what this means is these 200k tokens sort of actually equate to
103:25 somewhere between like 140,000 words to about 700,000 words. Okay. Um but this
103:31 context window obviously gets filled up the more that you talk with it. And
103:36 unfortunately, one common and major problem in large language models,
103:39 specifically the types that we're dealing with in this course, are as time
103:45 goes on and you talk to it more and more and more, what you find is the average
103:51 quality goes down. So quality as a factor or a byproduct of token
104:01 count typically starts pretty high up here at maybe I don't know 100%.
104:07 And then the longer and longer and longer the number of tokens in your
104:11 context the lower and lower and lower the quality gets. So maybe this is at
104:18 10k, maybe this is at 50k, maybe this over here is at 200k. And what that
104:23 means is let's just hypothetically say you're at your 199,000th
104:31 token. Okay, that means that on a equivalent query that you might have
104:37 previously scored 100% at at I don't know 5 or 10k tokens at
104:42 199,000 tokens you might only score 40% at. Now, these numbers I basically
104:47 pulled out of my ass to be clear, but the point I'm trying to make is the
104:51 longer the token count, basically the bigger the context length, the lower the
104:58 performance of the model. And so, understanding context windows and then
105:01 learning a little bit of context management, ways to proactively manage
105:04 all of these things, some of which you have control over and some of things
105:09 which you don't, is very important. It's also important, of course, because of
105:14 billing. The more tokens that you use up, obviously the more money that you
105:18 spend. And so, not only is it best from a quality perspective over here to try
105:23 and push to the left side of this graph as much as humanly possible, it's also
105:28 very relevant from a financial perspective over here because obviously
105:31 the more tokens that you use, the more money you spend. Case in point, just to
105:35 make this video, I've spent something around $500 or so in tokens. That's
105:40 because I'm using a particular agent uh fast mode which bills me directly
105:44 instead of just using a monthly plan. But the point remains any sort of
105:48 serious AI agent application will start spending and using a fair amount of your
105:53 money. Okay, just before I move on, I want to talk a tiny bit about the
105:57 differences between each of these in length. If I open up an actual clawed
106:02 instance here, open up one of these and then I go to terminal, which is the
106:08 current best way to visualize this. Then if I just maximize this panel size,
106:12 let's just make this as big as humanly possible. Then I go /context,
106:18 claude will show us all of the things currently consuming its tokens. I'm
106:22 going to zoom in here to make it really, really clear what's going on.
106:27 This right over here is your context usage. And as you can see, they've
106:32 illustrated this as sort of a series of squares where every square is, I don't
106:37 know, let's see, 1 2 3 4 5 6 7 8 9 10. Okay, every square is I think 2,000
106:43 tokens or so. And so what we're seeing is despite the fact that we have put no
106:50 conversation tokens, so we haven't spent any tokens at all on conversation,
106:59 we're still at 9,000 used. You're probably wondering where the hell
107:02 are these 9,000 coming from? Are they shadow billing me to try and uh rinse my
107:07 wallet as much as humanly possible? Well, a little bit. I mean this system
107:12 prompt here, okay, which is partially composed by your agents, your Gemini or
107:16 your clawmd and partially a few additional things is actually already
107:21 consuming 4,900 tokens. So 2.5% of my entire token count before I even send a
107:25 message is being used by in this case probably the claw.md. But in addition, you have other things
107:32 like memory files which are consuming 2,000 tokens. Okay. And the way that
107:37 they do that in cloud code is they use something called a memory MD which
107:41 stores your preferences and some some previous highle things. Then next up we
107:46 have skills which are consuming 1,700 tokens. What are these skills? Well, you
107:50 guys remember when we made a bunch over here? If I go to the top lefthand corner
107:54 where it says docloud skills, you know, every aenta coding platform has their
107:58 own configuration for this stuff. But the way that it works in cloud code is,
108:01 you know, you organize these workflows into these skills. Well, guess what?
108:06 These skills aren't free. In order for Claude to be able to use these skills,
108:11 okay, this multi- aent orchestrator, it needs to store all these tokens
108:14 somewhere and then give it to the model. And that's what's going on over here.
108:17 Right now, the actual messages that we've used are only at eight tokens. And
108:22 guess what? That's actually I think it's um this word here, context usage.
108:28 I'm not entirely sure, but I think context usage just because the way that
108:31 it's broken or maybe context usage plus this term here /context, you know, is
108:36 equal to 8 tokens, but know that, you know, that's that's it. That's it for
108:40 our whole token count. So, the other 158,000 of our 200,000 limit is
108:44 currently free. And so, I mean, this is a quick and easy way obviously to
108:48 visualize it inside of Cloud Code, but other platforms have their own
108:50 visualization mechanisms. Now, next up are these MCP tools. A way to look at
108:55 these mcp chromedev tools_click. What is that? Well, this is the tool
109:04 that allows Chrome to click on parts of the page. Remember earlier when we were
109:09 building that little nadn flow? Well, we were doing it by clicking on various
109:14 parts of the page. How about drag? Right? We can drag things. Get console
109:19 message. These are all basically buttons in some colossal spaceship. Basically,
109:26 we are in the cockpit with clawed code and we're telling it to do stuff for us.
109:30 We don't know what these buttons are. It does because it's, you know, the ship
109:34 technician or the navigator, whatever. And so, it's clicking these buttons
109:37 left, right, and center for us to do various things. That's how you can
109:40 conceptualize all these MCP tools and all these skills. And what I really like
109:43 about this is it breaks everything down. So here are our memory files, okay,
109:46 which I talked about the memory MD, the cloudMD. You can see this is being
109:50 contributed to in a variety of ways. We have a global cloud.mmd which is sort of
109:53 like a very high level one with some sparse instructions. We have the local
109:57 cloudmd and we actually have the memory down here and then we have all the
110:00 skills. You know what was really telling about that is despite the fact that in
110:03 this diagram conversation history is like the biggest chunk of it all. Notice
110:07 that in reality conversation history for us at least at the beginning was
110:11 nothing. you know, it was actually a tremendous number of tokens, about 10%
110:15 of our entire context window dedicated to just this little chunk. And that is a
110:21 problem because if you're not careful, your cloud MD with all its rules are
110:25 going to get really, really long. Your memory MD with all your preferences is
110:28 going to get huge. Same thing with all the skills and tools and stuff like
110:31 that. And your conversation history when you actually do get to conversating with
110:35 the model will be very, very, very small. you know, instead of starting, I
110:39 don't know, somewhere in this region here, you're actually because the
110:44 context window of all of your BS is so big, you might actually like start in
110:48 the effective area over here. And obviously, this is not what you want to
110:51 begin an agent conversation at because if you start here, then it's obviously
110:54 only downhill from there. Now, this takes me to the logical question of, you
110:58 know, hey, Nick, what happens when you run out of context? Because obviously
111:04 that's going to happen. Well, when there are 50,000 tokens, let's say out of
111:09 200,000, okay, no problem. You are having the full conversation history
111:13 with the model, the model basically gets literally every message starting from
111:18 message number 1 2 3 4 all the way down to, I don't know, message number 25. And
111:22 then what it does is it takes all of this context and then feeds it into its
111:26 big neural network to generate message number 26, right? However, when we get
111:30 to a certain length, right? I don't know, let's say message 50. Obviously,
111:34 there's no there's no tokens left anymore in in the context window. Maybe
111:37 we're like 199. Well, not actually probably like 155k out of 200k. Well,
111:42 what happens is all models now have some sort of um what's called like autoco
111:47 compact limit. I mean, basically all of them have adopted this convention where
111:52 when the um number of tokens that you're using, let's say the limit is right over
111:58 here. when the number of tokens that you're using gets to this point, okay?
112:04 You know, it fills up then fills up, fills up, fills up and fills up. What
112:10 happens is this triggers a mechanism called compaction where we take all of the information
112:18 here, which is I don't know maybe like 80% or so of the whole context, and then we
112:25 compress it. I want you to imagine right now there's like a a big hydraulic press
112:30 type thing over here and it's pushing all of this context down. It's squishing
112:35 it. Basically, what's going to occur is we're going to erase the vast majority
112:40 of this and then, you know, instead of it consuming 80%, we're going to cram
112:43 all that information into maybe I don't know 30 or 40% or so.
112:50 And so that is um called compaction. and it occurs on a relatively regular basis
112:55 across the uh the model ecosystem. The issue with the compaction or you know
112:58 compression or whatever the heck you want to call it the same idea is during
113:03 this summarization and like densification process we are going to
113:09 drop outputs from tools. we are going to remove some information and context that
113:13 might actually be useful to you now at you know message 50 for message 4 which
113:17 might actually like eliminate a mistake and so because of that you know you are
113:21 going to lose some of the quality um the benefit is obviously you will
113:24 significantly improve the information density and what do I mean by
113:26 information density I mean literally like if it's like um hello
113:35 how are you doing I mean let's say somewhere in your context you have the term
113:40 The sentence hello, how are you doing? Well, hello is actually two tokens. How
113:46 is one, are is one, you is one, doing is three along the question mark. So, in
113:50 total, if you count all this stuff up, depending on the uh thing you're using
113:54 to do the embedding, that's uh eight, right? Well, you know, compaction is
113:58 literally going to take this sentence and then it's going to compress it. So,
114:04 it's going to say, "Hi, how are you?" So now instead of that's one token here,
114:08 one token here, one token here, one token here equals four, it will
114:13 literally get all of the context that it can and then try and squish it so that
114:17 the same meaning is available in fewer tokens and fewer words wherever
114:20 possible. And then it'll just run this naively across your entire um context.
114:24 And so this is going to occur every single time. Obviously, it's something
114:27 that we want to avoid occurring if we have sensitive and important data, but
114:31 you know, it does allow us to continue conversating with models. before we had
114:34 context compression and some form of autoco compaction. Um, basically we
114:38 would just run out of tokens and we'd have to restart a totally new session.
114:41 So this just does sort of that intermediate step that most people were
114:43 doing before where they would like like take all of that, you know, try and
114:47 summarize it in some other model and then paste it back into uh into into
114:52 another one. Now that takes us to what most model practitioners are using
114:55 nowadays, which is a variant of something that uh people call the
115:00 iceberg technique. Now, in case you guys have never seen an iceberg before, I'm
115:04 Canadian, so we have them everywhere, including literally in the river across
115:09 the street from my house. Not uh actual icebergs, but little ice floats. The way
115:14 that they work is basically you have um a section that's visible above, which is
115:17 usually quite massive and quite intimidating. And then you're like, "Oh
115:21 my god, that's a really big iceberg." And then what you don't realize is
115:24 underneath the iceberg is actually like two or three times as big. And so above
115:29 the stuff that's like immediately visible or in our terms for the model
115:36 accessible immediately is actually only a very small percentage
115:42 of the total iceberg. And so in the context of our model what
115:46 we store above which is immediately accessible. It's just sort of the stuff
115:50 that's like visible to the plain eye is we'll store our memory. We'll store our
115:56 claude or our agents or our gemini.md. We'll store our local memory as well.
116:00 There's different types. There's global local. We'll store all of the current
116:04 task context. So everything that our tools are doing then maybe any active
116:09 file content. Okay. And so all of this stuff here is like basically always
116:12 accessible to us. It's literally just in our prompt. But then what people are
116:16 doing to reduce the total number of tokens that they require is they're
116:19 abstracting away everything else. and then they're just making it accessible
116:23 to the model if it needs it. What I mean by that is instead of putting all of the
116:27 files in your codebase in a prompt, what it does is it gives you a tool called
116:31 read. And what read can do is read can at any time read a file. Okay, but
116:35 instead of putting the entire file in there for now, all it does is it just
116:39 puts the titles. So if you say, "Hey, I want you to grab the um information on
116:43 the iceberg technique." And then in your workspace, you have a file called
116:47 iceberg technique.md, it'll know. It doesn't actually have to
116:49 read all of the files in your workspace. It only has to read iceberg.mmd. Right?
116:54 Same thing with the full codebase. You have tools like gp and glob. And these
116:58 tools are sort of analogous. Instead of reading a whole file, what this does,
117:01 this allows you to hone in on a specific segment of text. You know, if this is my
117:05 entire code file, okay, and hypothetically, let's just say it's
117:09 really big and you know, there's there's a lot of stuff, but the only thing that
117:14 I actually care about is this little segment over here. Then why would I load
117:22 all of the I don't know 10k tokens? I don't need to. Okay. Realistically, what
117:27 I can do as a smart model is I could use GP and Glob these tools to maybe hone in
117:32 only on this segment over here, which is I don't know, let's just say 2K tokens.
117:37 And because it contains some of the text before and some of the text after, you
117:40 know, this still usually gives us enough context to um tell the model what it
117:45 needs in order to to finish its function. Now you also have web data via
117:49 web fetch. Web data is pretty cool. You can kind of think of it as the same
117:51 thing. Obviously it doesn't have access to the whole internet, but it can make
117:55 search queries, right? And so because it's able to use some general reasoning
117:58 when you say, "Hey, what's the iceberg technique?" First, it's going to start
118:02 by looking for file contents called iceberg technique. It can't find any,
118:06 maybe it'll quickly type fisceberg to look through the codebase. If it can't
118:09 find that, you know, maybe it'll look through some other things like uh memory
118:13 files and the skills library. But if it can't find that, it'll say, "Okay, cool.
118:16 So, we don't have this in our in our context window right now, and we don't
118:20 even have access to it within our space, but it's probably somewhere on the
118:22 internet. So, I'm just going to Google iceberg technique. And then what it'll
118:26 do is it won't even take the entire thing. It'll just grab tople links, and
118:31 then it'll look at the URL, you know, uh, and the URL is about, guess what?
118:35 Icebergs. I'm going off the map here, but hopefully you guys understand what I
118:38 mean. Then, if there's three URLs, one of them is about icebergs, then instead
118:40 of reading all of them, it's only going to read this one. And so it's sort of
118:45 like a successive narrowing of the lens until eventually it gets to, you know,
118:50 what you want. And it doesn't load the entirety of all of the context, but it
118:54 just has the opportunity to select all of this. You know, it starts here, it
118:58 goes here, goes here, goes here, goes here, goes here, and then goes here, and
119:01 then finally it's at its goal. You can do the same thing in a variety of other
119:05 ways. You can use bash, get history, and so on and so forth. But um essentially
119:08 what you want to do is instead of storing all of the information like the
119:12 full code base, the file contents, the web data, all the git history, all the
119:15 skills and everything like that, what you do is you just store the ability to
119:21 access it on demand. And then inside the context, okay, the tiny chunk that you
119:24 do, I mean in this diagram it says 10% 90%, you know, I think in reality it's
119:30 probably closer to 20 80 or maybe 30 70. Um here's where you store stuff that's
119:34 just like it it needs to be the same all the time. it always needs to be present.
119:37 There always needs to be some sort of patterns that are learned, some sort of
119:42 active file context or uh contents or current task context. You can think of
119:46 this as the difference between naive versus strategic context loading. Now,
119:51 way back in the day, and when I say way back in the day, I mean like 2023, good
119:55 god, I'm getting old. You know, when you were working with an agent, you would
119:59 dump in the whole codebase. You would honestly just copy and paste everything
120:02 and just hope to God that it that it knew what it was doing. Obviously,
120:06 because this was extraordinarily infeasible, there were so many tokens in
120:13 the context, tons of it was lost and you routinely ran out of context limits.
120:17 Well, nowadays what we've done is we basically built in a whole tool stack
120:20 where instead of all of the file, you just read selectively and only the
120:24 relevant functions. You know, you have a clawmd which is basically a compression
120:28 a compression function which just stores your preferences. skills instead of all
120:32 being read, you know, we only read a specific segment of them. This is
120:36 technically called the YAML front matter, which is just a tiny little
120:39 section at the beginning of the skill. You can actually see this if we go back
120:42 to the skill.md and I make this visible, right? Only this section up here is
120:47 actually, you know, you can actually see this up here. Um, the YAML front matter
120:52 is this segment. Only this segment is actually loaded into context until you
120:55 ask for, you know, more information about create proposal. And that's just
120:59 because this little space invader sees you. It has the ability to call a create
121:03 proposal skill, but it doesn't need to know all the rest of this because what's
121:06 realistically going to happen? Well, 90% of the time you won't even ask. There's
121:09 so many other skills it'll probably be using. We're also now doing things like
121:13 summarizing tool results. So instead of storing the entire thing in your
121:17 context, you know, we just store like a very short summary of basically the
121:21 input output. So way back in the day, I used to think that all of an agent was
121:25 really just its intelligence. So it was just the core model, right? Which in
121:30 this case would be Opus 4.6. But what I've come to quickly realize is
121:34 although models themselves are quite intelligent, it's really the
121:38 architecture that we've wrapped around it. I want you to pretend this little
121:42 space invader, it's kind of now in, I don't know, a house of some kind. And
121:46 the house has a little chimney with a fireplace. It has a little place where
121:50 it could, I don't know, roast a nice turkey. You know, has a nice bed that it
121:54 can go to sleep in every night. you know, this agent by itself probably
121:57 wouldn't last super long out there on the savannah. But because we've built
122:00 all this infrastructure around it because we built roads for it, we built
122:04 ways for it to communicate and stuff like that. You know, it's capable of
122:07 actually doing a lot of very economically valuable work for us. And
122:11 so, just like human beings, uh, way back in the day had to conceptualize the idea
122:16 of, I don't know, like a like a spear or something to hunt, um, um, saber-tooth
122:24 tigers on the planes, you know. So too do these agents use tools effectively to
122:29 solve problems in their environment and ultimately get us the users what we
122:34 want. Now obviously there's context window management like we just talked
122:38 about and that's for optimizing the usage of a specific model okay in the
122:44 choice of a model. But there's also the ability to choose different models for
122:48 different purposes. Now, throughout most of the course so far, what I've done is
122:52 I've just used mostly naive like Opus 4.6 agents in order to spawn other Opus
122:58 4.6 sub agents. And that was mostly for capability sake because at least in my
123:02 case, cost is not a a concern. And I really do want to eek out the marginal
123:06 quality um benefits wherever possible. But there are a lot of cases
123:10 specifically like enterprise and and big infrastructure ones where, you know,
123:13 people are actually comfortable making a miter tradeoff for quality. I'm going to
123:18 draw another one of my famous graphs. Um, people are comfortable making, you
123:23 know, a trade-off in terms of, you know, cost and then quality. Now, in a lot of
123:28 disciplines out there, biology, physics, chemistry, and stuff, there's this this
123:33 idea of like this inverted U curve, and you can call it whatever the heck you
123:37 want. Um, I think the actual name is the Yorky's Dodson curve.
123:42 And what this is is this is basically like the the optimal point
123:48 that combines two different factors. And so in our case, if we're optimizing for
123:51 both cost and quality, okay, simultaneously, not just cost, not just
123:55 quality, you can imagine that the optimal place to choose on this graph is
123:59 probably going to be something like over here. It's going to be it's going to be
124:02 somewhere around here. If you wanted to minimize cost exactly, we'd probably go
124:05 like way over here. But obviously, we care about quality as well. So, we're
124:09 going to push up a little bit, right? We don't want this point because one, it
124:14 costs a lot. And then boom, two, the quality is pretty low because despite
124:17 the fact that quality is really really high, um you know, cost is like many
124:21 many times higher than it was over here. And so this is sort of like our our
124:25 optimal point. And you know, a lot of large enterprises, since we're dealing
124:28 with hundreds of millions of dollars here, um, are actually comfortable
124:31 making a little trade-off where, you know, if this quality is 85% and then
124:36 this one here is, I don't know, 95%, they're okay taking like a like a 10%
124:41 hit here, if it means that they also reduce their costs by, I don't know,
124:46 40%. Or something like that. And this is really where all this stuff comes in.
124:49 Okay? So, I don't mean to talk your ear off here. It's not super important, but
124:52 uh basically uh what a lot of people have taken to doing now is doing a 60/30
124:58 10 rule where they'll use some top level agent router which is sort of like the
125:03 orchestrator in our multi- aent Chrome window example. And what that agent
125:08 writer does is it calls dumber models and then it assigns different strengths
125:13 to tasks. So that you know if you're giving it a really simple task and you
125:17 say hey you know I just want you to classify this into one of three
125:21 categories and it's really dumb. It's like you know red, blue or green, angry,
125:27 serene or or healthy or whatever the heck. um you don't have to use like
125:30 Opus, which is like space age intelligence and costs you a ton more in
125:35 token cost in order to get that done. Instead, you know, you can get all the
125:38 way down to like a haik coup or maybe a Gemini flash model or something like
125:42 that. Likewise, if you have some other task here and maybe that task requires a
125:45 lot of um I don't know, research or something and you say, "Hey, I want you
125:49 to go and compile 200 million tokens worth of stuff and then um give it to me
125:53 in a big report." Well, you know, you probably don't want the dumbest model to
125:56 do that for you, but you also don't need the most expensive model. So maybe
125:58 you'll use something like a sonnet or like a a lower level GPT model which
126:03 might cost 2 or3 million uh $2 or $3 per million tokens instead. And so you know
126:08 this allocation where you have 60 3010 if you think about it like sort of a pie
126:12 chart. Um what you do is you designate you know the vast majority of your token
126:18 usage to stuff that is in that first category which is kind of you know
126:22 dumber. And then what you do is you do the other 30% or so in sort of that mid tier and then your
126:30 really really really smart models, you know, they do the highest level tasks.
126:33 And basically what'll happen for the most part is this would be, you know,
126:39 your Opus 4.6 or your Gemini 3.1 or your GPT 5.4. It would be
126:45 responsible for routing decisions and obviously you want the smartest model
126:47 possible for that. But all of the heavy lifting, all the contacts and stuff like
126:51 that is um through sub agents that are spawned either Haiku, Sonnet or uh you
126:57 know, I don't know if if you wanted to do a really smart call, then you'd
126:59 obviously spawn an Opus sub agent like me as well. And if you do all this, you
127:03 could significantly reduce the cost. I mean, like just just think about it
127:06 mathematically. If previously you were doing 100 million tokens times, you
127:13 know, $5 per 1 million tokens, what's the cost there? Well, that's obviously
127:17 going to cost $500. And so that's like your opus only, right? But if you did 10
127:25 million * $5 plus 30 million * $3 plus 60 million * $1. Well, what's the total cost going to be
127:39 now? Well, it's going to be 60 + 90 + 50 or in total 200. And so you know
127:48 200 expressed as a fraction of 500 is 40% of our total cost and we will have
127:54 just saved you know 60% with probably minimal impacts on quality because the
127:59 things that we're now spawning you know dumber agents um to do are things that
128:04 to be honest the quality was already okay a few generations ago back at the
128:07 haikus and the sonnetss. So just to give you an example from something that I do
128:11 pretty often, okay, which is going to be some form of lead scraping. What you can
128:16 do is you can actually traverse a very large portion of the internet using a a
128:20 relatively dumb model. These haiku models, what these do is these um scrape
128:26 a vast majority a vast amount of internet data. Okay, all of the code of
128:30 uh I don't know, let's say 10,000 websites or something. And then in doing
128:33 so, they just like that little magnifying glass, use some sort of GP or
128:38 extraction prompt to look for things that are formatted like email addresses.
128:42 So if you have like something and then it's an at and then it's a, you know,
128:46 the term gmail.com, odds are this is a real email address, right? And you store
128:50 that to a database. And because this is just such like a mass data application,
128:53 you use haik coup that drives the cost really, really low. Well, then maybe um
128:58 the actual enrichment point, you know, takes significantly more intelligence.
129:01 And so maybe here we'll use sonnet and it'll cost us uh 0.008 cents per lead.
129:09 08 per lead. The actual outreach part is mostly templated. So we'll use sonnet
129:12 for that as well. And then maybe at the end we just have a quality review step
129:15 to make sure things aren't absolutely nuts. Well, when you do it this way, um
129:20 you know, the math ends up being uh 08 plus 05. So that's 013 plus 01.014 plus
129:29 015 is 029. And then if you were to go 100% opus, then it would be I don't know
129:34 uh about 12 cents or so per lead. And so on a list of let's just say a thousand,
129:38 which is approximately how much I'm sending a day right now. I'm much
129:42 farther down than my maximum. But if we multiply all these together, we move
129:46 that one to three decimal points over to the left. Then that um ultimately would
129:51 be $15 a day or you know $450 a month. Well, instead I'm doing that for literally one quarter
130:00 of this or something like, you know, $120 a month instead. Obviously, I'd
130:04 much rather the latter. And if my quality is only going down a few
130:07 percentage points because of that Yorky's Dodson curve, you know, I'm okay
130:13 being over here instead of over here because this gap to me is fine on tasks
130:18 that aren't super high and high quality. Then, uh, this is a very, very efficient
130:22 stack. And you know, the bigger and bigger my company gets, whoever I'm
130:27 working with, the more the cost per lead is going to be important versus the
130:31 actual quality. I've included just a little LLM API pricing cheat sheet. I
130:35 don't expect this to be super relevant or useful to you guys. There are a lot
130:41 more models for OpenAI and Google, but I am actually using um not this model
130:45 series anymore, but this one here for some queries. I'm also using the Flash
130:49 model series for some queries as well. And then what's really cool is a lot of
130:54 them offer what's called a batch API now where you can submit a bulk number of
130:59 requests over um simultaneously and then if you're comfortable waiting like a day
131:04 or so what the companies do is they batch it and then they serve your
131:08 requests during periods in which they have very low inference or low
131:11 competition. So maybe in the middle of the night or something like that. And in
131:14 doing so they actually get to load balance. Like if you think about it,
131:18 like if this is like a day and then this is their load like on their servers and
131:21 their neural networks and stuff, you know, it probably peaks somewhere around
131:25 like noon and there's probably like a couple things and then it's like low
131:28 during the during the day, right? I don't know, this is like 4:00 a.m. What
131:32 they'll do is they'll actually take all of your queries, batch them, and then
131:35 they'll just like run them over here when there's very little competition.
131:39 And then later when you know the things go up and stuff like that again, that's
131:42 okay. And in doing so, what they want to do is they want to shift some of the the
131:47 really top end of all of these users um to the low end to basically fill this so
131:50 that they have a lot more like dependable load instead of these like uh
131:56 jagged peaks and whatnot. But anyway, don't worry too much about that. I just
132:00 wanted to cover um some LLM pricing principles as well, so you guys know not
132:03 only how to manage your contacts better, but also how to um save, especially when
132:08 you get into more sophisticated multi- aent setups like I've been showing you.
132:12 And that's it. Thank you guys very much for watching this video end to end. If
132:15 you guys have made it all the way to this point in the course, you're part of
132:19 like the 2 or 3% that actually do. Um, I'd really appreciate a big solid if you
132:22 could do me a favor and subscribe to the channel. Something like 70% of you
132:26 aren't, which significantly hurts my reach and uh, despite me hating asking
132:30 for it, it does help the channel grow. So, if I've given you guys any value
132:34 whatsoever, please do that. You can also send me over a comment down below asking
132:38 any question about any point in the video. Um, I'm much more engaged than
132:41 the average YouTuber. So, the probability that I will reply is pretty
132:44 pretty far up there, I would say, statistically. Um, if you guys have any,
132:48 you know, suggestions for future videos or future courses, please drop them down
132:52 below as well. And above all else, keep learning and growing with AI agents.
132:57 This is by far the biggest and most impactful of economic changes that I
133:00 think any of us will see in our lifetime. It's a blessed time to be
133:04 alive in general. You might as well not waste it. Make the most out of it. All
133:07 right. Um, thank you very much. Feel free to use the chapter headings to
133:11 revisit any section in the course. And uh looking forward to seeing all y'all
$

AI Agents Full Course 2026: Master Agentic AI (2 Hours)

@nicksaraev 2:13:14 28 chapters
[AI agents and automation][content creation and YouTube][developer tools and coding][productivity and workflows]
// chapters
// description

🔥 Join Maker School & get customer #1 guaranteed: https://skool.com/makerschool/about 💎 All course files: https://drive.google.com/drive/folders/1Sjo54rul7zJXDES-Zu8snNuhYS0SC-IL 🎙️ Listen to my silly podcast: www.youtube.com/@stackedpod 📚 Free multi-hour courses → Claude Code (4hr full course): https://www.youtube.com/watch?v=QoQBzR1NIqI → Vibe Coding w/ Antigravity (6hr full course): https://www.youtube.com/watch?v=gcuR_-rzlDw → Agentic Workflows (6hr full course): https://www.youtube.com/wat

now: 0:00
// tags
[AI agents and automation][content creation and YouTube][developer tools and coding][productivity and workflows]