What Your AI Agents Really Are When You Strip Away The Hype

If you tell the AI look out that the humans are happy and it knows well if I administer these drugs they are happy when I'm just doing that. There's also another experiment of tasking an AI. If you think about AI agents, producing a product, mine everything on Earth, including humans, to reach that goal and then build spaceships to mine other planet. If you vibe code a multi-agent system and you don't know how to properly do this and it will pass test. It might look reasonable but then after the first months in production your CTO will be stomping in very angrily into your office because you burnt lots of tokens. You still need to be an educated operator.

However you choose to design your system, remember reasoning mistakes are recoverable, but ungoverned execution isn't. Once an agent can touch files, networks, APIs, or every implicit assumption becomes a potential breach. Your responsibility is to ensure that your agents are safe to fail. Today I have Nicole Königstein. She's writing the literal textbooks on transformers and AI agents for O'Reilly. And she's also the person who can tell you why your multi-agent system works perfectly in testing and falls apart in production. Nicole, welcome.

Thank you. Thanks for having me.

Yeah, it's it's such an honor to have you. I know your book is coming out and I have the honor to read a few of your chapters and I find it very very useful already. It kind of filled in some blank space when I'm building AI agents and some parts I wasn't paying too too much of attention.

Okay, glad to hear that.

Is there a version of you that you would love people to know you about, you know, beyond just the book that you're publishing?

Gosh, good question. Well, I think the important thing about myself is that I do care helping people to understand the domain on a deeper sense. So I'm more of a systems thinker. So I approach things from a different angle and when I write or teach or explain some things to people about those concepts. So I'm I'm trying to approach it from a different angle.

So I still get this question a lot especially from nontechnical audience like how would you explain what is AI agent to these non-technical audience?

Well, let me think. So to a non-technical person, I mean you need would need to think about or I would think about if I would first even explain what the underlying model is because it comes down to the underlying architecture, right? So depending on what you're doing actually with your agents because these are just small units in doing a task and they do it either autonomous or sequentially but you have underlying you have that kind of that intelligence and that comes from the model right and then for me since I'm mostly into well not mostly I'm a big defender of transformers. I'm I'm not using another architecture at the moment. So for me it would be first making sure they understand the technology the intelligence beyond that. Are driving the kind of a decisions. And so for me it would be like just telling them that transformers is an architecture in processing information.

And what it does is it breaks information into smaller pieces and learns then the relationships between both different pieces and maps the relationships of the task and task. When it learns and then of course we have those tokens and the tokens is just a small unit of processing and it can have different shapes different forms. So you can have language where the token is a word or subwords or small pieces of the word. Then you can have images right we have both multimodal models. So you can have images where you have patches for that.

You can have audio. So that might be little sound segments. Then you have videos where it's little frames or parts of a frame. And yeah, I think so the important part is to know that it's but all of that also the agents it's not only text right you can interact with it with via voice or via text and yeah you can give them also videos and they can process it yeah so that is the part of a transformers and then let me think that I don't lose my thread then if you look at it in terms of the agent so part of that is a part a small part of that is the intelligence part. So the transformer part I just explained and then the rest is the behavior and the processing around it. So small units of collaborating working and handing piece of work to each other.

When a user chat with ChatGPT they think that they're talking to an AI agent it's

It's understandable if you if you think about it right so like if you would explain and further if I would further explain it to a non tech person or even to a child about what all that technology is I mean how it learns and interact and it focuses the attention and things it does it's fairly similar to how we a way process information and perceive. And also if you think about it, the main core of transformers is attention, right? And we humans if we are in a crowded space or what wherever we map our attention to what we want or focus on focus on what we think is important and we then also we map over time behavior or relationships to let's say if we have a TV show and we watch that TV show we learn the characteristics of those roles and we can already kind anticipate what comes next because of that character over time, right? So if they make a certain face, we think, yeah, now that follows because that's just the pattern. Kind of natural that people kind of humanize AI because it's nowadays it's so close.

It's not like the one of the first bots like Eliza just having bad dictionary vocabulary and after a while starts repeating itself asking the same questions.

Like how can I help you or whatever and it nowadays they can adjust to the behavior of the of a human and I think what you said the first six chapters right so the first chapters of the book so you're you're not on the memory part so actually if you would have read the memory part you would have or this would even shed more light on it why it feels so human because what we can do is with giving those machines a state and I did something very funny at the beginning of that memory chapter is I had a stateless agent and I was asking it during a conversation or telling my name and telling it I'm writing a chapter about memory for agents and then after that I asked it again what my name is and it told me I don't know your name yet but if you tell you me your name I will call it by that you by that name but I already did and it was just stateless so that was kind of the point if we don't give them certain tools behind in act interacting like that they

Might seem a little bit naive or dumb it's the tool it's the machine machinery outside or around it and with that memory you can store different kinds of memory so that it can learn certain preferences of the users like they always want it in let's say American English and always with bullet points and they like their UI in dark and things like this and it stores that memory and over time over the interaction with that particular user with that particular human it adapts the style and learns how to hold that conversation kind of a preference they have and they will adjust because it's also just pattern recognition they are good at that so it feels very human

When you when in a sense would you agree if I say that an agentic system is kind of like a bunch of if then rules with prompts

Not fully in my in my mind agents are distributed or are distributed state machines. So you have that you've seen in the beginning of the book I make that kind of parallels to those finite state machines. So you have that fixed vocabulary and the fixed states and decision points where you're in where they can do things and it's kind of that way. They have a certain amount of actions. So tools interactions they can do and if you have multi-agents they are distributed over the system depending on the topology they have certain ways of interacting. So depending on if you have centralized, decentralized, independent things like this, the topology, the communication patterns, how they talk to each other, how they collaborate or not collaborate, how they delegate work changes. So that way in my mind they are distributed and depending on how you set them up. The kind of the interaction changes and also of course the failures and the behavior and how you need to take care of the memory and all of those things.

When you say topology, what do you mean?

The architectural layouts there are like topologies. It's hard to paint a picture not being able to paint a picture but for instance if you if you have centralized so centralized multi-agent system is very much like if you have a normal team structure within a normal organization.

So you have that orchestration which is maybe your tech lead within a programming team and then you have your sub agents. So all the programmers and they are the workers doing that and the tech lead knows the goal the overarching goal for that production team and they follow the lead right but maybe then you have a completely different organization and let's see see or think about you have a creative writing team or a creative agency or something like this. So what you do then is you have those people sitting in a room and do brainstorming together to come up with a concept or idea or even a research group. Right? So you have those independent minds but collaborating and have that blackboard or whiteboard there they are writing at summarizing their ideas and things like this. And this is adopted also for different topologies for different architectures for multi-agent systems.

A lot of people are talking about this concept this concept of autonomous agents. What's the difference between autonomous agents versus a generic AI agent system? And what the autonomous agents look like?

I think there are a lot of different opinions on that and I might have a certain one. So I gave last year a talk on that on agentic workflows and also on autonomous agents and I was telling them even back then still and I would do the same even now that we are not fully autonomous. Not really because fully autonomous in my mind would mean we give the agent the a way of redesigning themselves in what kind of tools they are accessing, how they access it, and in what kind of architecture. And not just us as engineers, as people, as humans setting it up and have them decide with in that framework because that is then still for me that finite state machine where you as human give them that vocabulary. So to X to Y to Z what whatever like different things they can do. They can communicate with certain agents. You have certain rules how they can improve themselves and things like this. So they are not looking at their own code and say hey that function did fail 10 times I need to rewrite it.

No and that would be for me a really an autonomous system or I would even go more into self-improving and autonomous going in kind of the same direction. So if they can self-improve and see their their errors then they can look at the architecture the communication with each other and start improving and maybe then say all right so that other agent there it doesn't contribute anything so we cut that one out of the system things like this or that tool isn't working

Can I can I understand it this way let's say if we are the manager of a team and then we have two employees, two human employees. One is a motivated person who's going to fix their own errors and who's going to learn even if we don't prompt them to learn and that's an autonomous agent. And another employee also a human if what they do is only we tell them that you do this task A B C D today or this week and that's all they do then they're they're the current state of agent agentic system basically.

I would say in that it goes in that direction. Yeah. There's there's no no no awareness of how to improve about certain things or task without prompting anything from the outside or something specifically programmed in.

Do you think that tells us something about what's missing in the current state of agentic system? How do we make them motivated so that they Well, it's also a design and engineer problem as well. Right.

Yes. And actually, I've I've been thinking a lot about that. And actually, to spoil the beans, my PhD research is about self-improving AI agents. So, this is why I have maybe a certain view on this. Look, I think like we know how we people learn, right? Like and I say every time I teach things about reinforcement learning, I'm kind of saying the same thing that is we know how to do this. Yeah. In our sleeps because we already do this over and over. So if we teach our kids how to behave, so we use reinforcement, right, to nudge them in the direction we want, we learn from our mistakes. We learn from the interactions with our friends and colleagues and also our paths learn a certain kind of behave and improve over time to get a certain outcome. Right? So for me it's the same or goes in the direction that you need a certain signal and certain motivation to nudge that system in the way or give them the system not only an incentive but also a metrics and ways of how to improve themselves.

And then of course if you would want to have them autonomous you would have to give them a way of yeah not really replicate or duplicate themselves but alter modify their own code and solve their own problems and I don't want to get into kind of Gödel incompleteness theorem or something but it goes in that direction or also if you heard about there's something very interesting and I think it was 1950s or something von Neumann thought about self-replicating probes and it also improves and modifies itself. So I mean the theory is around for ages so to speak.

Do you think we will have systems that can freely redefine their own architecture or reason about their goals in a robust and general way? Do you think this is this is coming?

I think it's not a question of if it's coming or not because I think at some point it should be technical technically feasible. The question is do we really want this? Because how do we guard this? Because I think like this is this is really hard not only from a philosophical standpoint but from a human yeah from a standpoint of being a human. This is this is really really like important to think about.

There are like I love reading science fiction and there has been and I'm I'm not talking about some doomsday thing specifically but there been like also like some thought experiments about well if you have an AI and you give the AI the wrong incentive if you just think about it if you tell the AI well look out that the humans are happy and it knows well if I administer these kind of drugs they are happy well then I'm just doing that right without the underneath the ramifications and problems you're just drugged all your life or other things there's there's also another thought experiment of yeah tasking AI if you think about AI agents in producing a certain product and if that goes out of hand. So that thought experiment was like all right so they will do everything and mine everything on earth including humans to reach that goal and then build spaceships to mine all the other planets to just reach over and over over that goal. So do we really want AI agents to being able to alter themselves towards what goal and what end?

That's a tough question. I think that's yeah that's that's very dangerous. I think if you box it into that self-improving what I mentioned, what I'm doing is really like learning to be more reliable in certain use cases, collaborate better with other agents in the network, but not redesigning your overall application or your architecture itself or you yourself like having that loop in. Right. So it's there's some coding agent included and the coding agent sees that this and that is not working and then it improves itself and tests it. I would be concerned. How do you set the guard rail for that?

Is the is the engineering capability ready to build something like that?

I mean we have coding agents right? So the question rather would be if it could refine itself while doing so. I think you would probably still need some part of that at least it interacts a bit with something outside and being able then to alter itself. I'm not seeing

It's like a

Sort of Yeah. Or some reinforcement learning other methods involved in actually learning things and moving from there.

So I think philosophically I'm I'm wondering this is the if you think about our goals right what is human goals intrinsically it's very complex and we have different opinions so I don't know how we can bake that universally would agree kind of goal into this agentic system

This is very hard and it's also I think it was like 2016 18 or something. MIT did actually an experiment I think like they collected millions on millions of different opinions on that and that was the project on autonomous driving and how people would decide in different situations.

So there were like little animations of like people crossing and the person who would do that experiment would have to decide right so I'm sitting alone in the car and an old person crosses it do I and there's an accident either I hit the person and kill the person or I kill myself how would you choose and then like homeless person pets you are with your kids in the car and things like different scenario areas and there was not one clear conclusion like yeah that's the way to go or that's not the way to go right so alone this around that the driving and how do you decide in that situation because if we are in that situation we don't have that much time to think right so it's it's just in that situation you do whatever you feel in that moment and what you can kind of compute put in your brain what will likely be the best outcome for probably everyone involved if you have more time to think. So well do you save yourself? Do you save someone else? Do you save your kids and so on? Yeah. And if you then go to what you said with agents wow then it gets more difficult.

Even if you have all the time in the world to think about you don't we don't know the answer. We cannot sort of just write that goal into the system and tell the system to follow this type of goal forever.

Yeah. And that's and that's actually to kind of loop back to what you ask. And I think that's also kind of a bit of a problem. And this is why I feel like it's good we have it orchestrated autonomy so that we have it in a kind of a certain area where it can make decisions but it's still bounded in what it can do, what's allowed to do and what's not allowed to do. I was I was just earlier today I was reading a funny post on LinkedIn about OpenClaw and about well I was happy that it reads my emails and does those things and then it canceled my workshop. So how much autonomy do you want and how unbounded do you want it? Right.

No. And I was just going to say that I was working with different LLMs as judges and they are already if they are just judging a single sequence without having a comparison like relative comparison to the task and having different samples and being able to see all right so and actually form not an opinion but kind of have a reference and see all right so this might be better because of that prompt and they have a better sense of deciding. But even then different models will have different biases and if you change the underlying judge the result still can differ a bit. So how do you decide right that's very hard to put the knot on that lever.

Yeah. So it sounds like we can the imperfect humans can only make the imperfect systems. We probably live with it. Let's say I want to build an agentic system, right? I want to ask you where should I start? I want to build a patent filing system using AI agents. Patent filing is a lot of work and it's legalese and it's hard to write, right? And it's expensive. You hire lawyers. So I can think of at least two agents. I'm going to use a patent researcher and a patent writer. The patent writer has to understand what how to write a patent. You understand the legalese. It's totally different writing. A patent researcher maybe someone will go out and research if this has been filed. It's an art prior art research something like that. What are your thoughts on how we built this agent system together?

Well, I would probably not only have two agents involved, but more to teams of agents just because it might make sense to break those things down. But in general what I mean we need to distinguish between different things. So in general if someone breaks down or wants to build an agent from a use case the easiest thing is first to think about the roles involved in that application. So what I usually recommend or do myself is and also when I'm consulting for companies tell me they want to kind of have parts of their existing applications or interactions with clients mapped to AI agents. I asked them what kind of skills and they want to do and from there you can decide all right so what kind of task are involved and how do you break it down and then it depends also on the sensitivity of the outcome. So do you need to verify it? Do you care about hallucinations?

How do you check this? Do you need feedback? How do you do the research? So for instance, if you want to go wide and also like look for competitors and what things are what others are doing. So you might want to go and say all right so maybe it's good to have a swarm of agents like little ones working searching the web bringing you all kinds of different information. Then you have another agent synthesizing that what comes back and gives you kind of a summary of a finding. So for me it depends what you're trying to build. If you just want to build something kind of simple I would still say all right so you still need some researchers. So I would have there's actually an API, SerpApi, where you can access the Google patent API. So you can actually very easy get that information and they the agent can track this. You could also then have an agent which can operate the patent office website and just look for the different yeah numbers according to like for software I think that is 40 or something.

Look in if there are any patents with your company name or in the direction you have and then bring that information. So you synthesize all of that from that analytics and then you might want to have depending on how good or established this should be you might want to have even a team of writers. So not just one writing agent but maybe one starts writing you have another refining it and look that it's proper checking facts citations and all of that. So there's depending on how good and nonh hallucination hallucinating your system should be or reliable. Yeah, it would get more complicated.

I should think of the task I actually wanted to do, right? And then break down the roles of agents according to the tasks

Probably not all the time. There's actually also and there was also wrote an article about that but there was also recently I think it was beginning of a year end of last year and I think it was DeepMind a paper about the science of agent systems or think something like that study and I've I've done the same or in the same direction that not always is it necessary to overly complicate and have that extreme topology. So sometimes if you are and this is actually also at the beginning of I think it's in chapter 2 of a book where you just can also break it down into just subtasks and still have probably different LLMs but I wouldn't call it as in a multi-agent system but to guide the language model and maybe properly have smaller models involved and working together in those things but that might get a better outcome for the task you're trying to achieve.

So it really comes down to some task or some domains benefit really from multi-agent systems but others might just benefit from two or three just sequentially put together different models in collaborating or even just one good model in having a specific guidance how to achieve a task bit by bit. So that really depends on your use case. So I wouldn't say that you always need to have a multi-agent system for all of that. And I think this is also kind of a common mistake in general or kind of a false perspective on agents or multi-agent systems like we kind of in past two 3 years about yeah I think yeah last yeah even maybe 3 years or more however there's there's that kind of understanding bigger models it's better and now we are trying to transfer this well more agents will fix it better like if you if you line more together and it might work better but if you don't design your system properly with validating what comes in and out of your agents how they communicate, how you wire them together, then you might have the yeah complete opposite effect.

Like I mean you come from that part of saying all right so can the model solve the task and maybe then you did go to another architecture or a bigger one because it's a better foundational model it can generalize more and now we are actually or should think more about can the system execute kind of a stable sequence of decisions and this is why I say think about what kind of task and roles are involved. Do you need to break them down? Do you need to verify? Do you need to get feedback? Can you design that maybe within one system? Do you need several agents involved? And then also a thing to consider is do you need the same model family? Do you need another provider to control the bias from the other? How do they collaborate with each other in that multi-agent system?

Having more employees doesn't mean that you can scale linearly, right? Because there's in communication in multi-layers that you're putting in like managers through different layers. All those are over could be overhead in a sense.

Yeah, exactly. There's that what is the book the mystical one month or something. It's about in software engineering adding more people to the project does not necessarily speed up the project because there's that overhead the communication and you need to find a way of working around if you have maybe a very good team already working together giving them more freedom to actually collaborate better together. This helps you solve a problem faster than just adding friction in adding five more people.

How do I know like if I have this use case in my mind, how do I know which solution I should go to like using a multi-step single agent, multiple agents or versus using just one very powerful API call?

That's a good question. This is also part of my research I'm doing and trying to find that out more reliably. But I would say in general definitely there is like you need to evaluate things right. So also what I also like to do or do from the very beginning when I'm designing any agent or multi-agent system is I try to have a evaluation as early as possible in there to see how the system works and also have as soon as possible subject matter experts in testing that system then you can also try to stress test it in terms of well if you have for instance customer service agent set up right so you need to think about well some people might have erratic behavior because they are mad because the system is not working they are 5 minutes away from their company presentation and they just can't log into the system and they are really mad and whatever angry and so can your system handle that?

Do you need really another agent to jump in or is there one very good system, one model just being able to reason about and be maybe even specialized in being more socially adapt to certain situations. So these are kind of the questions I would ask and look at. So test this early. Test the interactions. And I mean the other thing you need to ask yourself is or be aware of every additional layer you add, every additional call you have consumes token, right? So if you have another verifier, it costs. If you have another touch, it costs.

What I often times do and what I really like is there's there's a framework it's open source and it's called RULER which is just a very easy way of implementing relative judge into the system and it gives it looks at you can adjust how many examples it looks at. So the generator generates let's say three examples and then that judge can look at it and by a specific criteria judge if the output is good or not good enough and this gives you an floating score and also reason why it's good or not so good. And this already helps in having like a little control point. And of course, it costs more, but as far as my experience goes, it's worth it. And it's also fairly fast in terms of latency. And it gets especially if it's it has to be a specific wording or phrasing of text, which matters, you get better results.

Assuming we're going to build a multi-agent system, right? How do I teach the multi-agent system how to divide the work, judge the outcomes and how do we how do we, you know, orchestrate the agent systems, you know, enable a smooth communication and handoff process.

So yeah, this comes also down to what kind of topology you're using. Of course, because what you also need to be aware of is if you have different agents working together, you don't want to, how can I put this? So, not every agent involved in that system needs to have every information about what the other agent is doing. So you need to decide about does every agent in the system involved needs to have the local state and memory of that specific agent and how do you orchestrate then how they communicate. So that might be if you have more of a blackboard architecture, it benefits of having a more centralized memory and sharing ideas and then it they are collaborating and this has also their own or its own difficulties in managing the memory growth of that, right? And kind of the token cro of that and in and king this along. Or if you have that supervisor architecture like the one I mentioned earlier where you have that kind of tech lead then does the supervisor need to have all the context of the others to decide where to route it next?

Do the other sub agents need to know about the others? So it comes down to what kind of architecture you're building and also if you and one thing what you would want to consider as well is maybe it might even make sense if it needs to be very reliable. It makes sense to teach them actually to collaborate better and this can be done then with reinforcement learning. So there are methods that you can teach them to yeah to be better in as a team

Is that fine-tuning or training like when we hear people say oh we need to train our agentic system

Well I don't know what they mean in that sense but for me can train or what I did is for instance to train it I don't know if you know Exa AI which is search API right and I trained agents to better search for the domain. I want it to be able to search and use lesser tokens for that and be more reasonable in doing that search. So that you can do with reinforcement learning and you have you teach it with different examples with different scenarios. You record the trajectory how they get from A to C in fulfilling a full task. And then you can teach a better behavior for that.

Is the end result of the training something like a markdown or skill file to give better instructions because the traditional when you say training I'm thinking about you're updating some weights in this in this model

And this is exactly what I mean. Same update weights. Yes. You really train.

You really train and what is the what is the model that's being trained? Is it the LLMs underneath each one of the agents? Okay, I see. Yes. So, it's not like you're training the agent loop or the multi-steps itself. It's actually training the LLMs using each one of the steps.

Yes. So, to based on a specific scenario, it learns the underlying model learns to better behave in that kind of scenario or situation. Yes. And you expose it to different scenarios, but it's still more than I wouldn't say fully isolated, but it doesn't adjust then the routing or to which agent it gives the task necessarily, but it's more reliable within its own ecosystem of fulfilling the task. I hope this makes sense. It's still a bit static. Like otherwise you look at more dynamic way of coordinating the setup of different agents working together and that would be then another thing to adjust.

How's this fine-tuning this training compare with vertical data training? For instance, training something to understand fintech to understand finance or healthcare that requires a different different context and different data. And by understanding that data, we update the weights of the for instance a small language model understand the domain better. And then when we use that small language model throughout an agentic system, it has better understanding of that specific domain at every step. What you're talking with this or how do these two compare?

A good question. Let me think how to make this easier or better to distinguish. I think the easiest way to distinguish what I mean by that is so you have those open-ended tasks, right? So it's not about knowledge or classifying like for instance if you would use supervised fine-tuning you have a very specific end goal and it for instance classify the sentiment of that sentence and then you can have it very easily narrowed down to what and you have that data and then it maps. But if you have open-ended task for instance talking with customers things like this then there's and this is the most important part there is no single correct answer and this is what where it gets difficult. So the model needs to learn to reason how to act or behave in certain situations and this can be done by reinforcement learning. So this is kind of you don't teach the understanding or knowledge of the data or classifying the data but behaving under certain conditions.

So the so the feedback is not whether something is correct answer or not rather you give rewards on choices of pass objective is uh

That's the reward function you're you're modeling right exactly. Yeah.

Cuz when we say fine-tuning, I always think we need an objective function otherwise like what are we targeting for talking about hallucination right this is like intrinsic problem for using LLMs. Do you have a solution for this while building AI?

I wish

One silver bullet to kill all the demons.

I wish but and I tend to repeat myself on this but the thing I always think about or say when I've got asked a question about hallucination is the problem is that we do this ourselves right so if we don't have the answer but want to make the others believe we do know the answer we come just come up with something and we hope the others won't find out if we are convincing enough they don't. It's just natural that most models do the same as we do. Yep. For whatever reason, right? I mean, there are some research that but I believe we won't be able to fully eliminate hallucinations. I think that's a really tough one to resolve.

I think what's the proper way is that you and this is why I also mentioned right. So if you have a verifier or a judge, do you really want to have the same model family or provider even? So it might be better to have some other checks and also like if it needs to be really validated, you need to want to look up the source. So if something is cited or referenced then you better have if it's important for needle or whatever other cases you want to have a model actually looking up that specific reference and cross check and then can correct the other model to be able to eliminate and of course I mean with rag there can be a lot of things be done to give it access and also with AP I to giving internet access in grounding it in more knowledge. But yeah, there's no silver bullet in eliminating hallucination at least not as far as I can think of.

Do you have this experience when you give very strict instructions sometimes the assumptions within that strict instructions can be wrong at certain times, right? Because the real world data varies. So sometimes your premise is wrong. When your premise is wrong, the LLM is forced to give some answer and then that's when more likely give they give the wrong answer, the hallucinated answer because the real data meet with like very constrained instructions, you know. Have you not?

I think there comes also a bit down to how much context and information you're giving your model. And the more you give it, the more confused it can get, which is totally natural because it comes down to the attention mechanism and soft max and how the probability distribution is then mapped to it and how it can actually see what's important and what's not important without going into the technical details. But yeah, that this is an important part and I've got funny experiments. I won't name the names of those model providers, but I've done some funny experiments with them and I noticed that a certain family gets confused if a context window gets too long and then even if you correct it two chats later, it still invents things and concludes things you weren't talking about and also other another model was also a bit bad of keeping track around the right things and just took things in its own hands in doing stuff. So yeah, so even from we the models I'm talking about were all state-of-the-art really really good models and model providers but still they are still suffer

Only two choices of top models that you're going to you know use for building an AI agent system what are your top two choices?

Oh, don't ask me because it's really hard because I just recently did actually compare different models from Anthropic, OpenAI, I did use Qwen, I did use Grok and Sun for some task and I did this for judging different things and it was very interesting. I had different task and different difficulty of task in a data set. So I did some experiments and evaluation of something I'm developing right now and most of those agreed. So let me think which one was involved. I think it was the first judge was a model from Anthropic and then the most recent one from Grok and the most recent one from Qwen and I think I even had Gemini in the mix. And I have let me think I think it was 10 different task and data sets in that thing. And for the easier one they agreed mostly for the more difficulty ones where it was multistep there was more disagreement.

So it's it's really hard to answer. I would really decide based on the behavior of a model and also there it's not even a provider specifically it's even the model itself behaves sometimes differently. So it's a tough one to decide and actually did some research end of last year in for some roles comparing some of those SOTA models and the more expensive ones. I'm not kidding you, did overthink literally their tasks and spend compared to a smaller open-source model at some points at some cases four times as much tokens with not as good results.

Is it because they're you said they're overthinking? So they're consuming a lot of tokens.

They were like going over and over and over and I mean you can of course you can restrict the amount of thinking like usually depending what kind of API you're using I'm using mostly OpenRouter because you can access literally almost every model and provider and you can switch between open source and closed and you can ex adjust between medium and high and xhigh, I think it xhigh for extensive thinking and I would never recommend to use that extreme xhigh. I tested it and it just blows up in your face. It's just crazy.

That's a really back. You know, in the consumer world, people generally believe that things that are more expensive has better quality.

It's not the case. And like mentioned that I'm doing that research and I did like in how to wire different models with different tasks and skills together and I actually found that a lot of time the cheaper model got preferred because it was just doing fine for that task. So it's not always the most expensive one needs to be over the top notch needs to be all all the time.

Is there something that can choose models depending on the usage or the tasks at hand so the developers can focus on the tasks and building the design the architecture instead of thinking which model to use.

This is what I'm doing right now. I'm breaking that. So yeah, I'm I'm developing something and I will be open sourcing that hopefully helping some people choosing that easier because that's also yeah consuming a lot of compute in my brain. How would you do that?

Do you have a name yet? What do you call it?

I actually have a name.

Are you still debating? Because this is a teaser. This is a spoiler alert cuz it you haven't released it yet, right?

No, I haven't released it yet. And so and there's actually a deeper thinking about that behind. So and hopefully people don't get it wrong and read the information beyond that. So it's called AgensFlow and not agents with a t but coming from the Latin word and the Latin word means it's acting and driving and conducting and that means it's not the agent itself but it's the agency in motion. So hence the agents with so it's a e n a n e ns right yeah with just like agents without the t and then the flow so it coordinates the flow

You should share the your GitHub repository once it is ready so we can you know blast it in on my channel as well.

I mean my tests show it works and it actually uses lesser token over time because it learns how to co coordinate and choose cheaper models and better models for the task.

I have a question for nontechnical users who are using skill files with like ChatGPT suggestions on how people can you know optimize their skill files to get the best results without touching the steps.

I would say like for me that's that's also and I think I mentioned this in that direction in the book as well. For me it's it's kind of going in the direction in having that employer contract right so employee employer contract. So you define certain rules by how you expect your new team member should interact and produce an outcome and you do this also during the onboarding and all of that. So you frame the rules and if you think about the same for the model and usually if you have a rather unexperienced let's say intern you will give it more rules and more guidelines right and I would do the same with agents I would not assume that they know everything and yeah they seem very intelligent But if you don't tell them how to do stuff you want it or how to you want them to do this, they will not do it. They will not invent certain I don't know phrasing for your text or how you like your presentations or how do you I don't know want the images it generates to look like or not look like. So be as specific as possible and boil it down to something very concrete.

How do you iterate the skill files to improve the

Let me think how I usually approach this. So I think that there's a lot of intuition already coming in because I interacted so much with transformers and language models and you know kind of like I think it's the same if you have had a lot of interns learning from you know where they get stuck and what they are up to right so there's a lot of intuitions I don't have that much iterations but what I usually do is if I design things for where I'm not a domain expert which of course I can't be for every system I design for companies what I do is I ask them what their hard kind of requirements are what and what stoppers they have and then from there I can design how the system needs to interact with the person. So this is kind of that guided and then of course in terms of iterations I then look at how it comes out and then I have to I let the subject matter experts iterate over it and rate it and give me feedback. I mean this is the only way to improve it

For everything we discussed today. What do you think if I just use a vibe coding tool to build what we talked about like the patent filing agentic system? I just I just forget about all the learnings just use a vibe coding tool. What do you think?

So there's not a no or a yes answer. I think there are some benefits of using coding agents and intelligent IDEs to help speed up the development for sure. But what I can tell you from my experience in trying those things out and I was long against that and I did look into that past couple of months more into this and what I can tell you and it doesn't matter what kind of sophisticated system without naming any names I tested different ones if you if the person operating the system doesn't know for instance if you go into someone just vibe code a multi-agent system and the problem what I see is if you don't know how to properly do this you might end up if you vibe code it with a system that works and it will pass test and it might look reasonable but then after the first months in production your CTO will be stomping in very angrily into your office because you burnt lots of tokens and whatnot because you didn't know that you need to ask your coding agent how to optimize certain things like memorization.

So don't repeat all the things, store different states in between, cache prefixes, all of that. Because if you don't tell the system how to design your system, it will just choose something. So if you don't know about it, you can't properly instruct the system. So you still need to be an educated operator. We are not there that you can't just say all right, I want an application like this and that. Of course, you will get an application. It will run, but will it be the best and the most effective and the most reliable and not burning down your credit card? Probably not.

Yeah, the burning down credit card is a is a the scary warning.

Yeah, I mean we give the system if you give if you say to an agent coding agent create me like what you said with that patent research if you say it yeah create me a multi-agent system for patent research and it might create like let's say two agents or three but it might not necessarily optimize for certain outcomes and the problem is that these systems s lack the experience in operating themselves because how do they know? They don't operate themselves and they don't see and learn from that user behavior or then you got ped a.m. In the morning because your system just shut down and puts out garbage, right?

The good things in this day and age still takes time to build. If you want a good result

I wouldn't say necessarily time. I would say it needs knowledge and acquiring that knowledge needs time and then know how to guide the systems in helping you building it. So I'm not against or saying it's wrong to use coding agents or vibe coding to speed up development. I think a lot of people do that to be competitive and productive. That's fine. But you can't outsource fully outsource the thinking. You still need to understand the system, how to design the system. This is something you can't outsource yet.

Well, I think this leads to a natural question of do you think AI agents will replace software engineers or anybody, right?

Yes. And I get this asked a lot and my go-to answer is usually that the let's say more mediocre or not so good people. Yes. Because those fell through the cracks because they would just do the maybe not so creative or not so challenging tasks where not so much thinking or thoughts or creativity was required. And now you can have those agents doing that kind of footwork. But if it comes to very specific things, you still need the creativity or the input of an engineer. How to design a system, how to write an article, how to write things. If it comes also to like authors and like journalists and things like this, I don't think that AI agents could replace a really good journalist. No, even if they can write good, but no

Not good enough. Right? So there's one type of fear which is the gap between how a junior of any profession g grow into grows into a more senior position because it takes time right they have the practice of learning a craft for instance junior engineers the challenge now on the job market is a lot of the company I talk with they want to hire senior engineers who already has the judgment what you're talking about the knowledge judgment they don't want to train train a junior engineer because AI agents can replace the junior engineer. However, they cannot replace the senior ones with the judgment with the taste with the knowledge. How do we address that gap?

Actually, I would go so far in saying parts of that needs to be addressed with education. I fairly recently at the beginning of this year I actually went back to one of my universities and talked to the headmaster where I got my bachelor's in computer science and we were talking about because he was like some engineers or students better ask why should they still learn programming because it can be done by machine right so why do we need to understand and program this And from what I can tell you from having even to code on paper during my exams, this forms a very specific type of brain and thought processes even if I hated it at that time like even doing algorithm algorithms like algorithm classes very difficult algorithms on paper you couldn't even compile it and see if it's going or not. So that is really cumbersome but it gives you a different way of problem solving and I think like the teaching and this was actually what my former headmaster said what they are looking into is in helping students get more that problem solving mindset and more being hands-on actually doing work.

And they already did this when I was studying there. There was every semester you had real contact with customers and you had a real project where you were working with fellow students like in little groups in really developing for a client software and applying things. You actually learned that semester and it was kind of ongoing and you had different like three or four different clients during during your studies. So you learn and study on the fly and you get already exposure to problem solving to the experience to judgment and also the critical thinking and I think this is the only way to prepare juniors to get that critical thinking. How they can approach or do approach projects differently and be already more senior if they come out of a university because they have more hands-on. They are already interacted with clients. They already worked in groups in solving problems together. Where they know how to collaborate, how to work in groups and solve a project and bring this to fruition and collaborate with coding agents for instance the right way. So they learn it on the fly.

Which university is that?

That one was in Switzerland.

That's the that's one of the best curriculums that I've heard in preparing students for the AI era like to just go do real projects versus a lot of the classes here I'm understanding teaching a lot of the series still you know on a lot of series so I love that

No this was really really good it was really hands-on of course we still had classes where like I that we had those exams and some some were just like you have to had to memorize things for the exam and yeah because it was closed book and it was just not realistic because in a real world you would have access to at least some notes and things like this but yeah the hands-on part was very important and we even had like I majored in data science science and we had even workshops where we had to analyze papers, disprove parts of a theory and run experiments and extend this. So this already also like helped in understanding and how to read papers, how to question what was contributed there and sharpen the mind.

I'd like to close our episode with this traditional question I that I ask almost every guest. If you could recommend a book to maybe one of those, you know, 20 year old 20 year olds who's, you know, just facing this disruption in the industry. Do you have any book recommendations?

It's important to think outside the box. And for instance, I don't know, you might have heard of Ada Lovelace. So I mean she imagined the computer beyond just calculating numbers and I found that a couple of years back when I read this I found this very interesting in not putting the limits on yourself or so like I've read Surely You're Joking, Mr. Feynman a couple of years or so back like in just enjoying the moment in finding things out and experiencing things like I think like if you if you are a junior entering the job market there might be the case that you put too much pressure on yourself instead of enjoying the process and this then if you overthink at least for me this was also for me at the beginning when I entered the field or in general also with learning if you I did put too much pressure on myself it had sometimes the opposite effect. So I think like if you enjoy the process the process of learning and knowing that you make mistakes and you enjoy making the mistakes and finding things out the way then the process is different and it's easier to do.

Yeah. And keep the imagination like a

Love it. I you know Nicole I really enjoyed this conversation. I also want to read one paragraph I found in your book that I really like. Can I just read out loud? Yeah.

Yes. Now I'm curious.

This is in chapter six. You're saying so what it reads is I'll move here so I can read it easily. So it says, "However you choose to design your system, remember that reasoning mistakes are recoverable, but ungoverned execution isn't. Once an agent can touch files, networks, APIs, or external services, every implicit assumption becomes a potential breach. As an architect of your agentic system, your responsibility is to ensure that your agents are safe to fail." I really love this paragraph. I felt like it speaks to the responsibility of for any developers, building agents. So, thanks for the book. It's really well written.

Thanks. Thank you.