Agents Don’t Think— That’s OK

Recently, we have seen quite a bit of discussion online about computers becoming intelligent.  In fact, some people have claimed that the holy grail of AI—the ability of a computer to truly think and reason—is now possible.  This is referred to as “Artificial General Intelligence” (AGI).  Sam Altman (CEO of OpenAI) for example said:

We are now confident we know how to build AGI as we have traditionally understood it.

https://www.forbes.com/sites/johnkoetsier/2025/01/06/openai-ceo-sam-altman-we-know-how-to-build-agi/

However, others are quite doubtful.  Some have said AGI is simply not possible.  Apple recently released a paper pointing out that current LLM-based solutions don’t actually think at all:

https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf

For those of us working in the field, that’s not much of a surprise.  An LLM is simply using a very large statistical model (hence the term Large Language Model) to predict what the next word of their response should be.  Thus, we are seeing an amazing display of math, not thinking.

But, that’s OK.  There is nothing wrong with really amazing math.

Let’s talk about why LLMs are so amazing, and more specifically about why LLM-based agents are so transformational.

They’re amazing because they deal well with ambiguity.  They have enough context to largely figure out what we mean even if we ourselves are not being precise.

For example, let’s say I gave an instruction to a traditionally-coded microservice.  I wanted to find all entries in a database that contained the word “California.”  However, I don’t know if the database uses “California” or “CA” or perhaps even the older “Calif”.  What to do?  Well, what we normally do is list out all the possible values.  So I would say something like:

SELECT * FROM your_table_name WHERE state = 'CA' OR state = 'California' OR state = 'Calif';

Notice that I need to be specific and I need to handle all cases.  You could also write some regex or use some other techniques, but in the end, it’s up to the programmer to deal with the ambiguity. 

But what if I forget one?  What if other people refer to California as Cali?  Then my query would be wrong.

However, for an LLM, this isn’t an issue.  For example, you can use Perplexity to answer this question even if you use “Calif” instead of the more common CA or California in your query:

It’s a trivial example, but notice that even though nobody says “Calif” any more, it understood me and correctly answered.  I could have said CA (although CA also means Canada) or any other variation.  Of course, it only knows what’s in the database.  Notice the error for the A’s who used to play in Oakland but currently play in Sacramento.  LLMs are not perfect; they simply reflect the data they’ve been given.

You may not be impressed by this—after all Google has been using similar techniques for many years to make search work.  However, doing this yourself for an internal application used to be really, really hard.  Google had the size to do this, but your four-person feature team?  Probably not.

With LLM-based agents, you get the capability basically for free.  That’s amazing. 

Well, it’s not free.  Nothing in software is free.

While being able to handle ambiguity is amazing, it comes with a cost: uncertainty.

Because the model is guessing at the correct answer, the model can be wrong (as we saw with the previous example).  It could also go down a completely incorrect path.  This is so common, there’s a standard term for this: hallucination.  Hallucinations are very common for LLM-based software, and those of us building agents spend a huge proportion of our time building gates and evaluators to try and avoid these tangents.

However, the gates we build to voide these things are not perfect.

My personal test for agent control is the “History of the Peloponnesian War” test.  Whenever I build an agent that’s supposed to do something really specific, I always check by asking it to discuss the history of the war.  Ideally, it should not do this.  It’s been instructed not to, but sometimes LLMs (like puppies and junior PMs) just can’t help themselves.  They get excited and go off track.

So, let’s use an example.  We can create an agent that is specifically instructed to only discuss insurance:

The above example is a custom GPT I instructed to ignore non-insurance topics.  Notice the correct “I can’t help you” response to the Peloponnesian War question.  All good, so far.  But will it remember those instructions all the time?  Will it persist and ONLY answer insurance questions?

Remember that LLMs are smart, but inexperienced.  They’re easy to fool as I demonstrate here:

Whoops.  I did get it to tell me why the war happened by asking a hypothetical.  A human agent would know that I’m just punking them and would probably refuse to answer or just laugh at the silly question.  If you are an insurance company, you probably don’t want your agents giving theoretical advice to soldiers in the Peloponnesian War.  

Once again, we see the issue around managing and coaching.  Just like a junior employee, you need to watch them a bit, give them coaching and help them gain experience in a safe way.