As part of my work on AGNCY, I’ve been working with literally dozens of organizations working on Agentic AI. At this point, it’s not really a question of if agentic will enter your organization but a question of when and how. Last week, both Google and Glean announced their enterprise class agent platforms.
Regardless of your platform choice, you are going to have the ability to build agents relatively quickly. This means that eventually you will have hundreds if not thousands of them running in your organization. This is not hype, this is just a fact.
The real question then becomes, how do you ensure that these agents are doing what they are supposed to do, and how do we know if they are actually providing value to your organization? Yes, some questions are truly evergreen. This is exactly the question that every single technology innovation has to answer. The buzzier the technology, the more important this question becomes.
Years ago, I was heavily focused on cloud for enterprise before that was really a thing. Whenever I got into a meeting with a CIO or other IT executive, it was super common for them to tell me all about their “cloud strategy.” I would patiently listen and then ask, “What are your business goals for this initiative?” Many times, they could not tell me, which was concerning. The most common answer was, “to save IT costs” which was even more concerning because I knew for a fact that cloud wasn’t cheaper for most customers. I wound up writing an entire book on this topic (Why We Fail), to help customers figure out what they wanted to do and how to manage the implementation of cloud from an IT business perspective.
I meet with customers frequently about GenAI, the Internet of Agents, and related topics. Almost all of them have an “AI strategy.” So, I ask them, “What are your business goals for this initiative?”
Aaaaaand, they don’t know.
Sigh. Here we go again.
I’ll probably wind up writing another book. In the meantime, here’s the compressed version of this discussion.
Just like anything you do in your organization, any investment in GenAI, agents, or any other related technology must support a business goal. Technology for technology’s sake is completely pointless. You’re just lighting your money on fire. Start with a specific problem. Focus on the outcome. Measure the result. Repeat. This is the way.
For most of my customers, they have started out with GenAI-based chatbots. This makes sense because LLMs are uniquely suited to building chatbots. We have had chatbots for some time and it’s become pretty common for them to be deployed as customer-facing agents. Of course, because this is very common, it is also adding very little competitive advantage. If everyone is doing it, you’re not distancing yourself from the competition, you’re just treading water. Stop treading water.
To truly gain business advantage from AI, I strongly suggest that you pick out classes of problems that are currently difficult to solve with traditional software. Pick some sample projects, try the technology out. Use that experience to guide your organization. At the moment, AI has some specific characteristics that make it good at some problems and not others.
AI does not really reason. It attempts to find an answer based on training data. LLMs literally predict word by word the correct answer to the prompt they are given. This makes them good at answering abstract questions and allows them to interact with users in natural language.
AI is good at summaries. You can take a large unstructured dataset like a white paper or an instruction manual and ask the LLM to summarize. The results are often quite good.
AI is not linear. What AI is NOT good at is producing the same result over and over. Given the same prompt, the result will vary over time. This is good for things like chatbots but terrible for things like banking transactions. Focus on problems that don’t have a “correct” answer.
All of the lessons we learned about SaaS and cloud work for us here. The AI state of the art is moving VERY VERY fast. This means that formally evaluating tools and having a single company standard just doesn’t work. Anything you choose today will be completely obsolete in six months. So don’t try. Get small teams focused on solving tractable problems and iterate. You will need to move quickly if you want the result to mean anything.
In some ways, an agent is similar to a microservice. It needs to have a discrete business outcome and it needs to be something small enough to be developed by a single scrum team. An agent should have a stated goal, a set of inputs, and a list of dependencies. It also MUST MUST MUST have evaluation criteria.
The evaluation criteria is the actual tricky bit. It’s easy to say, “The custom keep warm agent should send customized follow up emails to customers who haven’t logged in for two weeks” or something similar. That’s a business requirement. You can also pretty easily say that this agent needs access to say, SFDC, Gmail, and the ops database to determine the last login. All good so far. How do you know if the agent is doing a good job?
Well, you have to read the emails. Do they make any sense? Are they more likely to provoke a positive customer response than the boilerplate template you are using now?
And here is the really interesting point. Any AI agent you build is essentially a very junior employee you hire. They need to be managed. They need to be coached. You have to watch them. Within the Outshift team at Cisco, we say “agents are the most enthusiastic interns you have ever hired.” If you interact with LLMs regularly, you know what I mean. Always eager to help, not super experienced in how your business works.
So, how are you going to manage this junior employee? How many juniors can you manage successfully? How do you compile and provide feedback to them? Under what circumstances will you terminate their employment?
For people, we have process, policy, and training to handle all these things.
Do you have all that for AI?
You need it.
This is what goes into your AI business plan.
