The Power of "I Don't Know"

I recently read one of those influencer posts on LinkedIn. You know, one of those posts that say "Don't do this.... Do this instead..." This one was about acing the interview and it said, "Never say 'I don't know' in an interview."

This statement really surprised me and I started to wonder how many people are actually taking this statement to heart.

Before I get into this, let me tell you a story.

Many moons ago, I worked for Microsoft in the professional services group (Microsoft Consulting Services or MCS). In those days, Microsoft was growing massively and MCS more than most orgs. This meant we were constantly hiring and everyone senior in MCS was doing technical interviews all the time. As a member of the Exchange team, it was my job to interview candidates who claimed to know Exchange. I had a standard list of questions I would ask. There was one and only one hard and fast rule: the interview could not conclude until the candidate said "I don't know" at least once.

 
There was a very practical reason for this. The goal was to accurately assess the candidate’s technical knowledge. Until you reach the EDGE of that knowledge, you don't really know how much they know. So, you keep asking more and more detailed questions until you get beyond their ability. Then you stop because you found their limit. It also had the added benefit of finding out what happened when they didn’t know something. Did they just make up an answer? Not a good habit for a consultant.

 
When this strategy was explained to me, I was perfectly fine. Made logical sense. However, when interviewing candidates, many of them were distressed. Typical conversation:

Candidate: "Gosh, I guess I failed the interview!"
Me: "Oh, why do you say that? You actually did pretty well."
Candidate: "Oh, well I didn't know all the answers."
Me: "Oh, don't worry. We would have kept going until you didn't know the answer."
Candidate: "WHAT! Why would you do that?"
Me: "So I know what you DON'T know, which is way more important than what you do know."
Candidate: "I have never failed an interview question like this in my life."
Me: "Welcome to Microsoft?"

I really didn't know what to say. I had literally never done a technical interview in my life until I worked at Microsoft (I was only 27 when I joined). It seemed to make sense to me but apparently wasn't common. I had one candidate drop out of the process because of this. TBH, if that kind of thing rattled them, they would not have done well at MCS anyway (at least in those days) but I did feel bad that the candidate had such a bad experience.

 
Here’s the thing though: The people we hired didn’t all perform well. We exhaustively tested their knowledge. We were extremely diligent in this process. However, we routinely brought in candidates who knew everything about Exchange and they utterly failed as a consultant.

 
It turns out that there was almost zero correlation between their technical knowledge at time of hire and their ability to work in MCS. In the end, someone had the correct skillset, abilities, and drive to get the job done. Those things made them successful. So, the ability to learn was vital. The current knowledge of the current product was just a side note and not really a strong indicator of success. After all, the software will change with the next version. Technical product knowledge has a very short shelf life. So, we changed the interviewing process from “Tell me how X feature works” to “Tell me about a time when you didn’t know the answer and how you were able to figure it out.”

 
Perhaps this early experience broke me, but for my entire career I have been completely uninterested in what a candidate knows. This seems completely irrelevant to their ability to do a given job. I mean, we can all use Google, right? Now if a candidate makes a crazy claim on their resume like "I invented SMTP" or something, we are going to talk about SMTP quite a bit just to be sure that's true (it's usually not). However, the vast majority of the time we are going to talk about your ABILITIES and the way you solve problems. You see, knowledge is transient. I know all kinds of things. Most of them are totally useless to my current job. Wanna know how to do IPV4 subnetting? I'm your guy. Or you could Google it. Actually, just go ahead and Google it. We'll all save time.

 
On the other hand, if you are really struggling with interpreting contradictory customer feedback from your target persona interview session, you should probably call me. You are not going to Google that one. You see, I have experience and a skill set in this area. This is my core job function and I've been doing it for quite a while. You don't get that from Google.

Thus, when you interview someone, the amount of core knowledge in their head is interesting, but not the most important thing. I mostly ask factual questions just to make sure they have actually worked on things they claimed to have worked on. Once that's settled, I don't do that any more.

What the heck does that have to do with "I don't know"?

Well, because if you are being interviewed and the interviewer starts asking factual questions, they are just trying to figure out where you are on the experience spectrum. You know or you don't know. Assuming you're not claiming experience you don't have (don’t do that, we will find out), the fact you do or don't know a specific fact shouldn't really matter. When that happens, you just say, "Sorry, I don't know that detail, I didn't work on that part of the system. I can look it up for you if you like. What I actually worked on was...."

And off you go. You show that you have related knowledge, even though you don't know the exact fact they're asking for, and show willingness to look things up. If you want to go the extra mile, go ahead and Google it after the interview and mention it in your follow-up email. "Hey, thanks for the interview. By the way, I Googled that detail we were talking about and the answer is X. Your question made me wonder about the answer so I went ahead and looked it up."

Boom. Done. You showed that you were paying attention, you cared enough to follow up, and you were able to come up with the correct answer quickly. I’d hire someone who could do that.

The bottom line: If the interviewer turns you down because you didn’t know fact X, they actually did you a favor. Yeah, you don’t want to work there. Organizations that hire primarily on the ability to memorize facts are not going to push you to learn and grow. They will focus on the “right” answer and you’ll be expected to repeat the same facts each day. Personally, I don’t want to work in an organization like that, sorry.

 
My ideal job is one where I can solve complex problems and do things that require me to stretch my brain. This is why I love product management. There is no “RIGHT” answer in PM land and I love that. That’s just what I like, but I think most people like to do challenging work.

If you find yourself bored and doing repetitive work, you may want to seek out a role where the interview is about your abilities, not about the facts you know.

Similarly, if you are hiring, think about what the candidate is going to be doing for your organization three or five years from now. Does it matter what facts they know today? Probably not. Focus on things that are evergreen like their abilities, their inquisitiveness, and their grit. Not on short-term things like factual information.

Layers of Abstraction

As I mentioned in a previous blog post, my team and I have been working on GenAI lately.  You may have seen Outshift’s announcement about the AI Assistant in Panoptica:

https://docs.panoptica.app/docs/cisco-ai-assistant-for-panoptica

Building the AI Assistant for Panoptica was an interesting learning experience.  It helped us understand what the limits of current LLM-based agents are and allowed us to start work on what the next generation will be.

This has also caused us to start thinking about how applications will change over time due to LLMs.  There is a very interesting debate raging at the moment that ranges from “nobody will write code in five years” to “everyone will write code in five years” and pretty much every other variation.  It’s exciting because we are right on the cusp of a major paradigm shift and nobody really knows where this is going.  I’ve been around long enough that I’ve seen these paradigm shifts happen a few times in my career so I do have some expectations about how this will go.

The first thing that I think we should remember is that this trend towards higher and higher layers of abstraction is not new at all.  If we look back into the history of computer science, we see a steady progression into higher and higher levels of abstraction.

For extremely old computers like the Altair above, you had to manually enter the boot code via switches before it would start (yes, they had ROMs but you had to program them yourself).  Then boot loaders were introduced.  Eventually, shops would write their own operating systems.  Older mainframe shops didn’t buy their OS, they built it custom.  Later, operating systems became a standard commercial product.  Then came the GUI revolution.  Later, we moved to distributed computing and the Web.  Then came virtualization.  Containerization, microservices, PaaS and cloud followed.

At each stage of this evolution, higher order services were offered.  When I was in college, I learned to program and I had to talk to storage devices directly.  For example, you could write to disk.. Today, with something called S3, I can create an object, persist information to that object and then refer back to that object whenever I want from wherever I want.  Yes, there are still disks sitting somewhere in a data center, but I don’t really need to know how S3 works and I don’t really care.

Does that mean that I don’t write software any more?  Well, as a Product Manager, I don’t write software at all, but you know what I mean.  We still write software.  We just don’t write low-level stuff any more and we don’t use low-level constructs.  Of course, this evolution wasn’t even or orderly.  Does anybody remember 4GL?  Fourth Generation Languages were supposed to take over and make things like C irrelevant.  That didn’t happen.  I used one called R:Base for a while but it was just too limited to write serious applications.

Just like the rest of the stack, programming languages have evolved but the low-level stuff is mostly still there.  In school, I learned machine language, assembler, Ada and C.  Other than Ada, all those programming languages are still around even though most software teams would never touch assembler any more.  However, if you work on an operating system project like ESXi from VMware, you are still writing pretty low-level code.  You’re probably writing in C, but you’re still down at the hardware.  So, the lower levels don’t go away.

What happens is really more about leverage.  Instead of EVERYONE writing an operating system, we buy that operating system from someone who only does that.  Then we use it without worrying about the down-in-the-weeds details.  It is similar with cloud.  Yes, I probably COULD write my own object store or set up a Ceph server, but I choose not to most of the time because renting S3 buckets from Amazon is super convenient.  There is still a team in Amazon deploying servers, disks and all that stuff but relatively few people in the industry are dealing with that detail because most of us just rent it by the hour from AWS, Azure, GCP or another cloud provider.

What does this say about GenAI and the future of programming?

Well, it’s pretty clear that writing source code is going to get cheaper.  We know that LLM’s can already do this and are getting better every day.  Tools like CoPilot mean that a developer can produce way more code more quickly, and thus the code costs less to produce in terms of human hours invested.

Andy Jassy recently claimed that AWS saved 4,500 developer years of work due to GenAI based programming tools.  That is an insane number.  Even if the number is only half that, it means that their cost per line of code is WAY WAY less than it would have been if that work had been done manually.  You can read his quote here:

https://x.com/ajassy/status/1826608791741493281?lang=en

However, it’s extremely unlikely that traditional algorithmic programming is going anywhere anytime soon.

What is much more likely is “hybrid” systems where traditional code sits alongside GenAI-based agents and they work in concert to solve problems.  This means that even if you are not writing your code by hand, you are still writing and maintaining software.  All the labor and effort that goes into running a SaaS-based application is still there, just hidden behind a new layer of abstraction.

Thus, GenAI is likely to change HOW we do things but probably won’t change WHAT we are doing that much.  We will still have security issues, we will still need observability, we will still need to diagnose and fix bugs just like we do today.  The tool chain will change and we will probably be more efficient, but we’re still building and running software.  

So, the question for all of us is to figure out how to build and optimize this hybrid world we will likely wind up in.  How do we automate away the toil?  How do we move faster?  How do we operate more efficiently?  Not surprisingly, these are the same questions I was asking ten years ago when I first started working on cloud and SaaS-based products.