Like most of us who have been in the industry for a while, I’m fascinated with the concept of vibe coding. If you read this blog, you know that I have used the majority of tools out there in the market. Initially, I only attempted to use them to create prototypes. As a product manager (PM), that’s all I really care about. I don’t write code, I don’t manage production sites, I don’t own the release process, etc. etc. etc.
However, the quality of the tools continues to improve. This leads me to the obvious question: Can you actually produce high quality software using AI?
I think the answer is yes.
As I work with AI, I’ve come to the conclusion that the trick is to treat AI as a very bright but extremely inexperienced and naive colleague. (See what I did there? Trick or treat?) With apologies to college students everywhere, I tend to think, “Would an intern understand this instruction?” And if the answer is no, I dumb it down until I think the intern would understand. This technique has worked well for me in previous interactions with AI, so my assumption is that it will work well in having AI build software from scratch.
Is that true? Let’s find out.
First, when building any software, I always start with the end in mind. Carefully lay out the goals of the project. How do we know if we’re successful? If I am working with an experienced engineering team, this discussion can be pretty brief. I could say something like, “I want the site to be enterprise grade and support 1,000 concurrent users” and the team would probably build a site that scales well, is secure, has a good login system, monitoring, and so on. There are tons of things that I assume a good SaaS site has and my engineering team knows all that.
However, AI doesn’t know all that. So, what to do?
Well, I know what all those requirements are. I just need to write them down. We call that a Product Requirements Document (PRD). Yes, the same document that we used to write all the time. Some product organizations don’t use PRDs any more and I haven’t written one in five years, but I’ve written hundreds of them in my lifetime. Start there.
A good PRD includes business goals but it also includes nonfunctional requirements like the site must be secure, the site must be tested, only authorized users can use the site. Usually, if I am working with a very professional team I skip all those things. They already know that, why repeat them over and over? But again, AI isn’t experienced, and doesn’t know things. So, I listed all those things.
After writing a detailed PRD, I used Claude Code. As I mentioned in the Claude Code Micro Review, it allows me to give it rules that it compiles in the Claude.md file. So I created a new directory, put the PRD in there and told Claude Code to write up an architecture document based on the PRD. Surprise, I didn’t tell it to write code, I didn’t tell it to make a site, I just said, “What would the architecture be?”. Here’s the actual prompt:
I would like an architectural framework for building my SaaS business that will connect authors to readers. This SaaS application will run on AWS and should make use of appropriate AWS PaaS services. It will be deployed via Terraform and use GitHub Actions for CI/CD. Propose a full tech stack for our MVP. Reference the PRD in the docs and put your recommended architecture in a file called architecture.md in the docs folder. Keep it simple, this is just the MVP and we can add advanced features later.
And this is what I got back:
The document I got back was decent, but I wasn’t happy about all the choices it made. No worries, I reviewed the document, made some changes and now I had a rough sketch of how the site would be architected.
OK, so then I typed /init into the Claude CLI and Claude read the directory. It saw the PRD, it saw the architecture and it said, “OK, you’re in the early stages of development for a SaaS site, let me help you set that up.” It proposed a phased implementation plan that again I reviewed and made changes to as shown here
The pattern is this: set a high-level goal, ask for a plan to achieve this goal, review the plan, repeat. Just like a very junior employee. Minimize assumptions, give them key learnings, provide relevant documentation.
Based on this I asked for a detailed plan to implement Phase 1. This plan was pretty good also. Note that I’d been working for several hours with Claude but zero code had been written. That’s the point. We are talking about goals, we are setting up our environment, doing architecture and design work, but we are not writing code yet.
The other thing that I’ve done in my testing is to download best practice docs and ask Claude to compare our state to best practice. I then take that output and use it to stack rank epics to fix those issues. Here’s the plan Claude developed from the AWS Terraform best practice document:
Notice that the work items are assigned priority. So, I implemented the two P0s and put the P1 and P2 items into the backlog. Just like I would do normally in my PM role. By treating AI like a smart but inexperienced team member, I’m able to operate just like a PM with a larger team. Also, hilariously, Claude makes the same mistake a junior would make and assigns time estimates to each item. Those estimates are wildly off just like any estimate from a junior. That’s OK, I’ve worked with plenty of juniors in my career. Just a little coaching and we’ll be fine.
I have continued with this process and now I have a very nice setup based on Terraform running on AWS. It may not be the best designed application ever, but it has all of the components I would expect. It has a real CI/CD pipeline, it has automated regression testing, it has a proper backend and frontend, a rational scale plan, and security checks in the pipeline. All the things I would ask a junior team to look at before we went to production.
So, I think that this process works.
Is this vibe coding?
No idea.
No comments:
Post a Comment