Is Claude Smarter than your Intern?

As part of my investigation into how GenAI will change the way PMs work, I’ve done some experiments with LLMs and their ability to integrate with Jira.  Because Jira has a query-based interface and a native MCP-based integration with Claude, it’s pretty simple to connect Claude to your Jira environment:

Jira is one of the pre-built integrations.  You can see the full list here:

https://support.anthropic.com/en/articles/11176164-pre-built-integrations-using-remote-mcp

I was able to connect Claude to my Jira account pretty easily and after doing so was able to ask it questions about my Jira project.  As an example, I dumped a list of possible future blog posts into the project and then asked Claude to stack rank them based on a business outcome:

Notice that Claude made up specific factors that would “drive readership”. (It also ignored my mis-spelling of Label and correctly figured out what I meant.)  I could then re-prompt with different criteria, but I think that this exercise shows that Claude is capable of using pretty significantly detailed business criteria to make decisions.  So, that part is good and this is a win for the “quality” check mark that we discussed in my previous blog about agents.  Does Claude deliver answers at a high quality?  Yes.

However, we are still missing context.  The examples above were complete divorced from the real world.  Let’s say I had an actual application and I wanted to compare my possible epics against the code base.  Is that possible?

Well, in theory, yes.  Claude also has a GitHub connector.  I set that up and asked it to review a private repo. This was the result:

Foiled.  It turns out that even though Claude can access my private GDrive, it cannot access private GiHub repos.  That’s pretty disappointing.  I hope the Anthropic team will remedy that.

So, let’s try a public repo.  There are several issues already in the AGNTCY ACP repo—could Claude automatically write Jira stories to resolve them? Using the Jira integration for Claude, I was able to get Claude to talk to another repo that had issues in it already:

Aaand no.  When it failed to read the actual issues page, it just made some stuff up.  That’s actually worse than doing nothing.  Back to zero on the correct scale.  Anthropic folks, if you’re listening, don’t make stuff up please. 

Changing my approach, I was able to manually import GitHub content.  I went back to my original private repo.  I was able to import the code into Claude using the built-in “Add from GitHub” function.  Strangely, this worked fine even though Claude refused to read the repo via the chat interface.  As the next step, I asked it to compare the current code to a PRD that I wrote and received this result:

I used this flow to analyze the sample PRD and compare it to the actual running code of a sample app that I built with Lovable.  It did a decent job.

I then had it upload those items into Jira.  It correctly created Jira epics and assigned them based on criteria that I gave it.  

So, interesting.  Overall, I would say that Claude was capable of acting as a very junior PM on my team.  I was able to issue specific instructions around how to write Jira epics and how to prioritize, and it was able to investigate the current code base to compare requirements to the implementation.  The latter is interesting because most junior PMs can’t just look at a repo and understand the code well enough to do that analysis.  So, in some ways, this is superior to the work that a junior PM would do.