After a particularly tiring vibe coding session with Claude, I shared some of the resulting grumpiness in a few posts to the Write the Docs slack. I’ll confess. I’m probably not AI’s (as the term is currently bandied about) biggest fan. But I’m not a hater. I’m just, well, disappointed with it. It’s just not living up to its purported potential (a.k.a. hype, these days).
I’d been writing code (i.e. prompting Claude to write code for me, a.k.a. vibe coding) and Claude’s code was getting buggier and buggier. I’ve seen that happen before after it’s written a lot of code. It acts like it’s tired, but I think it’s due to having too many things to keep track of in the conversation, so it loses its place (I’m not an AI psychologist…yet).
In any case, I was beginning to wonder if it would have been faster to just start typing my own code (it wouldn’t), but I wanted to see how it played out. Eventually, after Claude had gotten stuck, again, I was troubleshooting in parallel and suggested a fix. Lo and behold, Claude agreed (as always). With that experience still fresh in my head, I went to the Write the Docs slack to see how others were faring in their AI journeys.
Thinking of past bubbles
In one post, I compared current AI hype to the hype I recall when PCs (as in IBM PC) came out in the early 80s. They promised the moon and in microscopically fine print, mentioned that “some assembly was required.” Sometimes some C, as well. (If you know, you know.)
In the 80s, it’s not that PCs weren’t amazing pieces of technology that could fit on your desk and still have room for your phone and desk blotter. Remember, this was a time when computer hardware had to have its own air-conditioned office. It’s that they lacked the “killer app” (the application that solves a high-value problem for a large audience), until Lotus 1-2-3 & Multiplan, two of the first spreadsheet apps), came out and ran on the PC.
Those killer apps transformed the PC from a geeky novelty to an absolute necessity. They enabled regular people to see the value that these machines could provide. Fast-forward to today, that’s how AI seems to be positioned: A novel (to most non-AI researchers and developers) technology waiting for its “killer app.” Just like the PC could do anything, but nobody cared until it did something useful. It’s hard to tell what AI’s killer app will be until we see it. If I knew what it was, I wouldn’t be here writing another blog post; I’d be working on AI’s killer app!
“95% of AI projects fail!”
YouTube’s AI must have read my mind, because it suggested a video titled, MIT Shows 95% of AI Projects Fail — Artificial Intelligence Might Be Stupid. The speaker took a rather pragmatic view of AI and expressed a lack of surprise at the 95% failure rate. Hardly flattering towards AI, but it seemed honest.
Rather than take the word of a YouTube video title, I looked up the paper, and it attributes the failures to the LLM-AI’s inability to learn. To me, as a technical writer and researcher, I’d attribute the 95% failure rate to a lack of appropriate documentation of the tools. When a tool’s documentation is missing or inadequate and the mandate is to “Fix [the problem] with AI,” you’ll use AI whether it’s the best option or not. How would you know?
I’d imagine that the projects covered in the study failed because they were probably not a good fit for an AI solution. If you expect an AI tool to learn as it goes, but there’s no documentation to tell you how or if that’s possible, failure is the most likely outcome. In my experience, customers don’t get it 95% wrong without some serious documentation deficiencies.
I’d suggest that AI was simply the wrong tool for 95% of the jobs in the study. That also mirrors my personal experience of throwing AI at a variety of challenges. Occasionally using AI tools makes the task easier, sometimes not, sometimes, it seems to depend on someone’s mood. There are times when I’m not sure if it’s my mood or the AI’s mood that needs an adjustment. I haven’t been keeping score, but it might be a good idea to start (something to ponder…).
On a positive note, this mismatch seems like a great tech-writing opportunity: to identify the jobs for which AI is an excellent tool as well as those for which there are better options. Ironically, when asked, the popular LLMs will tell you what they think they are the not good at, but I haven’t asked them to provide alternatives. However you get it, that information would provide a better developer, and ideally, user, experience and tech-writing could help save the day! Win!-Win!
Let’s talk solutions
The presenter in the video, had a very “solutions-oriented” perspective and described AI as “just another tech stack” to apply to a problem. MIT’s study, however, indicates that AI isn’t the best fit for every problem.
There are many reasons (that I won’t get into here, today) why that might be. That AI is not a solution to every problem, however, is not a problem with the technology; it’s a property of the technology. Unfortunately, it’s something that isn’t talked about (or is simply drowned out by the hype), however vital that information would be for EVERYONE.
A commenter to my post replied that talking about AI’s limitations would only reinforce the voices of the naysayers. It might, indeed. But it would also help those with problems that the tool is actually suited for identify this while helping the other 95% look elsewhere. No longer would that 95% keep trying to pound the square peg of AI into the round hole of their problem.
So, the commenter could be correct. Talking about AI’s limitations could help the “haters.” But I don’t see how talking about AI tool limitations is a bad thing; especially, if it helps people match AI’s abilities to the problems it can actually help with.