We had a new customer ask us about how we’re leveraging AI at RoushTech, I’ve realized this is worth sharing more broadly (we’ll soon be doing all kinds of articles on internal process, how we do project management, our branching system, it’ll be fun!)
Prelude
This is going to be part of a multi-part series on modern AI (mostly centered around GenAI), we’ll start with how we use it, some predictions about how things are going to go.
Additionally, RoushTech always practices pragmatism, we evaluate platforms, we determine where they’re useful, we don’t get sucked into hype (but we do understand the usefulness of leveraging it for funding, investment, and general attention).
This isn’t our first rodeo with the latest tech fad (and just because it’s a fad doesn’t mean it doesn’t have practical uses). We’ll go more into this in another article about flavors-of-the-month. Microservices, Serverless, Blockchain, NoSQL, No/Low-Code… we’ve been correct on those.
Our Personal Experiences
- We’ve used LLMs to slap together some layout code, this had to be mostly rewritten when we attempted to polish what the LLM produced. Looked good at first, bought us some time, but TCO was higher than if we just did it right the first time
- It’s been really hit or miss, I personally had it recommend an entire way that you implement BigCommerce widgets inside of a theme using JSON files — this… isn’t how any of this works. 100% wasted time. Super confident and hallucinated many steps including corrections where it got things wrong, until the house of cards came tumbling down (I refer to this as the “dead man walking” AI scenario)
- Other times it’s been useful, catching out weird behaviors inside of Magento’s dense documentation? It’s come in clutch saving a significant amount of forum and documentation digging.
- I use it any time we need to parse/deserialize some kind of file, it’ll get 90% of the way there, and I know when it can fall on the final 10% (mostly just fumbling column names)
- Summarization – It can do pretty well (it’s a language model after all), but sometimes will hallucinate things like quotes, be wary about specific details, good for broad strokes if you weren’t going to pay much attention otherwise.
- Tone-policing – This is actually one of my favorite pieces, let it evaluate something I’ve written and determine if the tone is what I’m intending, suggest edits to adjust.
- It’ll do a decent job of marketing copy, though an A+ copywriter can make things a bit less cringe.
Ultimately, it gets used but we’ve learned it isn’t a silver-bullet to disengaging our brains.
AI as a Tool, Not a Replacement
With the above in mind, you can see how in some cases it can save us time, and in others it can waste significantly more. You need to know the problem space, you need to know how well the LLM will handle the problem, and understand what our goals are:
- If we’re banging out quick examples, drafts, prototypes, or blocks of code, it works decent enough, for larger generations expect to throw it away at some point.
- If we’re doing anything specific to language itself, it’s decent, but keep it on a short leash, don’t let it endlessly re-draft stuff, don’t let it blindly rewrite.
- It may not be exact with documentation, but can be quicker than a Google search a lot of the time, cross-reference early to prevent dead-man-walking scenarios.
- Context control is important for performance.
- Engineers need to understand the tool well enough to disengage if it’s wasting their time.
This is a stance that both allows us to leverage the technology, but doesn’t leave us brain-drained once the VC money dries up and subsidization of the tooling goes away that we can operate without it.
Where We’ve Seen Error
- Being a warm body in a seat
- We’ve had people effectively just copy/paste LLMs responses to questions we ask — at that point you’re being more of a tool than the LLM is. If I want to talk to an LLM I can save a salary and get rid of you.
- Getting caught in forcing the LLM to do the work
- Sometimes you need to know when to cut your losses and just do the job yourself. If you spend more time fighting the model than solving the problem, you’ve already lost productivity.
- Trusting it blindly
- LLMs can be confidently wrong. You need to know enough to cross-check– and not just with another LLM.
- High downtime cycle time
- Waiting for responses on some LLMs leads to a start/stop load/unload cycle time, which means you can be burning lots of time not actually working (and task switching that much is rough)
- Chasing Hype
- I’ll go over this in our flavor of the month update later — but chasing hype only helps one person in the business: the person bringing in investors. Put on the act, get that money, but don’t get high on your own supply — be realistic about what you’re actually doing.
The Lesson In All of This
Be careful about promises from the same people selling snake oil every cycle, there’s no surprise that a lot of people hyping up AI were hyping up Cryptocurrencies a few years ago, we saw how many businesses were putting blockchain into everything, we see how many of those died (vast majority of them). No large corporate was immune from saying they were going to put blockchain in their products, you cannot disengage your brain and follow the leader and be correct.
These tools have practical uses, get familiar with them, learn to disengage before hitting your productivity loss cliff.
Some Fun Terms We’ve used for AI
Dead man walking scenario
Where an LLM has convinced you you’re on the right path, but it’s a dead end, but one that it’ll gladly lead you on for a long time if you don’t know to basically tell it “this doesn’t exist”
Knoll’s Law of Media Accuracy
Knoll’s Law of Media Accuracy states: “everything you read in the newspapers is absolutely true, except for the rare story of which you happen to have firsthand knowledge”
We find this true in AI in two ways: one, anything you’re less familiar with you think AI is great at, and anything you are an expert at you think it fails at.
And two: if you think you’re an expert in it and still think AI isn’t flaky, you aren’t actually as good at your field as you think you are (much like if you think the media is accurate in your field).
Honestly — we should coin a term for this for the AI-era, maybe someday…
