Good Enough

In my work, I deal fairly often with measures of system availability like the n-nines metric, and back in the day, my naive junior SWE brain used to wonder just how relevant these actually were. Like why do we need at least 99.9999% availability? What's the difference between that and 99.99%? Isn't just 99% good enough? Who cares if 1% of the requests fail on my system that doesn't even have a single user? - Well the answer to that is: it depends on what you are doing.

Another thing that comes into my head is: What happens if you have 100% availability? (Hypothetically of course). Would we be living in utopia by then? In my experience of using cloud services, they never really have faulted me unless I'm doing something a little out of the ordinary. Isn't this good enough? It can let big businesses like social media, banks, and card payment networks operate just fine without major issue. What more could you want?

Another more timely example is quantum computing. Currently you often hear about the problem with 'error rates' - the higher the error rate you have means the more qubits you need to do any reasonable work. If we were to reduce the error rates, or make enough qubits, could we get a quantum computer that's "good enough" to perform quantum computations on?

I recently thought about an old project of mine that I had to do for work, and this concept of "good enough" resurfaced itself, so I thought I'd tell you a little about it in the hopes that you've been brought here because you have a similar affliction, or perhaps you were doing what I was doing at the time - trying to find a way to earn a promotion to Senior Engineer. This blog post is going to stay high-level and not be too technical because what I really want to convey to you is a real, quantifiable example of what getting to "good enough" looks like when considering product depth for a competing solution within an engineering org.

Let me set the scene for you...

Imagine you're on a weekly meeting with your manager, and you are informed that the company's internal administration panel needs some maintenance. The vendor has told us that they are moving to a new major version, and it is a mandatory upgrade. In order to continue using the software after the cut-off, you will have to migrate the implementation to the new framework that they've developed, and for our codebase, this is very extensive. So much so that it would be a far better experience to build it from scratch in the new framework.

A bit more about this software. It's an internal administrative panel so you provide the schemas and the endpoints to access the data, and they provide the interface, hosting and identity access, etc. Your manager also tells you that the company is paying $5k/month for this service (for context, that's almost the minimum wage at the time. So that means we could be paying for a whole new employee if we weren't paying for this software!).

I hope that you think this is a bit ridiculous, because I think so too, and it's easy to think that this is utterly preposterous given what we can do with AI agents nowadays. All this admin panel does is serve data from an internal portal that no one can publicly access, and to add some fuel to this flaming pile of garbage, the portal is notoriously slow, partly fault of the vendor and partly from the way we had to architect our system to conform to their SDK. Many people from sales and customer support would poke fun at it and complain, so you could tell that this is a decent pain for not just the business, but for the people on the ground who have to deal with this shit daily, or worse, could not work without it. I know for certain that I don't want to be using something so slow that I'd have time to complain about it in Slack.

Fortunately, at the time, AI agents were slowly starting to emerge as a worthwhile product for software engineering. I was lucky enough to be working at a company that had the resources and ethic to be on the frontier of AI tech, and I got access to Cursor Compose (not the model, the product). For those that may not have been there at the time, Compose is like an AI agent that can only read files and write files. It couldn't run commands or call tools, so all it could do was code and also talk to you about stuff.

Knowing all this, there was still a strategic decision that had to be made regardless of tooling. Do we migrate our current software? Or do we build a new solution from scratch? How can I sell either approach to the head of engineering or the CTO?

Both options have their own risks, and we were even debating whether we would need to shuffle human resources around to get this done. On one hand, the experience of the software that's already working has a momentum attached to it when you consider the allure of familiarity with something you've been using for a while - the certainty of its functions and its flaws means that you know what it absolutely can do, what its limitations are, and where it falls short of your needs. On the other hand, if we decide to step onto the path not yet chartered by many organisations just yet, we could bring about a means to fix all of the problems the end users were having by creating a brand new system that we control end-to-end, and architect it to our needs. There were many other arguments like this, and I spent tireless hours tribulating over the objective details, often trying to weigh them quantitatively against each other only to find that I couldn't confidently back this approach if I presented it to senior leadership, not even considering if it was possible to deliver it coherently. So I decided to take a different approach to thinking about it, and one way that I think would be palatable to even non-technical stakeholders is to answer the question: "What would life be like if we go down each path?". I found this to be so easy, that I can answer it in two dot points:

• If we keep our existing software, the current gripes would linger, people would be suspicious of the efficacy of the software, and most importantly, there'd be no pride in what we've accomplished. That's to say we would simply be running on the spot (for about 2-3 months) and scrambling to make sure everything is working rather than moving forward as a business and engineering team.
• If we build something brand new, it would mean a chance to challenge the status quo, to uphold what we demand most, to breathe a breath of fresh air without the shortcomings of everything thusfar, and have the pride in getting what we want on our terms. As an engineer, it would be a delight to engage in such a system (so long as it's architected well).

I hope those points made it clear which route I decided to take. It was one hell of a ride, so let me tell you about it...

Basically what I was embarking on was a "startup in a startup". Sometimes you read about this in software engineering or business books, and sometimes you see evidence of such in real life. In this case, I was about to start building competing software to our existing admin panel, and if it is going to work, I needed to somehow convince people to switch. With a small amount of thought, you could easily argue that this is actually fairly trivial. Because of the existential risk, senior leaders will be more likely to be in accord with this change, and if worst case were to show itself, I could rely on them to make the changeover. Even regardless of that, if I were to tell you (the end user) that there's a better and faster piece of company-authorised software to do your work for you, I'm certain that you would not really second-guess switching over.

So in summary, all I needed to do was build a new internal admin panel that:

1. Is faster than the existing one
2. Does everything that people want it to do

Only one problem: There are A LOT of things that this old piece of software does. As you can probably imagine over the years, small changes were made to it so that it would eventually cover a large repertoire of product requirements, turning it into an exceptionally valuable product, but also a very complicated one too. As with most corporate software projects, there are a handful of niche and oddly specific product requirements that turn into unconventional solutions in the code. Often these are 'sacrifices that have to be made' by engineering which favour business objectives over good coding patterns and software architecture. In my experience, these complications can subjugate even those experienced engineers who think they have a panacea, and write a TAD or some other document to proclaim its worthiness, only to be shut down by a single sentence from another engineer with niche domain knowledge who refers to one of these edge cases. To be a bit more prescriptive, an example of such edge case in the admin software was displaying related data tables in a standardised way that can be supported irrespective of SQL or noSQL DBs with minimal changes required to data schemas (and I'm sure that you can conceive of countless ways to approach this specific problem).

Now here's where I can finally begin to introduce the principle I've been wanting to showcase. Consider if this were a customer-facing application. I'm almost certain that neglecting a problem as big as the one I mentioned above will come to bite you (or the customer) down the line, it's a huge risk that any senior engineer would certainly call-out during an architecture review. However, given that this is an internal project used by about 50 people at most, there is some implied leniency that the end-users can withstand a little flakiness because they can accept the underlying cause that shapes this solution. So even if this issue is deferred, or implemented in a way that develops an anti-pattern, no one will really care too much. You could say that it's good enough if it just works, and is a fast, responsive web app that can get me my data. Neither the end-users nor the engineers care too much about what's behind the curtain.

With that said, I proceeded to create a MVP which can prove that my basic architecture can stand up on its own feet, and also is extensible enough to quickly and reliably cover the scope set out for the project.

Before I explain the next chapter of this project, I think it would be helpful for you, the reader, to visualise the scope of this project with some sort of imagery. Imagine a room with countless small drawers that cover the walls completely. The room consists of corridors that form a maze, and each drawer is labelled with a small note on what needs to go inside, and you have a store-room just outside the main door to sort through and organise into the drawers. This is how I felt embarking on this project. There would be no point modelling all of the drawers, noting their location and item on a notebook because it is practically uncountable with no clear rules or patterns to generalise a solution. So really, the only option I had was to just get started and then adapt, which is a process I quite enjoy. The way I see it, the easiest way to find a pattern in the work is to immerse yourself in it. There's nothing wrong about being the cog in the machine for a while, so long as you have the opportunity to tinker with it and make it better. Being the cog will give you a fantastic sense of how the (local) system works, and it's what I think is the basic principle to learning anything.

There are many people in the organisation that use this software to do their job, and thankfully there were a lot of engineers who work closely with these people, and they know a lot about how they use it, but that only covers a small subset of the boxes I needed to tick. After interviewing these engineers, I then interviewed some of the end-users to see how they do their thing. This includes people like account managers, senior leaders, customer support, finance, and product managers. Essentially, in the end, I had interviewed one person from almost every department in the business, focusing more effort on the power-users (in this case, customer support, and sales). I can tell you for certain that, at least at the start, it was a very awkward process!

Imagine someone coming up to you and saying that they want to see you work for about half an hour, and they are just sitting behind you with a notepad and a camera. Creepy, right? Although I exaggerate here, it was still an awkward process. People who don't do software engineering really don't understand why engineers would want to know the exact buttons you press, which screens you go to, and what you punch in, why, and what format it needs to be in. They truly cannot fathom why you would need such depth of detail, and I'm certain they had been quite befuddled with the questions I asked because their faces all showed it. These people use these interfaces so often that it becomes subliminal. The precise actions and minutia are no longer processed by them (to no fault of their own) because their brain has already turned it into a reflex. I'm sure you've also heard of a similar concept known as banner blindness, and it can be very jarring to have someone get you to explain something that you've never explained before, even though you are the person who is widely regarded as the expert. Consequently as you interview these end-users, you will find yourself listening to a conversation where 10% of the information consists of important specifics and intent that will drive your design, and 90% consists of botched explanations and superfluous details.

Was this process worthwhile? Without hesitation, definitely. Even though I feel like (cumulatively up to this point) it covered only about 70% of the scope, I'm certain that it would have saved me from having to naively and regrettably introduce friction with end-users by trying to gather their product requirements and intent through a hands-off approach. Consider for a moment how it would feel if you were the end user, and you had not been consulted about a change in the way you do your work, and if you had a problem, you would not know who to go to until the problem became critical. At least if you had been consulted prior to the change, you would feel like you've been listened to, and at minimum have the hope that the person working on the problem is trying to balance your requirements with those of everyone else. With the consultation, there would be far less back-and-forth of you having to go and report another problem when it shows up rather than if you were to have explain the process in detail up-front.

So if you ever have to do something similar and want to interview end-users:

• Expect to not get the full picture, but rather enough hints to get you started, find similarities and specifics in their demonstration, and figure out which features are more commonly used so you can tackle them first, then also store the more niche ones for later.
• Do not expect to capture every single requirement from these interviews. The product manager usually assumes this role so that you as the developer has certainty that what you are building is the right thing. In a greenfields project like this, you will have to play the role of a product manager to work out whether or not what the end-user says is truly a product requirement, and it's okay not to get all of them. Generally if you capture enough, the rest become either self-explanatory, or will bubble-up as a casual slack message.
• Unless you struggle to maintain attention, do not bother recording the sessions. Personally I never even captured them. Simply immersing yourself in their world is enough to get a qualitative sense of what the end-user needs to do with it.

From here, it becomes fairly straightforward, and I chose to work in layers:

1. Get the app good enough to deploy both locally and on infra
2. Get the app good enough to visit via a web address
3. Get the app good enough to view data on
4. Get the app good enough to mutate data on
5. Get the app good enough to complete a common flow from customer support
6. Get the app good enough to use daily for customer support
7. Etc...

Each one of these stages raises a new set of questions and assumptions to validate, and I found that following this process eventually led me to a flow state. It's a bit grueling at first because you are working at such a low level and don't have much to show people, but once you get to step 5, it really starts to pick up the pace and lead you down the path rather than you having to pave it yourself. The end-users I talked to would often be in awe with how fast the app is, and how easy it is to navigate, then they ask: "Okay, but can it do this?". If you ever want to feel what 'agile' truly feels like, this is it. In a couple weeks, I saw this project go from being a gimmick, to something that they pull open for about 10% of cases, then 20%, then you see them have a tab for the old system, and one for the new, and then some of them come up to you and say that they've hardly had to open the old system because the new one is so good, "oh and by the way, can it do this?"!

By the time they were using it for pretty much everything, I was still not totally proud of what I've made. There are so many different bandaids that I haven't ripped off, and tiny little papercuts that you get when navigating. It was imperfect, and I had to hold myself back from going too deep into these trivialities because there was clear evidence that the solution was "good enough" for what people were using it for, and they could withstand the rudimentary design and gripes that I have with its incompleteness, not because they had to use it, but genuinely because they felt that it was leaps and bounds better than the experience they got before.

It's a weird feeling. It's like placing a bet on a soccer team, watching them score a loss in the first half, and then making a comeback, but only by a series of lucky penalties. Your team won, but not because they evidently exhibited any clear skill. Watching this new system evolve to surpass the old system with such level of intimacy has got to be a very rare experience for those in my industry, and I hope this blog post helps you to navigate your own path through one of these projects.

With AI agents at the level they are now, the sky is literally the limit, and I hope you see, as I do, that software engineering has become more about imagining what the boundaries could be rather than the monotony of reaching them through individual effort.