My instinct after reading this article is to pull back a bit and ask some larger questions. Why is it necessary for big tech companies to act this way? Why does bad code bother engineers so much? Are they actually misguided for feeling like bad code is a catastrophe, or is it really the fault of the broader economic sphere we all inhabit? Is it actually maturity to reconcile ourselves to drift powerlessly as faceless and titanic forces sculpt our reality? So many possible questions.
I’ll take a stab.
Because I’m being held accountable for the bad results of the bad code. Because I’m being held to task to fix the problems caused by the bad code on a schedule that I didn’t agree with. Because management is using someone else’s bad code to deny me a positive annual review.
“You’re a senior engineer - why did fixing this take so long?” Because of the garbage I had to wade through to even find the problem. Because I wasn’t yet working here when this was written and I have to take time to get my head around the insanity.
Yes, these are management problems. I’ve spent years managing managers and attempting to educate them on how bad code causes errors and delays. But for many reasons, it’s a Sysiphean task.
This is due to the way how incentives are aligned. Systems that are powering things for, say a decade at least but worked on Quarterly basis.
Why is this alive and well, then? Because it doesn't actually matter as long as money keeps rolling in. It is also possible that the losses caused by or efficiency not achieved do not show up in the accounts.
This really is the worst, and that's why I left my first job. Funnily enough, I just took that job back after a few years but I am now the lead and sole developer on it, I'm having the time of my life doing what I've always wanted to do back then, and seeing the product now flourish.
The bad code didn't really matter, it was the fact that I was not allowed to improve it and forced to build new features on top of crappy code that made me quit in the first place.
exactly!
If a senior engineer is responsible for the problem, but he wasn't given room to develop a proper solution in the first place, then he cannot be blamed for the outcome. And yet, this is exactly what happens.
If a system had been made well, and functions well, same senior engineer does not get the credit for not causing problems!
This is what happens when management/stakeholders don't understand what is going on, but have an expectation that is not well communicated to the engineers (and also inversely too, where engineers are not communicating the situation properly to management - or they refuse to hear it/ignore it).
I got into this career because I like mucking about with this kind of work. Programming is sort of like building thought sculptures. Ok, maintaining them too. I spend a lot of my adult life working with these systems, so of course I care about their quality.
It is unpleasant to work on a hack-filled creaking monstrosity of a codebase. My skills, such as they are, don't help as much as it feels they should, because I rely heavily on understanding a system based on how it sort of has to work given what I know about the constraints, and hacky systems constantly confound my expectations. That means I have to spend way too much time and effort learning the idiosyncrasies and unpredictable corner cases, and so have less time to make the changes I want to make. It's like a carpenter working on a house where nothing is straight, nothing is level. Or a cook given flavorless or half-rotten ingredients to use. You spend more time picking out the bad (or picking out the good and discarding the rest) than actually cooking.
Any my scope is limited. Some changes are just not feasible to make, because there are too many hidden and/or unnecessary dependencies buried in the code. If you change one thing, everything falls apart. If you patch it together well enough to more or less work, you're faced with a long tail of subtle bugs that you'll be battling for a long time to come. There's no conceptual integrity.
Furthermore, working with a team that accepts a never-ending accumulation of creeping technical debt is also unpleasant. You can't trust that a positive review means anything. If you try to do something right and take pride in your work, your teammates will be resentful that you're not closing your bugs fast enough. Who cares that it means less maintenance overhead in the future if nobody's planning on sticking around, or people are depending on things being constantly broken so that they can spend their days knocking off relatively easy whack-a-mole style bugs?
Bad code is actually fine. It happens. But I want to know why the code is bad, whether the reasons are defensible or not. If they aren't, I know that the codebase is going to be littered with similar crap, and that my job is going to suck. That's what bothers me so much. This is true even if I'm the author of the bad code. Why didn't the reviewer point it out? Is it like the article said, and there's nobody competent enough to spot glaring issues? That is also a component of a job that sucks.
From the perspective of the organization that pays you to do it, it is? At best there may be another mission that it genuinely cares about, usually there’s only a profit motive (which is also fine). If you want to create software as an end in itself or enjoy the craft without compromise, it usually can’t be in the context of a job or a business
Our job is to produce the best possible outcomes given the constraints we’re faced with, and inform leadership so that they’re aware of the tradeoffs when they make their decisions. Sometimes those decisions are going to be bad, and obviously it’s justified to be frustrated then. Other times they are correct, even when it means compromises on the engineering side. That’s when we have to just suck it up (or go elsewhere)
I still think there’s room for enjoying the work of creating software even under imperfect conditions. Striving for perfection is for hobbies, or the very rare circumstance when it’s justified by the goals of the organization
I'm now spending my weekend thinking about a problem where a really urgent feature I wrote doesn't work because the query behind it is too heavy for the database (the query isn't anything special, but it's against a view that clearly isn't properly indexed), and the DB people are complaining I'm overloading the DB. But it's going to be shown to the regulator for some important regulatory rule for which noncompliance is costing us tens of millions per year. Next Tuesday. Which they told me last Tuesday.
How am I supposed to write good code in that kind of environment?
Well, I don't know that we can separate external forces from why people might dislike bad code... being held to a standard that is unachievable because of someone else's mess and someone else's expectations is all too common in life, and the reality is that your livelihood might depend on meeting those expectations. If you like programming, you might also like maintaining or refactoring code; it goes with the territory, and it can be fun to see how something can be improved or challenge yourself to see how far you can improve it. But it tends to be less enjoyable when your ability to eat and house yourself rides on someone else's bad code.
> If you try to do something right and take pride in your work, your teammates will be resentful that you're not closing your bugs fast enough.
Right, but why would they resent that? Probably because management has expectations on the team, and they are concerned that you're sandbagging the metrics. It ends up being a management or cultural problem regardless.
> Bad code is actually fine. It happens. But I want to know why the code is bad, whether the reasons are defensible or not.
Agreed!
Not endorsing it, but, it just seems to me the way most performance management works at these companies (I came from a company that liked to think it was amazon with stack ranking) you have to do it or you will get shanked inevitably before you vest. This is just my personal experience. TLDR it’s how you make things “happen”
It’s very cynical. Thankfully these days I work on much smaller teams where everyone helps as much as they can because you’re constantly treading water, but there’s very little room for this kind of political maneuvering and bs.
Others have said some true things but the core is missed. It bothers engineers who care about mastery. It’s the same for architects who see design flaws in buildings that were done by those who don’t care about mastery. Or a filmmaker who watches a movie with lots of flaws. Caring about obtaining mastery means having the capability of seeing the flaws and resolving them.
It could be maturity to resign to powerless forces or it could be the inability to obtain that mastery or the willingness to let go of that mastery. I would think it’s more mature to fight the good fight, but at some point you just get old and tired I imagine. And it’s interesting to posit the author is nihilistic when this “uncontrollable forces” itself is a nihilistic take imho.
If you have only ever seen one pattern to solve a problem, trivial example of inheritance, and therefore do that to the best of your ability then you have achieved mastery to your ability. Once you see a different pattern, composition, you can then master that and master identifying when which is suitable.
Lack of mastery is just using inheritance despite seeing alternative patterns.
Naturally mastery also includes seeking alternative solutions but just because a codebase uses inferior patterns does not mean those that came before did not strive towards mastery, it's possible that they didn't know better at the time and now cannot get the time to revise the work.
There's always a juggling act in the real world.
Assume incompetence and not malice, and incompetence is not a state of being. A person without experience can be seen as incompetent but quickly become competent with training or experience, but the code they write still stems from incompetence.
Strive to see your previous self as incompetent (learn something new every day)
Bad, over-complicated code is code that requires you to form an unnecessarily detailed mental model of it in order to answer a particular question you may have about it.
People rarely read code in its entirety, like a novel. There is almost always a specific question they want to answer. It might be "how will it behave in this use case?", "how will this change affect its behaviour?" or "what change should I make it to achieve this new behaviour?". Alternatively, it might be something more high level, but still specific, like "how does this fit together?" (i.e. there's a desire to understand the overall organisational principles of the code, rather than a specific detail).
Bad code typically:
* Requires you to read and understand large volumes of what should be irrelevant code in order to answer your question, often across multiple codebases.
* Requires you to do difficult detective work in order to identify what code needs to be read and understood to answer the question with confidence.
* Only provides an answer to your question with caveats/assumptions about human behaviour, such as "well unless someone has done X somewhere, but I doubt anyone would do that and would have to read the entire codebase to be sure".
Of course, this doesn't yield some number as to how "complicated the code is" and it depends on the question you're asking. A codebase might be quite good on some questions and bad on others. It can be a very useful exercise to think about what questions people are likely to seek answers for from a given codebase.
When you think about things this way, you come to realise a lot of supposedly good practices actually pessimise code in this sense, often for the sake of "reusability" of things that will never be reused. Dependency injection containers are a bête noire of mine for this reason. There's nothing wrong with dependency injection itself (giving things their dependencies rather than having them create them), but DI containers tend to end up being dependency obfuscators, and the worst ones import a huge amount of quirky, often poorly-documented behaviour into your system. They are probably the single biggest cause of having to spend an entire afternoon trawling through code, often including that of the blasted container itself (and runtime config!), to answer what should be a very simple and quick question about a corporate codebase.
You're not wrong that there's something cynical or nihilistic about it. The core thesis is "do what the company wants, even if it's not what they should want". That idea may be unpalatable, but getting ground up in the corporate gears is worse.
My own thinking is that the big competitive advantage that big tech firms have over small ones is the power to mobilize very large numbers of developers onto a project.
Large projects that don't depend on a small core group of irreplaceable competent individuals are more repeatable for a business. So it makes sense to focus on making the repeatable processes more likely to succeed than simply hope that you happen to have the right team assembled for the job.
Assuming that any of your engineers are above or below average hurts the ability for the business to plan.
This question gets directly at the cause of the author's nihilism: the necessity is borne from the endless pursuit of positive quarterly growth, the "binding fiduciary duty to shareholders". Which is a lie, there is no such legally binding fiduciary duty. So the aforementioned necessity is also a lie. Companies could operate on a longer time horizon, let engineers write better code, make better products, maybe even consider societal good in their strategic planning, and still turn a healthy profit. But the cost of perhaps taking a few degrees off their YoY trend line is unacceptable to the insatiable greed of their controlling shareholders.
The writing comes across as “well this is the way it is and this is why it is.” That’s fine, but most of us kinda already know why it’s the way it is.
Maybe, tell us something about how to adjust the culture.
> Why is it necessary for big tech companies to act this way? Why does bad code bother engineers so much? Are they actually misguided for feeling like bad code is a catastrophe, or is it really the fault of the broader economic sphere we all inhabit? Is it actually maturity to reconcile ourselves to drift powerlessly as faceless and titanic forces sculpt our reality? So many possible questions.
These are all great questions. I have some thoughts, of course. But I'm not sure it's fair to describe OP as a burnt-out nihilist. The premise of the post is pretty reasonable actually.
My issue with the article is maybe more that the perspective and framing seem to want to lock out idealism from the conversation and encourage people to become ruthless, hard-nosed pragmatists in order to survive in this environment. Is that actually good, effective, or desirable? Again, lots of questions.
I feel like this article is the equivalent 16 paragraph stating you're likely to be correct only 10% of the time when you guess a random number from 1 to 10
I’m not going to find someone that was doing software 50 years ago. And if I do, their experience is completely unrelatable.
For instance, Github Actions being a meh product is called out in the article - that's a classic "check the box" feature that's good enough for a lot of people (let's not forget that Jenkins was no picnic before it) but is never gonna massively increase GH's bottom line.
Those sorts of projects are easy places for politics to fester since they are easy to ignore for the most influential-and usually strongest-parts of leadership.
On the other hand, if you're on a core, mission-critical team and other people's code is turning into your bad performance review, you need to figure out if the problem is (a) bad/toxic manager or (b) a failure to keep your management chain informed at what the root issues are and how you can improve it.
Nihilism is a defense mechanism when everything is moving against your world view.
STEM people of all walks of life join because of the challenge, the loveliness of an elegant solution, and the “art” you create that leaves your mark. Good engineers view their code as a makers mark. This code represents me as my art and I should do my best. Unfortunately (or fortunately) this is beaten out of you by senior engineer and the shell left is a nihilist and misanthrope.
Corporate programming strips you of your creativity, your autonomy, and your drive. It’s simultaneously strict, but too loose where it should matter. It’s spastic in its execution. There is often little rhyme or reason aside from “build fast make money”. No one appreciates your contributions. You show up to meetings where sales and PMs simultaneously wrestle to take the credit you rightfully deserve. You made it 10 years, here’s another low budget pizza party while the sales team gets an all inclusive in Ibiza. You wanted to make cool, elegant, things, and instead you’re just a factory worker that had to do 7 rounds of interview for the privilege of stacking premade widgets together until retirement.
The end result as I and many engineers at and over the one decade mark have realize. Nothing matters, no one respects you, and no one truly cares. You get paid and promoted whether or not your code is elegant, or safe. You get recognized by prostrating yourself in front of leadership. So why should I give a shit if my code works or not. It passes tests and I get to eat.
The industry is pathetic and we should stop calling ourselves “engineers”. Modern corporate programming is analogous to working on a widget assembly line in the limit. Most of us simply find our joy elsewhere and have learned “hearts and minds” as the rule of corporate life.
But I think it's a matter of motivation, Bob.
> The thing is, Bob, it's not that I'm lazy, it's that I just don't care. It's a problem of motivation, all right? Now if I work my ass off and Initech ships a few extra units, I don't see another dime, so where's the motivation? ... my only real motivation is not to be hassled. That, and the fear of losing my job. But you know, Bob, that will only make someone work just hard enough not to get fired.
The problem I observe is a fairly universal one: management doesn't care about good code, it cares about results.
It's generally hard for anyone without specific experience with a codebase to tell what you're doing with it. Management can't evaluate the value of maintenance work, so it doesn't value it at all.
People who ship sloppy code get promoted.
And management is probably right: code is a mean, not an end
It is us, as software engineers, to make that better code brings better results
Some benefits of better code can be:
- less infrastructure cost
- more speed / efficiency / whatever metrics available from the users
- easier to integrate new features
- less issues, less time spent in maintenance etc
Very political stuff
I try to do good code. My team is rarely cited in that list of "bad players". Management does notice.
Corollary: our roadmap is not messed up. Product managements notices.
Also, good code must mean robust infrastructure. We have very few incident. Almost all our changes are successful. Incident and change managers notice.
Anyway, my point is : if good code brings no benefits to the outside world .. how is it good, exactly ?
Absolutely!!!!
Our job is not to produce code, it's to produce a product. Code is just the way we happen to execute that goal.
I think this is a very telling statement, but perhaps not in the way you intended. I would agree that management only cares about results, but I would posit that maybe that's a good thing. If you don't have ground-truth knowledge of a problem, you must rely on either the word of someone who does, or metrics that can be used as a yardstick.
When all a manager has to go on is someone's word, it can be really hard for them to gauge the depth, severity, and impact of the problem being expressed to them— and without any metrics, they have no way of tracking progress on resolution. In a modern codebase, you could spend YEARS on improving maintainability and still not "finish". The key (that I've found, personally) in this situation is to give the manager some form of metric to describe the problem. If you can establish a number to measure what you're advocating for, and quantify the consequences of not doing it into actual business impact, I've talked managers into taking my suggestion more often than not.
how do you have a metric to measure a future issue that got prevented by having good maintenance?
Either you cannot actually imagine nor describe the problem that was prevented, or you could just make something up which cannot be disproved nor falsifiable. So if you asked for time/resources/budget to do maintenance, you cannot then give proof that this maintenance was useful!
The only way to get a metric is to have an incident or have issues crop up, and then in the retrospective, claim that certain maintenance work could've prevented it.
Occasionally you'll have a manager who manages in the day and codes at night, which is fine if not bad for his personal life. But HIS manager almost never does that. And his manager's manager? Forget it.
Management cares about what their management cares about. So this boils down to what the CEO cares about. The CEO cares about what the board cares about. The board cares about share prices going up.
I do believe that e.g. retaining engineers is something that helps the business. It's stupid that someone ramps up for 2 years and then goes to a different job just as they start becoming really effective. It costs companies a ton of money which they could instead just use to get people to stay. But I'm not on Google's board. It's only when Google board decides that fostering the right engineering culture is important enough for the business that something is going to change. And so far- they don't (s/Google/BigTech).
Re: Incentives- Obviously(?) the incentives are not right. So when you say "plenty of incentive" what it really means is incentive to ship sloppy code and get promoted. Or the incentive to go from Google to OpenAI and get a pay increase or whatnot.
Management doesn't know how good the code is. So they can only look at results.
Like you say, this easily ends up rewarding people writing quick and dirty solutions.
This problem is easy to describe and hard to solve. Yet some teams do it much better than others. One solution is that management also writes code.
If you set incentives that say that being sloppy and leaving landmines for the next group of people is the way to get promoted, guess what? the management is bad. Often because they are also looking at their own self interest, and expect to leave the consequences to whoever comes after them. This isn't new to big tech: You'll find this all described in books about corporate dysfunction written in the 90s.
It's all traditional principal agent problems, which just get worse and worse as you add layers of management, as the principal has agents upon agents underneath, all using the misaligned incentives. One either wants t avoid getting fired while doing the minimum, or sacrifice the health of what is around them for a good enough promotion packet/review. And since there's no reasonable way for individual objectives to align well with long term objectives, people leave landmines. When there's enough landmines everywhere, you are always better off in greenfield development. And at that point, doing any maintenance, or being stuck in a project that isn't getting fed a bunch of capital to grow it is career suicide. All about bad incentives, set by bad management.
Note that the minimum amount of time necessary for a change is ~zero. A few minutes in practice because CI checks need to run, but that's it.
But management depends on your manager; at scale it becomes likely there are bad apples in every management tree. Incentives may not align with what you want or need, with work From Home policies getting shrunk. Even money sometimes is a point of contention.
So it's the McNamara fallacy?
The thing is, good code is a form of a good result. You need to solve the underlying problems (which manifest as impact) but if you used code to get there, that code if well designed is extensible, reusable, then you pay low maintenance on it and that same code can be used to solve other problems (ideally).
It's a difficult judgement call to make though. If your org doesn't have the right technical leadership and the performance management structure doesn't reward it then you get what you see.
It's a lesson that's always learned far too late when it becomes slow and costly to deliver something new because you've amassed so much tech debt so it's often cheaper to start from scratch (which is saying something).
I see this all fwiw in big tech myself and with my peers at peer companies.
No, it is just standard operating procedure: deprecate a working system and write a new system from scratch, with 50% of features not supported. This side-steps the tech debt and gives everybody artifacts for promotion. It screws all users of the system but who cares about them!
I used to be an extremely motivated engineer. I cared about the code that I wrote, the other people on my team, making sure things were documented and understandable. I tried to write good code where I could, and detailed PRs and issue writeups where I couldn't.
Despite that, I was always paranoid I wasn't doing enough, because it always felt like there was someone else that was shipping more code than I was. Some of this was almost certainly social comparison bias and impostor syndrome-like feelings at work; but I also had a string of managers that pointed out all the work I was doing, and how I was helping the team as a whole.
Eventually, the company got acquired by exactly the sort of company this article is about, my manager got a new director from outside the company, and my manager had to go on extended medical leave after a cancer diagnosis, leaving the director with ~7 new reports. I started hearing about how the number of PRs I was opening weren't as numerous as some other people's, and the code didn't look "hard" enough to their glance. Never mind if the easy code was hard to come to, or if talking through it after the fact they agreed with my assessment, or if I had performed a detailed investigation and writeup, or if my peers left reviews or public plaudits about work I had done. Those weren't PRs, which is ultimately were what they wanted, since that was the metric they could easily see, and justify to their boss.
I did _try_ to do better by their metric, though I never had a definition of what "better" would actually be. Funnily enough, that person was fired a few months after I was.
Also kind of funny to me is that, if I weren't motivated and didn't care, none of this would've affected me all that much.
And also to "keep the balance of power tilted away from engineers and towards tech company leadership." The author touched on that and forgot about it. You don't want key projects depending on a group of engineers that might get hit by a bus or unionize or demonstrate against Israel or something. Network effects and moats and the occasional lobbying/collusion mean the quality of your product is less important.
The flip-side is: you need the money more than your employer needs you. Which puts you in a bad position to negotiate salary, makes it hard to stand-up against bad decisions, etc.
I think that it can (not sure if the author meant this) also mean that you have a buffer and are OK with switching jobs, but are also in a position where your employer wants you (because of something you can do). This puts you in a GREAT negotiation position.
in the short run (1-2 years) there is no repercussion for this, but eventually making changes will be extremely risky and complicated. The individuals that built the software will lord over everyone else with their arcane knowledge of this big pile of junk
Oftentimes the circumstances are "we don't know the requirements", not because of shitty management, but because the problem is inherently hard to define.
The business conditions that do heavily penalize bad architectural decisions, like physical structural engineering, can suck to work in compared to SWE.
It takes a decade or more before you're trustworthy enough to architect a building and there's a million layers of approvals. Then it takes years before groundbreaking, and years more as the building increases in size.
Your whole life might be dominated by a single large project like Hudson Yards, which has been floating around as an idea since 1956. The most recent proposal started in 2006, broke ground in 2012, and another 6+ years to finish. Then when companies were about to move their offices there, COVID-19 happened and the leases fell through.
I'd rather the system that gives average SWEs regular opportunities to lead large projects from scratch and make mistakes.
There is a phrase "million dollar problems". You do stuff at your startup that will take a million dollars to fix because it doesn't scale.
The point is that if your startup doesn't get to that scale then it doesn't matter. If you startup does reach that scale then you have plenty of money/people to spend a million dollars fixing it.
I don’t like all the fantasy about “just talk to the customers” - nah it is not just, it is super hard to get their time.
Nope, in the same vein of "lording" over others, they become the expert of knowledge of bullshit. The environments that allow such behavior have already engrained reward of such behavior.
People focus way to much on the superficial stuff like code cleanliness, formatting, organization, local structure of the code
Because that stuff is easy to talk about, kind of like bikeshedding.
Plus a lot of times code reviewers just want to comment something to show they aren't just rubber stamping it.
Whereas it takes a lot more brain power to think about logic, correctness, and "does the change actually make sense in the big picture"
Part of it too is that as a reviewer a lot of times you just don't have enough context to know if the change makes sense
Ideally by the time your at code review this is not a question. It sucks for everyone for it to come up.
I think people also avoid rejecting for this reason becasue of that
And its even harder to make sure everyone who reviews or tests that code has a similar level of understanding about the problem the system is trying to solve to review code or test for fitness for purpose, and challenge/validate the design choices made.
And its perhaps hardest of all to have an org-wide planning or roadmap process that can be tolerant of that well-informed peer reviewer or tester actually pushing back in a meaningful way and "delaying" work.
Thats not to say that this level of shared understanding in a team isn't possible or isn't worth pursuing: but it IS a hard thing to do and a relatively small number of engineering organizations pull it off consistently. Some view it as an unacceptable level of overhead and don't even try. But most, in my experience, hope that enough of the right things happen on enough of the right projects to keep the whole mess afloat.
AI makes these types of developers much more dangerous because they will accept anything the AI generates tha looks like it works, and they’re experienced at pushing nonsense through code reviews.
AI also provides more of a “productivity” boost to these types of developers because unlike everyone else they actually spend the majority of their time typing code.
I feel that is a very likely scenario.
Every company I've ever worked at (from ISPs to health insurance to finance) every organize was just copying the fad of something else.
At the time I felt like it was because that was "the best way" but it was more likely do to engineers not having the freedom to actually explore good solutions. The made up constraints imposed by organizations against their workers are rarely for the benefit of the company.
It's not a surprise to see this being the case, most companies on the planet are ran like centrally planned dictatorships with the results being obvious in retrospect.
So the mountain of syntactically correct functional slop is growing faster than ever before.
In the end it doesn't matter how normalized and pretty you design the database, someone will eventually show up and write a pipeline that dumps every row of it into JSON once an hour and ships it to some far away corner of the company. Someone will write a shitty script to deal with the fact that those rows don't represent a consistent point-in-time snapshot of your database. In the end it doesn't matter anyway, it'll all be rewritten or coerced through some migration into some ugly system in a few months anyway that it doesn't conform to and could never match.
The thing that matters is the process. When you decide you want to do it, do you have the process to mend the ether to do what you need it to do in two months? Do you have the processes in place to catch it when it's so catastrophic it's blowing up your balance sheet?
I think the issue is more that engineers face unreasonable pressure to deliver short term value and that there is no respect for the craft/engineering from many managers or even engineers.
I did that job, just after university, but that is not my comment. I bookmarked it though because that person said it so well.
You will write bad code, because what you already find there - and that one company is not alone! - is already so bad, there is no way to do a good job on top of literally millions of escalating hacks.
And don't think that you could clean this up - not even with ten years of time is that possible. You can only rewrite from scratch. Trying to rewrite even a tiny part is like picking up one spaghetti and always ending up with the whole bowl on your fork.
> A good senior engineer is for the most part able to not write bad code from day one
But what is good code is in context of the codebase. It takes time to get the context of a reasonably sized codebase and no one is doing that in a single day or single week, even the wizards.
I don't agree with everything the OP writes but I think they're correct in that many companies don't value institutional knowledge. To me that is lunacy. I'm not sure how any programmer could think a reasonably complex codebase could be learned quickly. The larger and more complex your codebase the more valuable institutional knowledge is. Adding new actors just accelerates complexity and redundancy. Pushing people to be quick only worsens this. The best thing to do with new employees is get them deep into the code. Have them write/update docs, do cleanup, or other tasks that force them to understand the code and how everything interacts
You say that until you are tasked with doing impossible - three lines, all perpendicular, five green, two anti-green, seven in ten or more dimensions, any color; while customer only uses purple lines.
Last guy that worked on it committed seppuku. Rest of team is in mental ward. Your only team member is guy that programmed his entire life in PHP, and doesn't know backend's language. Just teach him.
Documentation, is spread between Jira, wiki, Markdown, ftp server and some napkins.
CI stands for continuous Indians. You send code to India, where a team will assemble it. It may take anywhere between a few minutes or few hours. But it beats GitHub actions. Make sure to inspect artifacts, the Indian team has a habit to add some of their ""bug fixes"" covertly.
But you gotta finish it by Thursday. Good luck.
If someone asks you to do the impossible you have to say no. Better yet, you should figure out what they actually do want. They can't get the impossible, that's not on the table.
The worst thing an engineer can do is not learn how to say no.
I'll even say, if you don't know how to say no then you're not qualified to be a senior
How do you say no in that situation? Just quit?
As a senior I've been tasked with impossible tasks, with insane deadlines, in ""enterprise"" code bases. Sure, saying NO is an option, but being the NO guy is surefire way to getting fired. And nothing looks better on resume than repeated firings.
No.
It's easy to say if you have other options.
It's extremely hard to say if others depend on you saying yes.
If I asked you 6 months ago if you might ever consider something other than credit card payments, urged you to seriously consider this and you say no, you shouldn’t come to me now and say that bank transfers (bank transfers!) are absolutely indispensible.
Except this is the system working as designed. Leadership 1000% wants to do things as fast and as cheap as possible.
Being delivered faster or cheaper isn’t the goal. The goal is to look good while doing it. Telling your bosses ‘Yes sir!’ Is apparently a lot more palatable than saying ‘No can do’.
You have a very charitable view of the competency of the typical engineer at big tech nowadays. Ten years ago, sure. But with the advent of people purely studying for coding interviews that's changed.
Even if there was zero attrition, programmers with 40 years of experience would still be rare. The fresh newbie developers of 1985 were a small group by today's standards.
A $20 IKEA chair is not "bad furniture". It's just optimized for different constraints than a Herman Miller. Most consumers are totally happy with the $20 IKEA chair.
An underrated part of engineering skill is knowing what corners to cut. I think large tech companies have structures that impose this on engineers who view this as dereliction of duty.
I consulted for a large manufacturing firm building an application to track the logical design of a very complex product.
They modeled the parts as objects. No problem.
I was stunned to see the following pattern throughout the code base:
Class of the object
Instance #1 of the class
Instances 2,,n of the class
I tracked down the Mechanical Engineer (PhD) who designed the logical parts model. He desk was, in fact, 100 feet away from mine.
I asked him what he intended, regarding the model. He responded "Blueprint, casting mold, and manufactured parts." - which I understood immediately, having studied engineering myself.
After telling him about the misunderstanding of his model by the software team, I asked him what he was going to do about it. He responded "Nothing."
I went back to the software team to explain the misunderstanding and the solution (i.e. blueprint => metaclass, casting mold => class, and manufactured parts => instances). The uniform response was "It is too late to change it now."
The result is a broken model that was wrong for more than a decade and may still be deployed. The cost of the associated technical debt is a function of 50+ team members having to delineate instance #1 from instances 2,,n for over a decade.
N.B. Most of the software team has a BS (or higher) in computer science.
P.S. Years later, I won't go anywhere near the manufactured product.
But I rarely find this to be the case. The decision on whether or not to invest in static analysis tooling is usually made by people managers, not technical managers, and those people managers are loathe to pay short-term costs for non-functional gains when functional priorities have deadlines.
It's really as simple as, deadlines to ship trump all other considerations, including expertise of any kind other than how to ship when working in unmaintained codebases.
It's not that the goal is to write low quality code, it's that big businesses understand the sales cycle and how to maximize profits. If they over spend on employees, that cuts into their profits or causes the product to be too expensive. And if they spend the time to write quality code rather than developing features, they lose sales. Customers don't buy quality, they buy features at a price, and quality issues (like bugs) get thrown over the wall to downstream support staff.
As much as I dislike this, knowing how unstable it makes the overall software ecosystem, companies aren't wrong for making these decisions. The companies that choose differently don't become big businesses, they either stay small, get acquired, or go out of business.
One of my first strange and unpleasant realizations in transitioning from studying computer science to "working in the real world" came in a 1:1 meeting with my manager at my first job out of school. I was complaining about code quality both in the context of some of our existing codebases and some new code one of my peers (also a junior developer) had recently written. When the light bulb finally lit up in my naive little head, the question I asked my manager with a sense of horror and outrage was "... so you're saying they wrote bad code on purpose?". The painful thought was that I, too, would (or had already) found myself tasked with pushing code that I knew sucked, for reasons entirely unrelated to architecture or design or other purely "technical" constraints.
I used to fantasize about moving into a different software niche, maybe in safety critical systems, where correctness is more highly valued. But recently I'm coming to realize that the thing I crave (and miss from my school days) is the joy of the craft— something involving elegance and taste in a way that even the strictest standards of correctness doesn't necessitate.
I think for the most part, truly excellent code isn't something many businesses perceive themselves as needing (even if many practical benefits can flow from its virtues). And, probably, for many businesses, such indifference is right. So excellent code, where it exists, is probably more often "gotten away with", half-snuck in by stubborn engineers who are productive enough to burn time injecting some extra consideration and effort into their code, than it is commissioned by a business which understands that it wants good code.
I think about other professions. A cook cannot spend time making every dish perfect. A bricklayer isn't perfectly aligning every brick. Even in movie-making there's a shooting schedule. Things go wrong and the best filmmakers know how to keep the production moving.
I love the craft of programming, but I see a lot other craft-oriented programmers who want every line to be beautiful. If you want to write code poetry in your off-time, that's your business. But that's not the job.
At work we are bricklayers and cooks. We practice a craft, but also have time constraints. I try to do my best work while working at pace. Sometimes my code could be better, but it's not worth the time to fix. Ultimately the thing we make is the running software, not what's under the hood. The business people are sometimes right
That's too generalised. A fast food cook can't spend time to make things perfect. A tiny, fancy Japanese place will spend time to manually craft a perfect dish and you'll wait while watching the whole process.
I suspect that you can find something similar in every category you mentioned.
I think this is worth exploring. Not shoving the details of work in people's faces to assert its quality, but somehow creating some drama or interest in its quality. In a way compatible with the immediate practical needs.
This isn't an easy problem to solve. And the example of a boutique Japanese restaurant is a good one. In this case, the process is designed to make consuming the food, the immediate practical problem, more satisfying.
Perhaps the code equivalent, would be seeing changes in sequence, where each change is obviously well done from the user's and manager's perspective. A process more easily achieved by green field work. Which is also relevant to the restaurant example, where each dish is its own creation (within a well thought out process).
So maybe not every line of code needs to be even more beautiful than the last, but there's clearly a balance to be had. And yes, sometimes the business people are right. Sometimes they are wrong, however.
When I started programming I wanted everything I wrote to be museum-ready. I was slow as shit and waaay too precious about code. As I've matured I realize that's not a good way to work.
I think my lowest acceptable quality bar is still pretty high (and I'm fortunate to work somewhere that is valued). But as time has gone on I've tried to emphasize speed and knowing when something is "good enough"
I feel that it's an important skillset for the profession, but often craft-oriented engineers dismiss it at "business people not understanding"
As always this depends a bit on where you work and your projects
Depends how you define "one purpose". I feel like I could polish any code to perfection forever. But the threshold of bad is going to be very murky. Is it still bad after 5min? After 30? After an hour? After a day?
Wherever you think is the right effort/benefit threshold, it will turn out to be different in a few weeks. And you'll find people who think it's too fancy and people who think it's bad. Rarely is there objectively bad code. (Yeah, sometimes there is)
Very true. And experiencing this has absolutely been a useful lesson to me. I remember feeling pained over some code I wanted to write in a more robust and principled way in front of a big deadline, and after a bit of friendly push and pull with my manager, I agreed to console myself by throwing in some TODOs and FIXMEs in the comments instead of getting carried away overengineering or burning rubber just beautifying my style. I remember this ritual feeling painful at the time, though it helped me cross the finish line sooner. A few months later, it was clear that ~80% of that code would be good enough for the next year or so— by which time other parts of the application would have evolved as well. I even ended up glad I'd deferred a few of those changes just so the small refactors could be informed by things we'd learned in the meantime.
Sometimes it goes the other way, or course! Many times I've been glad for some extra care I put in early, or regretful about some I didn't. But you're right that that balance always seems to change in retrospect, one way or the other.
But even then, given time and number of people, bit rot and a slow downwards spiral feels inevitable. Which is why in those industries they will often do a rewrite every 5-10 years, especially in front-end. Often a redesign / rebranding (also every 10 odd years) will be used as an excuse to rebuild software entirely.
Experience CAN remove pitfalls, though developers even disagree about those sometimes.
Organization, chosen abstractions, naming etc are basically personal thinking and have differed on every team I've ever been on.
When it's been good is when it's been consistent and that's taken a strong personality the team trusted to have authority.
Especially with enterprise code you're hammering your thoughts into a shape roughly compatible with someone else's so its no wonder overtime with the constant revolving door of people that without careful shaping things can get nuts.
Bad code is one of those things that we can almost all agree on, often even the person writing it.
Alternatively: I don't know how to make a good movie, but I can recognise a really bad one, and you'll almost certainly agree with that opinion. You and I however will almost certainly not agree on what our favourite movie of all time is. The nuances and personal tastes become more important at the last few percentage points approaching 100% "like".
Just as we might agree on universally panned bad movies, but disagree on cult followings or one of us for a particular reason can't stand a popular movie.
That is to say we can all agree on extremes, but just because something isn't extremely bad doesn't make it good in everyone's eyes and that's where the contention is.
A big company is like a collection of small companies. Code quality varies depending on where you are in it.
Similarly, nothing leads me to believe small companies are any better. Some are excellent. Some are nightmare spaghetti.
The main cost I have seen at places which respect engineering is lower predictability. It is harder to budget and plan even if the end result in average is usually cheaper and always better.
Then this leads to people setting tight deadlines. It's a vicious cycle.
At the second, it wasn't hostility but more indifference -- yes, in theory they'd like higher quality code, but none of the systems to make this possible were set up. My team was all brand new to the company, except for two folks who'd been at the company for several years but in a completely different domain , with a manger from yet another domain. The "relative beginner" aspect he calls out was in full effect.
1) the focus is on shipping a new feature, often building on half-baked infrastructure and with a tight deadline. Corners have to be cut. 2) the usual “shipping features gets you promoted, maintenance work doesn’t”
I have not worked for a FAANG, so maybe things are different there, but I don't suspect so. People are people no matter where you put them.
Increasing compensation is not the solution. It can be a factor in a larger solution, but just increasing compensation increases employee entitlement which makes this problem worse, not better.
The best solution I have seen is risk/reward. Put people in charge of their assigned effort with real adult danger of liabilities. Likewise, award them for their successes. This is called ownership, and it works because it modifies people's behavior. The rewards and liabilities do not have to be tied to compensation. Actually, associating these rewards/liabilities to social credibility within the team/organization appears more effective because it reinforces the targeted behaviors.
I have seen this missing in all of my software career until my current employment. Conversely people in the military are pushed into this liability/reward scenario from the very beginning and its very effective. It has always been striking to see the difference in my dual career progression.
it is quite a bit different at FAANG. I've workded for small companies, huge companies that aren't software/FAANG, and now FAANG, and it's definitely better here.
The floor is very high for talent and just an overall ability to get stuff done. Google certainly doesn't have a monopoly on genius coders, i've met brilliant folks at all different size companies.
It is very good at making sure the caliber of the average engineer is quite high. Code quality is shockingly good across teams and codebases. I said good, not amazing, there are definitely differences in teams and I can cherry pick projects outside of google that had better code than some at google.
But the consistency of it being decent is very high.
I'm also dubious of your claim that compensation doesn't attract better talent. In my 25+ years of coding, it's a pretty damn strong correlation. The people who leave google to go to even higher paying places like the top hedge funds or Anthropic are not the most 'average' caliber talent, it's usualy the better folks.
Maybe higher compensation does work out for FAANG, but it does not at other larger employers, at least not for software developers. I am highly dubious about this at FAANG too though. My perspective is as a former 15 year JavaScript developer and a lot of what I saw coming out of FAANG in JavaScript land just wasn’t impressive.
I do see a lot of impressive things in the world of JavaScript but it’s almost always open source from small teams or single developers.
I noticed an issue with a new feature i couldn't fix in a satisfactory manner monday. I talked a lot, with the lead and the other senior early. First i started doing a shitty fix. Then i asked for a carefull review from the other senior, we discussed the issue and managed to find the origin of all the bad code. Then i asked for more time (well, i "told" more than asked tbh) and did a full refactor, correct this time (hopefully) (the deployment + migration script will run next monday).
Writing bad code happen to everyone, at every company, especially when you don't have a lot of experience and domain knowledge. The issues appear when no one catch this bad code, or when people don't have the time or the latitude to fix it before it corrupt all the surrounding code.
In my experience, while this line is often repeated, in practice it’s rarely really a “deliberate” tradeoff. Rather it’s mostly accidental.
My sense is that unless actively managed against, any org big enough to have a financial department and financial planning will work under assumption of fungibility.
Engineering and business value go hand-in-hand in a healthy tech/engineering business.
A business that was built on great/innovative engineering, became successful, and then got taken over by various impostors and social manipulators, who's primary goal is gaming various internal metrics for their own gain, is not a healthy business.
The point of a business is to generate revenue. The point of employees is to do work that helps generate revenue. As such, any decision needs to ensure it has a business case aligned with revenue generation.
Good engineering hygine has significant business value such as in speeding up delivery of new features as well as keeping certain customers happy, but in a lot of cases there is an inability to communicate from either direction (eg. PMs not giving Eng full visibility into business decisions, and Eng not being able to explain why certain engineering choices have business value). If you cannot communicate why this matters, you aren't going to get it prioritized.
Unsurprisingly, at big organizations, communication can take the backseat because communication is hard and at a large company, there is some amount of complacency because the product is good enough.
Edit: Unsurprisingly got downvoted.
The only reason you are employed is to make value (which generally is measured in revenue generated). You are not paid $200k-$400k TCs to write pretty or a e s t h e t i c code. You can make a case for why that matters, but if you choose to bury your head in the sand and not make that case, I have no sympathy for you.
I think that this may eventually become better now that there isn't so much dumb money around (no ZIRP) and with AI assistants taking on some low-effort work (enabling companies to lay off incompetent engineers). But it will take many years for companies to adapt and the transition won't be pretty.
At the end of the day it comes down to who the decision makers are and how they are incentivized to act. As a simple example - company X has product C, and they set a goal of increasing usage of feature F (of product C). Currently this feature F completely sucks and users don't want to use it - so the idea is to improve it and thus increase usage.
There are 2 ways of increasing usage:
1) Make the feature F more useful/better.
2) Force/push your users to use feature F, by aggressively marketing it, and pushing it within the product surfaces, making it non-optional, etc. and other dark patterns.
Option (1) is hard to do - it requires deep understanding of the product, user needs, the related tech, etc. It requires close tactical collaboration between product and engineering.
Option (2) is easy to do - it requires ~zero innovative thinking, very surface-level understanding of the problem, and relies purely on dark patterns and sketchy marketing tricks. You can almost completely ignore your engineers and any technical debt when following this approach.
If your decision makers are imposter PMs and marketing/sales people - they will almost always choose option 2. They will increase the 'apparent usage' of this feature in the short term, while reducing overall customer satisfaction increasing annoyance, and reducing the company's overall reputation. This is exactly how many 'growth' teams operate. Short term benefit/gaming of metrics for long term loss/reputational damage. Their success metrics are always short-term and linked directly to bonuses - long term effects of these kinds of strategies are ~always completely ignored.
We are an engineering discipline and engineering decisions can have revenue making implications. But it is hubris to assume why you should care about the nitty gritties of a codebase. It's the same way no one in leadership cares about the nitty-gritties of HR policies or accounting practices - people are hired to deal with the intricacies.
When I was a PM, I didn't have a difficult time making a business case for "keep the lights on" or tech debt work so long as I was able to attach tangible revenue implications (eg. X customer might churn because of subpar experience and we have both customer testimony and user stats showing that) or roadmap implications (eg. If we spend 6 months refactoring our monorepo, we can add new revenue generating modules every quarter instead of annually).
* Engineers are consulted for estimates to facilitate long-term planning
* Middle management slashes those estimates in half due to pathological myopia
* Executives enforce musical chairs to assert their authority
* Consultants muck everything up while collecting enormous payouts
And yet the business cycle keeps cycling.
Also, the process, security, approvals and compliance could dominate so much that less than 20 lines of code changes per week could become the norm and acceptable.
Everyone I hired is still there.
I'm fine with passing the baton, I enjoy mentoring.
Knowing those patterns is very helpful as a way to think about design problems, as long as you have the common sense to realize applying the pattern "by the book" is often overkill and you can just take some ideas out of it.
That article conflates as "Pure engineering" both reducing a software system to a small set of cohesive concepts, and architecture astronauts, when those are polar opposites.
“Bad” code is probably “good enough for now” code that was written some time ago on a bet that doing it better would never be needed as it wouldn’t need to change.
Also, “good” code is costly especially if taking longer to build the thing causes the company to miss its market.
I have met maybe 5 good engineers in my whole career. The size of the company did not matter. The reason is, the only thing that exists in our world today that can make you a good tech engineer, is yourself.
When you hear the word "engineer", you might imagine a professional who has done studies, passed exams, has certificates, maybe even apprenticed. They know a specific body of knowledge (which is maintained by some organization), they're held liable for their work. They are masters of their domain and they don't step outside of it.
But not if they're in tech! Then an 'engineer' can be a high school graduate or a PhD. Both can make the same amount of money, and have the same lack of real-world experience and job skills. They will both regularly apply technology they've never been trained on, never learning more than the least possible information to get a proof of concept working (and then that immediately becomes a production service). There's often no record of the decisions they made, no formal design process, no architectural review, no standards listed, no testing required, no risk analysis performed, no security/safety/reliability checklist performed. And they often are dealing directly with PII, with absolutely no thought to how to manage it. And they often have far more access than they should have, leak critical credentials everywhere, don't manage the software supply chain properly, don't even pin versions or even test rollbacks, etc. I have seen all of this at every single company I've worked for.
In any other 'engineering' profession, this would be illegal. Hell, it's sometimes illegal just to change a breaker in a subpanel in your home without pulling a permit, because doing it wrong has consequences. Think of all the times your personal financial records, health records, sensitive data, social security numbers, etc, have been leaked, just in the last year or two. 9 times out of 10 those happened because nobody cared enough to prevent it. But these things shouldn't be optional. There should be some kind of mandatory thing in place to force people to ensure this doesn't happen. And some kind of mandatory minimum requirements about what people know, what they're allowed to work with, and how. None of that applies in tech, yet we still call it engineering.
In my experience, big companies have the biggest incentive to write good code. They have the highest conviction in their bets, and they know with high confidence they will be around in 10 years. One large tech company I worked at had a rule of thumb that all code would need to be maintained for ~7 years - at which point, as the author points out, the entire team may have been replaced. This is precisely when the time it takes to write good code is a worthy investment
The code may look "bad" in a vacuum but if it yielded a successful outcome then the engineer was able to achieve his/her goal for the business.
The stories shared in this article are exactly what you'd expect from big tech. These are some of the most successful firms in the history of capitalism. As an engineer you are just grist in the mill. If you want to reliably produce "good" code then IMO become an artist. And no ... working at a research facility or non-profit wont save you.
Substitute "buggy" for "bad". The links in the first sentence of the article refer to bugs, which affect end users of the products.
> If you want to reliably produce "good" code then IMO become an artist.
This is not about aesthetics but rather about QA.
What I cannot stand and can barely tolerate is kruft and sloppiness. massive sections of commented out functions, leaving the poor guy to come along 2 years later wondering if it was important or why it was left there. Unused functions. bad, inconsistent, or nonexistent naming conventions. Terrible or annoying file/project structure.
None of this stuff has to be. Not doing this stuff requires a bare minimum of effort and time and doesn’t require any familiarity with a codebase. It’s a lack of professional pride, and that deeply annoys me when I inevitably have to clean it up because it’s an unreadable mess.
I think this is crucial. Even old hands working on their area of expertise can be compromised by deadlines.
In my experience at a FAANG working on one of the core services for both internal and external customers, essentially two kind of people crank out great code:
1. "rock stars": they joined the company at 25 and they're still there at 35+. they're allowed pretty much everything (eg: no RTO, work from home country) and they know many codebases across the services very deep. they aren't really motivated to go look elsewhere, their grass is already one of the greenest.
2. people with kids. the company pays enough. they aren't really interested in switching job, rather they want to provide for their family. they're good, and maybe every now and then will push through for a promo in order to face new challenges in life (another child coming or some kind of new financial burden).
i'm not saying either one is inherently good or bad.
but yeah. in such large companies you end up working in on a very large codebase that interacts with other very large codebases. all the codebases are proprietary and you're lucky if you can use some libraries that come from the outside world (that have not been heavily lobo^H^H^H^H customized - the libraries i mean).
you do what you can, you do your best, but you're essentially a relative beginner.
I tried almost in vain to fix these two ridiculous decisions, but the company refused to let code fixes through the review process if they touched "well-established, stable code that has been thoroughly tested." After being there for a couple of years, the only thing I was able to fight through and fix was the BEAM issue, which ultimately cost me my job.
My point in all this is that, at least sometimes, it isn't good engineers writing silly code, but rather a combination of incompetent/ignorant engineers making stupid decisions, and company policies that prevent these terrible decisions from ever being fixed, so good engineers have no choice but to write bad code to compensate for the other bad code that was already cemented in place.
I'd love to hear more about this!
> instead of using private functions as they're intended—created a pattern of SomePublicModule and SomePublicModule.Private, where he placed all the "private" functions in the SomePublicModule.Private module as public functions so that he could "test them."
Yeah, this is weird; you can just put your tests in the PublicModule. Or you can just solve this by not testing your private code ;)
He deployed our applications using Kubernetes and refused to implement libcluster. There was something else, too, but I can't recall what it was. It was seven years ago.
> Yeah, this is weird...
I kept telling this developer that you're supposed to test your private functions through your public interfaces, not expose your private functions and hope nobody uses them (which they did), but that fell on deaf ears. He was also a fan of defdeligate and used it EVERYWHERE. Working with that codebase was so annoying.
2. Eng teams that have mostly old-timers tend to get stale and slow in changes. This is bad for products that need rapid evolution or new ideas to break status quo. New engineers have way more incentives to make changes to prove themselves and collect credits, while old-timers tend to play safe and stay on the side of stability.
3. Bad coders, not new coders, write bad code.