To know that you don't know

I believe it was "The Mythical Man-Month" that postulated that the skill of people in their job varies by a factor of 100. Don't quote me on that, it's been a while since I read that book. But it tracks with personal experience, including my own career as a software developer. I have had the pleasure to work with colleagues who were absolutely widening my horizons, and I worked with star-developers who made me want to walk out due to their incompetence and insistance on factual errors.

For the record, I was a terrible developer starting out, and I also don't posit that I am "good" now. But I see a problem that seems to stem from the fact that it has become relatively easy to cobble together software that has, as its main positive trait, the fact that it functions. If you believe this to be good enough, then this blog post is not for you.

Back in 2016, the developer of a javascript package called "left-pad" had his code removed from the NPM repository, breaking a large portion of the popular javascript frameworks in the process. David Haney wrote an excellent blog post about this, check out his link for details and history, but in short: a javascript NPM package with a single function had somehow become a dependency in many JS projects and frameworks, and after the removal of left-pad, any project that depended on it failed to build. Worse, bad actors were able to publish a new package with the same name. I'll get to NPM in a second.

David Haney asked the question, "Have we forgotten to program"? I believe the question should rather be, "Have we not learned to program?" It appears to me that this is a matter of many people developing software who lack the required experience. I include myself in this, as i was way out of my league when I started out. I wrote code for things that I had no business working on, not without reading a lot more books and getting to know a lot more libraries and technologies. The thing is, I paid dearly for my mistakes. I received the calls in the middle of the night. I went on site to fix what I messed up.

I learned the hard way that lazy seasoned developers are usually better problem solvers. Their problem is work, so they want to get it done as quickly and efficiently as possible, and they don't want to get called at 4am. The most dangerous thing is a dev who walks into the office on Monday morning and proclaims that they wrote a new messaging middleware, or a new json parser, or something like that. Trust me, I did that. I was wrong.

I see two issues: One is developers with a lack of research capability. The other is a lack of oversight over available technologies. They are kinda two sides of the same coin.

An example for the first one is, again, the infamous left-pad "library". The fact that there has been a debate over whether a one-liner is a library is in itself a sign of a bigger problem: Recognizing bad code is difficult in a world where there are debates over whether the earth is flat, or whether medically trialled and tested vaccines are worse than the sickness. Many commenters argued that a library with a single function is still a library. This makes no sense. If you put a single book onto a bookshelf, you don't have a library. You have one book. There should not even be a debate about this, but since the required experience level to enter the conversation is so low, there is no end to the debates. When I look at the leftpad author's git over 300 repositories, there are "libraries" for one-liners such as parsing json in javascript. Left-pad was not a one-off, this was madness with method.

I will not link to the author's blog because frankly, reading his treatment about how it was all NPMs fault for "not checking who uses the code" is exasperating. But I will quote from it.

Most of my open source work followed Unix philosophy, 
so the packages did one thing at a time.

No. This is a bad idea. The Unix philosophy is to have individual programs that do one thing, and do it well. Conflating unix programs with javascript packages is a bad take. The way to go about this is to offer a library that covers a certain area, say, string manipulation, and curate that. Better yet, do what Rust, C++, Java etc. do and offer a set of standardized library functions that are extremely well tested and curated. Take the Apache Commons library for example, or Rust's "std" crate. The only time when this isn't desirable is when you have to write software for safety critical systems that have to follow ED109A/ED153 at a certain level, say, SWAL-2. If you don't know what that means, don't worry about it. We're talking airplane avionics, air traffic management, and spacecraft here. When you have to prove that there is no unused line of code in your project, then having a large standard library is actually a bad thing - but the way around it is to wrap it and only expose the functions you need. I've done this in the past and I'd much rather use Apache Commons than some package some javascript guy wrote without even commenting.

He then goes on to say this:

Several months after the left-pad incident, I quit my job 
and left US permanently, spent a year in Morocco, Jordan, 
Türkiye and Indonesia. I walked trails like Lycian Way, 
found new camping spots nobody knows about.

Left-pad was like a "death" and "re-birth" moment for me. 
The part of me passionate about open-source was dead, 
and something new took over. Now, I'm passionate about 
business, marketing, running companies / teams in 
different ways, as much as I'm about programming.

Life goes on.

Okay so... open source is the problem, and the salvation is closed source? I guess if you don't want to be called out for bad code, open source really isn't gonna be your passion.

Yes, NPM should have checked who uses this library before unpublishing it. In fact, NPM should not have had the option to completely unpublish a package, which exposes another problem with the Javascript ecosystem: The consequences of basic design decisions was not understood by the developers and designers of core infrastructure.

But aside from the issues surrounding NPM, left-pad was a bad idea from the start, just as other people using left-pad was a bad idea, and history has proven that it was a bad idea. The proliferation of single-function libraries is bad, because it fragments the ecosystem, well-curated libraries fall by the wayside in favor of some random google hit, and exactly such a situation happens.

Another reason why this is bad: A project with 1000 dependencies is more difficult to maintain than a project with 50. In the face of historic facts and overwhelming evidence, how is there still a discussion about this?

If the author cared about programming, his code would be documented and commented. He rarely bothered to do that. If he's as passionate about business/marketing/teams/etc then oh boy, that is not gonna go well.

The author of left-pad has learned nothing, and shifted the blame to everyone else. The problem was 1) You wrote bad code, 2) inexperienced devs pulled in your code as dependency, 3) the javascript ecosystem was designed by inexperienced devs. If your defense is basically "Oh I was just the first domino, and actually I was not a domino but a miniature brick", what kind of defense is that?

My issue with this whole ordeal is that people who don't face the consequences of mistakes tend to repeat them. Worse, old mistakes are not fixed because, well, they were not mistakes, right? It's all just, like, your opinion man.

Let's look at read-json, which is a library that, you guessed it, wraps JSON.parse():

var fs = require("fs");
var call = require("try-call");

module.exports = async;
module.exports.sync = sync;

function async (filename, options, callback) {
  if(arguments.length == 2){
    callback = options;
    options = {};
  }

  fs.readFile(filename, options, function(error, bf){
    if(error) return callback(error);
    call(parse.bind(null, bf), callback);
  });
}

function sync (filename, options) {
  return parse(fs.readFileSync(filename, options));
}

function parse (bf) {
  return JSON.parse(bf.toString().replace(/^\ufeff/g, ''));
}

There are many such libraries for reading json. They all do the same, but in slightly different ways and with extra stuff sprinkled in. For example, this one does a replace function on the string. Do you want that? Do you need that? Are you sure this is code you want in your project? I did not strip out any comments btw, because there weren't any.

In my experience, languages with a low perceived barrier of entry, or with a wide adoption by novices, tend to suffer the most from such bad-practice code stock.

Again, I am not saying that I'm perfect, and I have done plenty of stupid stuff like writing my own XML parser. But I eradicated all that because, no, you should not use that software of mine. I freely admit my mistake, and I am not offering any defense for it. I simply had no clue, or at least, only enough clue to make things compile and "barely work". And sometimes I was just too lazy for good code, which is also not something I should be proud of, or blame others for. I was ignorant, I discarded languages and technologies without ever having tried or understood them. I many cases I turned out to be wrong, but even if I was right it was a matter of luck, not some deep understanding. The difference is that I know that I made mistakes. I don't claim that I was right and that the world was just too cruel to recognize my tender genius.

The second issue is lack of technology overview. I have seen people try and implement a full text search using MySQL LIKE statements. In fact, I did something similar when I wrote a simple graphical online dune strategy game in PHP. My limited set of tools bit me. Now, I see enterprise java applications that take 15 minutes to start up and that do something you can code in an afternoon with modern technologies. If you use, for example, HP Systinet in 2024 then you are making a mistake. If your boss says you have to use MySQL to search addresses of an entire country, and you can't use something that's more suited for this? Get out of there. The principle of using the right tool for the job is not something that should require a debate. And while "I think MySQL is fast enough for this" is an opinion, sometimes it's just wrong. Does your project need to be written in Java? If you need a web-ui, does it automatically have to be favorite-js-framework+html? Do cross platform applications need to be Electron apps? 10 years ago, maybe, but today?

At the end of the day, we all make mistakes and nobody knows everything. But when I see authors of objectively bad code move on to found AI startups or go on sabbaticals only to then proclaim that it wasn't really their fault, and that they'll focus on people as much as they focus on programming, I get goosebumps. Because I know that people like that will write code that kills people, and they will never admit they made a mistake because, you know, it's just opinions man.

The problem is that not all opinions are validated just by virtue of being an opinion.