3 min read

Software development is broken

Software development is broken - tools have proliferated, processes have fractured and there has not been much investment in building better fools for developers. Every language has its own package manager, its own build tooling, its own ecosystem; dependencies have proliferated such that a bare-bones React application pulls down nearly a gigabyte of additional libraries and tools. Working on a new project requires any number of weird contortions in order to start contributing to it. Visibility into what goes into a release, where your code is in the process, what’s deployed into production, who is working on what, all of this is strangely absent from today’s ecosystem.

Investment in developer tooling

This is a strange notion, that there have been no major efforts at improving the development process. Given that developers are, well, developers, it would seem only natural that we would want to improve our process to be more humane. Before the explosion of programming languages building some software went pretty much the same way: ./configure [--options] && make -j ${NUMCPUS} && make install - you hoped that you had all the required libraries and include files on your system and, if the C gods were smiling upon you, you were able to build that sweet new IRC client.

Nowadays, if I want to contribute to an open source project, I need to spend time getting my development environment setup (do I have the right version of Ruby? Am I using virtualenv or pyenv? Is Node installed?).

Inconsistent Process

Not all software is “a service” (web services, monolithic web application, etc.) but most software follows the same process – download, install tools, compile, test, run, deploy. One of the problems with this flow is that my machine may be dramatically different than what’s running in production, or totally different from another developer’s system.

(Not-so) Continuous Integration

Many projects leverage some form of CI system - CircleCI and Travis being quite popular. However, these systems are only used when a developer wants to merge some code into a project meaning that there is a disconnect between local development and the way that the CI system is running things.

Lack of Isolation

Development is typically done in the presence of the remainder of the system - what I mean by this is that if project A uses PostgreSQL 11 and project B uses PostgreSQL 9, I need to either install both (which can be a headache) and do it in such a way that they are not in conflict with one-anothe or, if that is not possible, choose one of the two and hope that the other is compatible with it. Additionally, even if those projects both use the same version of PostgreSQL, they are now sharing an installation, which means that if one project requires a specific configuration then the other project implicitly must adopt that configuration as well.

Beyond external services, the same issue exists with C libraries – anyone that has had to compile a Ruby library like Nokogiri will understand the pain here. Depending on the version of Ruby and Nokogiri, you will likely need a different version of libxml2.

But what about containers?

Containers are great for this isolation, the problem is that they are not really being used as part of the development cycle. docker-compose provides some assistance when running multiple containers for a project but it is not tightly integrated with the tooling; developers need to do some work by hand to make the system functional.