What's wrong with Software Development a.k.a. Ranting to the wind

As someone who writes code for most of my time, either as my work or in personal projects I find it extremely difficult to understand the lacking of proper written documents describing what should be accomplished and the reluctance in using the written format to discuss and document the development of software.

First let's go through what I called "written documents". There are different forms these can assume, in common they have that they specify to varying degrees of detail, technicality, and/or context the purpose, specifications, and requirements of a given piece of "software idea".

I call it "software idea", because that's what most things are until they're fully materialized as "working software".

Some of the more common documents (or artefacts) that concern this are:

Written Specifications
- Document(s) detailing, usually with high attention to technical capabilities, restrictions or requirements, in an unambiguous language, the working of an intended software program
User Stories
- Written documents detailing at a high level the intended functionality of a software program/capability, from the point of view of a user of the software interfaces
Design Documents
- Visual document, either in the form of diagrams, and/or actual illustrations of the intended software

There's probably more ways, or specific terms to specify what a system, functionality, or capability is to achieve, both in itself or when in bigger existing system (when it's not self-contained) but these cover most of the "ideas".

I don't expect everybody (including myself) to master all these forms of laying out projects, in all contexts and all situations. If I'm working with someone who is not a software developer and just wants a website that does X, I know for sure I will have to weight in my own knowledge, and do extra work to get into the details of what is wanted and access its feasibility. It's normal in these cases to have to go from simple user stories, and it's expected that you as the developer will take care of the technicalities related to implementing them.

If I'm working through a Project Manager (PM) then my expectations change a little bit. I'm no longer working with someone that simply wants an idea converted into software, but instead someone that, as the title implies, is responsible for managing a project, and as such I expect them to have both technical knowledge and to dedicate part of their time producing more relevant and accurate documents detailing what is to be implemented. Again, it's not reasonable to expect that a PM will know in all details both the underlying technology and any and all roadblocks, but it's normal to expect a well rounded specification and at least some understanding of the system, languages and platforms in use. This last part is relevant and I'll touch on it later on.

We'll focus more on the software being developed with a PM, since the other cases are in themselves a bit off-target when discussing the need for documentation and written artefacts besides the code. Most projects, even small ones, will usually have a PM that is in charge of overseeing the implementation of whatever is to be built.

Let me start by prefacing that I understand that a PM has a non trivial amount of things to worry about - managing bikeshedding, going through discussions full of cruft, making sure development is aligned with upper goals, making sure it's on track, solving issues, allocating resources and making technical decisions. Let me also add, that although I don't claim that I would do their work better than they do, their work is their work and not mine. If I was to be hired or paid to both oversee and develop (i.e. code), then my compensation needs to take that into account and in case it does, I will be more than happy to take on additional responsibilities. If on the other hand my compensation doesn't have equity and is priced at a time unit, then I don't think it's reasonable to expect me to take on additionaly responsibility that someone else is being paid to be responsible for. As with software, good separation of concerns is important. I will do my best nonetheless, but if your advice, suggestions and observations are continuously ignored irrespective of their value (either immediate or solving a problem that manifests itself later on) at some point you just don't do that anymore.

The same if after careful consideration, and even spending "non-billable" time on a problem, your work is rejected and replaced with some dim-wit solution that "is simpler", and discards all the knowledge you had gathered while solving it (here I'm not saying in the sense of being attached to the code you write, there's certainly cases where you come up with a solution that is more complex than it needs to be and it's welcomed in that case that a PM or team mate rejects it, it's part of their work and part of being in a team, I'm refering to ditching something for something else that has problems from the get-go, not only imaginary future problems).

So it's in this situation, working with a PM, that the reluctance in requiring and expecting written artefacts to be produced becomes interesting. In a way I would expect, and if I work as a PM would require, that documentation (discussions, specifications, guides, how-to's related to the project, etc) AND tests are something fundamental. As a PM I can't think of any other way of keeping up with how everything works, how everything is SUPPOSED to work, and how we got into this given place.

People insist on doing video/call based meetings where the "team" catches up and courses of action are decided. This is ok to do, and there might be a need to even have these outside of pure business/development concerns (as team building rituals not concerned with specific work related issues) but it can't replace written documentation. Folks whine about tacit knowledge but then don't take any action to solve it which, guess what, can only be solved with written documentation. People will prefer re-writing a completely correctly implemented solution, with all the costs that entails, because there's no one that knows how it works. PM's end up having no frigging clue of how the project they're managing works, is tied (or glued) together, all because they can't be bothered to go through more than 1 paragraph of text.

Here I will add, that developers are some times actively part of the problem as well. The PM's work is nonetheless making sure this is taken care of, not making it normal though. We can put in the list of developer-made problems, the unwillingness to learn specifics outside their area of comfort. Examples can be something that is written in a language that they don't have as much familiarity with. You sometimes see perfectly valid, tested, extensible code, be replaced by orders of magnitude more complexity, untested, brittle layers, just for the sake of sidestepping having to learn how something is being accomplished with much less. No documentation, no testing, no how-to's for getting pieces of a system working, etc. Although these are all problems a developer contributes too, the role of the PM should be to make sure these things are required and properly budgeted. At least for me, it's fairly easier to document something as I'm writing it as opposed to after the fact.

This lack of written documentation (or performative documentation, such as tests - in the sense that it performs work to asserts the validity of its contents) then exacerbates the issues with keeping up with how things work, if they should be working, and having a complete view of a system. On top of that, it makes on-boarding fairly more difficult as one can't for the most part just go on "discoverability" mode and instead has to rely on lower level code to gather what the "systems" should be doing. It's also fairly more difficult to access if a system is complex because of the domain it covers or if because it's being wrongly thought about.

Things such as refactoring become a painful chore, because, why would you try to refactor something when you don't even have a baseline of expected functionality asserted? Or at least something explaining what that functionality is expected to do? So if it's not product breaking it just gets pushed to the "backlog" void. If it's product breaking then all the required tests to make sure the refactoring conforms have to be written - probably by someone who didn't write the initial implementation, doesn't have all the context, nor knows many of the pitfalls encountered during development because, you guessed it, there's no documentation.

Then, when the backlog void threatens to suck up the whole thing, the solution is to re-write. On top of this you might also have developers just deciding to quit or unable to have any semblance of productivity.

So when people go about how they don't have budget for documentation, testing, to put effort into making your software understandable what they're actually saying is they don't care at all for the costs of developing software. Sometimes this makes sense, again, if it's a online freelancing gig for a test-the-waters website, sure, go ahead - it might not even work so those costs would all be extra. But even in those situations, if the software is expected to have some modicum of complexity, tests and documentation will inevitably pay themselves in anything running for more than 2 months, or when you have to have another developer picking up from the existing work.

And what is really mind-boggling is that having documentation and written word as first-class artefacts in a project as the baseline helps beyond these directly related issues already mentioned.

Jobs will have descriptions such as "employee budget for improvement/knowledge material/courses", "attending conferences", etc. But just writing proper documents that are to be used by people of varying technical literacy is a valuable skill. No-one is expected to do all things required for software development super-well, but this is probably one of those things that translates into any field of work, any language stack, any problem domain.

Then you'll see companies hiring for remote positions, but they can't produce any written documents, or specifications. In terms of hiring I also think that this is something that can be greatly improved by having a culture of written first. If your processes, your software, your systems, are documented, it's fairly more easy to:

Produce a specification for interview takeaway problems (either new, or using something already implemented, in that case if you have tests then you can automatically measure the functionality of the returned solutions and then look into subjective aspects such as code organisation and others) - this is enabled because it's normal to specify things, and so it's easier to generate a new one because PMs, software engineers (lol!), and others have at least some experience doing it;
Communicate effectively asynchronously - if your company can't produce written documentation, can't communicate clearly through written form, how am I supposed to believe somehow you are able to foster a remote environment that has any semblance of working?
Onboard new people - yes, nothing replaces being able to talk directly to a person and have a question answered on the spot - the issue is that given the complexity of software systems, there is a lot of groundwork that needs to be done when coming up to speed - if the answers for 70% of it are written and discoverable, it's better than 0% and 100% of the questions needing some person to answer them, or having to figure them out by backtracking through actual code, instead of clearer high-level descriptions;
Inter/intra team communication - if nothing is written, again the overhead of sharing knowledge is extremely high - some people will also be reluctant to engage if the other person doesn't show that at least they tried understanding something first - written documents can help this as well, since someone can do their own research and be left only with questions related to details - or not have questions afterall - it also shows how the documentation might be improved or is somehow missing so effective documentation improvements can be fed into these loops;
Lastly, and this is perhaps personal, I think it's also a proxy for team fit - not a fit into a particular team, but fit into the idea of being in a team - people who take care in explaining how things work in order to be grokkable by others seem to me more inclined to care less about politics and not of being in a position just to take a payment at the end of the month/sprint - basically because by making their work accessible to others they're removing the need to rely on their tacit knowledge.

And this is why to me it's a bit mind-boggling that it's not a core part of software development team processes. It's not only for developers, it's for PMs too and other members. It's not only about facilitating knowledge sharing and reducing bus factor, it's also about improving skills that are useful in a wide range of fields beyond software development (the ability to express ideas in the written form - because that scales). It's not only about empowering developers but about steering product development to a model where things are based more on their merits than on who has the most clout, political or personal, or who is the smartest-talking ape. It's also funny that people eagerly will copy tech-stacks and easy to do rituals (like stand-ups, daily calls, etc) from successful companies, but then the things that are far-reaching, such as emphasis on written documentation/preparation/spec'ing, are just ignored, probably because you can't just tick an item off a list and have to actually produce usable artefacts.

So the take-away I want to leave you with is; if you want to improve the quality of software that you or your team is producing, you want to empower and make a more resilient team, and diminish the long-term cost of software - start by documenting, spec'ing, testing and making this part of the normal expected workflow.

Most software might start trivial, or start as different services trivilly glued together, but complexity grows exponentially as it develops - not only because it has to take care of much more use cases as it expands, but because it usually has to rely on an increasing number of moving parts, from infrastructure, to packages, libraries, different languages, deployment targets, requirements and constraints and the costs of lacking "written artefacts" remain hidden. A developer that quits because everything is an ungrokkable mess doesn't show up as a direct cost of this. A developer having their productivity hampered in half doesn't show up as this. The need for a re-write because no one understands the system anymore doesn't show up as a cost of this. A new-hire having to spend valuable time from other more knowledgeable developers, that could easily be documented, doesn't show as related to this. Your PMs making wrong decisions doesn't either.

In the end, most of this is related to two things:

the lack of willingness to admit that the written part of software development is essential - which shouldn't be a suprise since what we do as developers is writing code, sometimes comments, and we all know that the difficult part is clarifying the logic of what something should do and not the syntax specifics of given languages;
the build to be acquired mentality - it doesn't matter if it's a turd, as long as you can make it look nice on numbers - many endeavours in software seem to only have the goal of being sold - and when that is the case, effectively, these might be additional costs and just feature after feature in a checklist, even if the base is a steeming pile, make it look more palatable - the truth is, it can also sink these ones, there's a breaking point after which you might not be able to have functioning systems and produce new features, and no matter how you sell it, it will show.

(of course, as with everything, there needs to be a balance, the same way you don't need to test everything that is in your code base to have a good degree of confidence, you might only test the overall code "paths" in some end-to-end fashion, and then specific functionality isolatedly, some things are self documenting, but I would arguee that many, surprisingly, are not, for anyone else than the one writing them - and that has a time validity as well, just like yogurt, after which they too, no longer understand their "clear self-documenting code")