Briefly speaking, the scientific method is the idea that you should formulate a hypothesis about the expected results of an experiment before looking at its actual results. Then compare the actual results to the expected results. In computer science – and software engineering, its applied field – the scientific method seems to be widely neglected though. I read a lot of computer science papers that don’t follow this method and it’s practically absent in software programming (notable exception: TDD).

This doesn’t have to be case. The scientific method works very well even in the lowliest places of programming. The first time I remember consciously thinking about this topic was in college when I – then a TA – watched students step through code in their debugger. Students single-stepped over lines of code, observing what happens. That’s not how debugging works best. Rather, before you step over a line of code, form a hypothesis about what’s going to happen, and check the expected result against the actual result once you stepped over the line.

Outside single-stepping through lines of code, the scientific method works pretty much during all parts of software developments. Making a change to speed up a part of the program? Estimate how much faster the program is going to run; maybe 10%. If the actual result is far off, you need to backtrack to understand why your understanding of the program didn’t align with reality. Same with A/B testing of GUIs. Is a round button or a rectangular going to lead to more click-throughs? Don’t just test it and see what happens. First, formulate a hypothesis how many more click-throughs you expect from the button change. If the observed user behavior is far off, you don’t understand your users and need to backtrack.

Further reading:

• The Role of the Scientific Method in Programming (Robert Sedgewick)
• Debugging and the Scientific Method (c2 wiki)
• The Scientific Method (c2 wiki)

There are programmers who proudly proclaim ignorance of computer science fact knowledge because they can just search for definitions or code on the Internet. “Bubblesort? Why remember? You can Google that!” These people limit their own employability and career options. Facts fluency is required after a certain point.

One of the most interesting things I received in college was a cheat sheet of the minimal mathematical definitions of NP-complete problems. It was a double-sided single sheet of paper with maybe a total of 25 problem definitions. We used that to prepare for the final exam of one of the algorithm complexity classes. That was about ten years ago and I still remember plenty.

This was also the first time I realized that terms like “bin packing” or “maximum cut” in the language of computer science serve a similar function as terms like “dog” or “cat” in the English language. When you first acquire the language, you learn and rehearse these terms independently. After you mastered the basic terms you can combine them to create meaningful sentences.

The vocabulary of computer science is like any other vocabulary. You have to learn it and if you don’t use it, you lose it. After mastery of terms and sentences, you can reach the next skill level: to see a problem and immediately recognize its core, ignoring the frills that make it different from textbook examples. That’s a skill that can not be Googled because you don’t even know what terms to Google for.

I encourage every programmer to regularly brush up on the basics. You can do the minimum by reading Wikipedia. If you want to be an overachiever you can go to a page like the Dictionary of Algorithms and Data Structures and create a set of flashcards with terms to rehearse.

When I explain my idea of software design to people I draw a simple two-dimensional graph. The x-axis is the project size. The y-axis is the required effort to make a change to the project. There are two functions shown in the graph. One is slowly growing linearly and represents the cost of changes in a well-designed project. The other one is growing quadratically and represents the cost of changes in a badly designed project. The linear function starts above the quadratic function. A well-designed project requires more effort up front. Over time, a badly designed project requires more and more effort until the cost of changes passes and exceeds the cost of changes in a well designed project.

In the best case, the cost of change is independent of project size. This ideal can not be reached. A change in a 100 million LOC project is always going to cost more than a similar change in a 100 LOC project. However, we should strive for this ideal. In fact, decoupling the cost of changes from project size might be the ultimate reason for thoughtful project design. Maybe all of its other benefits are just steps on the way.

There is a big-O analogy in here. Let’s define big-P as the function that provides the worst-case cost for making a change to a project. In projects of P(1) the cost of changes is independent of project size. This is the theoretical ideal. Projects are in amazing shape if they are in P(log n) where a change of size n requires only twice the effort in a 10x larger code base. P(n) is still OK and probably better than most existing large projects. Beyond these complexity classes, projects quickly reach infinite costs for changes: desired changes become impossible.

Documentation that is closer to code is more likely to be read and updated. Information in a wiki or shared documents in the cloud go out of sync with code very fast. The effort and diligence required to keep them in sync is an intense job most programmers are unwilling to do.

The obvious conclusion is to tie documentation very closely to code. However, this does not help. Programmers seem to have a kind of comment blindness. Even existing comments that specifically answer the programmer’s questions are often subconsciously ignored. And that’s just the reading part and doesn’t touch on the problem of updating comments. It doesn’t help that many IDEs visually de-emphasize comments over code.

This reminds me of code testing. In the beginning, programmers wrote code. As code became more complex they realized how hard it became to make changes without breaking something. Many programmers started to adopt testing frameworks to quickly learn about new defects in changed code. However, code and tests quickly went out of sync and test coverage decreased over time. Code coverage tools were created and and some companies these days require minimum coverage percentages for changed code to make sure tests stay in sync with code. Many companies have dedicated test engineers who handle that.

Can the same be done with documentation? What if you made a change to a commented line and some part of the toolchain explicitly asked you afterwards if the old comment still applied? Make a change to a method and the toolchain asks if the method documentation is still appropriate. Same for classes, packages, and whole programs. Whenever you make a change, you have to confirm that the existing comments and documentation on the same abstraction level still apply.

Will we see companies with dedicated comment/documentation engineers?