Briefly speaking, the scientific method is the idea that you should formulate a hypothesis about the expected results of an experiment before looking at its actual results. Then compare the actual results to the expected results. In computer science – and software engineering, its applied field – the scientific method seems to be widely neglected though. I read a lot of computer science papers that don’t follow this method and it’s practically absent in software programming (notable exception: TDD).
This doesn’t have to be case. The scientific method works very well even in the lowliest places of programming. The first time I remember consciously thinking about this topic was in college when I – then a TA – watched students step through code in their debugger. Students single-stepped over lines of code, observing what happens. That’s not how debugging works best. Rather, before you step over a line of code, form a hypothesis about what’s going to happen, and check the expected result against the actual result once you stepped over the line.
Outside single-stepping through lines of code, the scientific method works pretty much during all parts of software developments. Making a change to speed up a part of the program? Estimate how much faster the program is going to run; maybe 10%. If the actual result is far off, you need to backtrack to understand why your understanding of the program didn’t align with reality. Same with A/B testing of GUIs. Is a round button or a rectangular going to lead to more click-throughs? Don’t just test it and see what happens. First, formulate a hypothesis how many more click-throughs you expect from the button change. If the observed user behavior is far off, you don’t understand your users and need to backtrack.
There are programmers who proudly proclaim ignorance of computer science fact knowledge because they can just search for definitions or code on the Internet. “Bubblesort? Why remember? You can Google that!” These people limit their own employability and career options. Facts fluency is required after a certain point.
One of the most interesting things I received in college was a cheat sheet of the minimal mathematical definitions of NP-complete problems. It was a double-sided single sheet of paper with maybe a total of 25 problem definitions. We used that to prepare for the final exam of one of the algorithm complexity classes. That was about ten years ago and I still remember plenty.
This was also the first time I realized that terms like “bin packing” or “maximum cut” in the language of computer science serve a similar function as terms like “dog” or “cat” in the English language. When you first acquire the language, you learn and rehearse these terms independently. After you mastered the basic terms you can combine them to create meaningful sentences.
The vocabulary of computer science is like any other vocabulary. You have to learn it and if you don’t use it, you lose it. After mastery of terms and sentences, you can reach the next skill level: to see a problem and immediately recognize its core, ignoring the frills that make it different from textbook examples. That’s a skill that can not be Googled because you don’t even know what terms to Google for.
I encourage every programmer to regularly brush up on the basics. You can do the minimum by reading Wikipedia. If you want to be an overachiever you can go to a page like the Dictionary of Algorithms and Data Structures and create a set of flashcards with terms to rehearse.
When I explain my idea of software design to people I draw a simple two-dimensional graph. The x-axis is the project size. The y-axis is the required effort to make a change to the project. There are two functions shown in the graph. One is slowly growing linearly and represents the cost of changes in a well-designed project. The other one is growing quadratically and represents the cost of changes in a badly designed project. The linear function starts above the quadratic function. A well-designed project requires more effort up front. Over time, a badly designed project requires more and more effort until the cost of changes passes and exceeds the cost of changes in a well designed project.
In the best case, the cost of change is independent of project size. This ideal can not be reached. A change in a 100 million LOC project is always going to cost more than a similar change in a 100 LOC project. However, we should strive for this ideal. In fact, decoupling the cost of changes from project size might be the ultimate reason for thoughtful project design. Maybe all of its other benefits are just steps on the way.
There is a big-O analogy in here. Let’s define big-P as the function that provides the worst-case cost for making a change to a project. In projects of P(1) the cost of changes is independent of project size. This is the theoretical ideal. Projects are in amazing shape if they are in P(log n) where a change of size n requires only twice the effort in a 10x larger code base. P(n) is still OK and probably better than most existing large projects. Beyond these complexity classes, projects quickly reach infinite costs for changes: desired changes become impossible.
There are some seriously old software projects around these days. Take SPSS which was first released in 1968 and is still updated. When a new person joins the SPSS team, how do they learn what has already been tried before? Are they bound to repeat bad ideas of the past? Is there a reference to tell them their idea was already explored and discarded in 1983?
You don’t need to have 50 years of project history to wonder how a project arrived at the status quo. Even if your project is just a few years old, new hires ask about past decisions. Often people who were on the team at the time of a decision can not recall why a decision was made the way it was made.
I have noticed that development teams seem to be focused on the future. Their process, documents, and tools are optimized for what must happen next. This is great for focusing on shipping product. After shipping, the usefulness of the documents and tools seem to crumble. They are not maintained anymore, become obsolete and outdated. It’s now difficult to recall questions that were asked and answered in 1993.
I feel that projects need scribes or archivists that track key decisions in a structured way. Their system would track key decisions and the reasons behind them. I want this system to be easily searchable to find decisions made years ago. I also want this system to be readable as prose in chronological order like a history novel. I want entries to be taggable and relatable, for example to highlight that a decision from 1997 was made obsolete by a newer decision from 2004.
I haven’t found tool support for this idea. Maybe it’s been tried and discarded but nobody kept records.
Documentation that is closer to code is more likely to be read and updated. Information in a wiki or shared documents in the cloud go out of sync with code very fast. The effort and diligence required to keep them in sync is an intense job most programmers are unwilling to do.
The obvious conclusion is to tie documentation very closely to code. However, this does not help. Programmers seem to have a kind of comment blindness. Even existing comments that specifically answer the programmer’s questions are often subconsciously ignored. And that’s just the reading part and doesn’t touch on the problem of updating comments. It doesn’t help that many IDEs visually de-emphasize comments over code.
This reminds me of code testing. In the beginning, programmers wrote code. As code became more complex they realized how hard it became to make changes without breaking something. Many programmers started to adopt testing frameworks to quickly learn about new defects in changed code. However, code and tests quickly went out of sync and test coverage decreased over time. Code coverage tools were created and and some companies these days require minimum coverage percentages for changed code to make sure tests stay in sync with code. Many companies have dedicated test engineers who handle that.
Can the same be done with documentation? What if you made a change to a commented line and some part of the toolchain explicitly asked you afterwards if the old comment still applied? Make a change to a method and the toolchain asks if the method documentation is still appropriate. Same for classes, packages, and whole programs. Whenever you make a change, you have to confirm that the existing comments and documentation on the same abstraction level still apply.
Will we see companies with dedicated comment/documentation engineers?
If you write Java or C++ code during a whiteboard interview you are probably doing it wrong. There are good reasons to choose Java or C++ for whiteboard interviews. Maybe your interviewer asks you to use one of these languages. Or maybe you know that one weird Boost trick that C++ standard authors hate. If you don’t have a very good reason, stay away.
Time is very precious in interviews. How much time do you usually have? Forty-five minutes? Sixty minutes? How much of that is left after small talk with the interviewer and discussing the problem? Thirty minutes? Maybe thirty-five minutes? Unfortunately, you are really slow handwriting code on a whiteboard. Don’t waste time on a programming language with a lot of syntactical overhead. Tokens per minute matter. Pick a language like Python to write denser code faster.
Donnie Berkholz crunched the data and tried to come up with a metric for the expressiveness of programming languages. Without discussing the validity of this particular metric, I like the idea. For a whiteboard interview please choose a language that’s far to the left side on Donnie’s results diagram.
Now you might say that other things matter more in an interview. Maybe think more before you write code. Yeah, maybe. Maybe whiteboard interviews are a terrible idea to begin. Yeah, maybe. Maybe you have other complaints about the common software engineering interview process. Yeah, maybe. None of these matter when you’re standing in front of a whiteboard, black marker in your hand. These things are also outside the scope of this post. Maybe I’ll cover them in the future. Yeah, maybe.
I see too many candidates who choose to put themselves at a disadvantage by voluntarily using a syntactically heavy language at the whiteboard. Don’t be one of them.