In this latest installment of Programming is hard, let’s talk about abstractions. (For previous readings, see Part 1 and Part 2).

I read a column by Ted Neward where he points out how software development is unlike any other industry. We have to deal with abstractions ALL the TIME. As he stated so eloquently, you can’t just open the computer and point out where the threads, locks, and classes are. Those are just abstractions we invented to avoid dealing in low level assembly code.

But it doesn’t stop there. Assembly code is another abstraction. I studied Computer Engineering at Auburn University, not Computer Science. Not only did I study programming, but I learned how to build circuits, flip-flops, and use logical gates to create certain outputs. Computer chips are basically a bunch of electrical impulses. We want to get the customer’s spreadsheet of data to translate into a bunch of electrical inputs to a plethora of gates, and then convert the output into some web page. The funny part is that we never deal with the real, physical nature of the computer chips.

This is why software is so different and consequently so hard. Even theoretical physics, thought by many to be the most complex and bizarre field of study, have to bow to real observations at some point before proceeding further.

Einstein wrote a few papers about his theory of how gravity operated. He came up with an assumptive model (elastic space influenced by the indentation of mass), hammered out the mathematics of that model, and derived a prospective outcome. His outcome was shocking, because it suggested that Newton was wrong. Newton’s theory of gravity had stood for 400 years with many observations that made it rock solid. The scientific community wouldn’t listen to Einstein without observed evidence. By “wouldn’t listen,” I mean that his academic career may have been cut short because no one else would proceed to build their research on top of his theory if it couldn’t be proven by some observation.

In a nutshell, Einstein predicted that the light rays from distant stars would be warped when passing around our sun by a certain amount. His calculations produced a different result than what Newton’s laws predicted. The only way to prove this was to observe the positions of these distant stars during a solar eclipse when they weren’t washed out by the sun. An exploration team traveled to deep jungles in order to verify this, and they succeeded. The proverbial inventor of the theory of gravity, Newton, had been corrected, and Einstein is now recognized as one of the greatest physicists of all time.

In software, we don’t have to observe the real, physical electrical signales generated by our code. Instead, we only have to measure if it provides our users with the output they want. This translates to spreadsheets, web pages, pages, or other “real things” that tell the customer their system worked. Why do you think so many you speak to get a glazed look if you talk about anything below the surface of software?

The ophthalmologist I grew up visiting retired at a few years ago, went back to school and got a law degree just for fun. He then tried computer science and gave up because it was too hard. This guy had a medical degree and law degree, but computers were too much! Heck, I knew someone in high school that made it to final round of the televised teen tournament of Jeopardy, but when we were both in computer science class, he needed me to tutor him because he just couldn’t get it. Programming is hard!

Why do you think multi-threaded programming is so hard? It’s because there are so many ways to implement it, and we only have the theories and abstractions invented so far. It’s not a real, physical problem, but instead an abstractive problem we need to tackle. I have read 20-year-old text books that talk about spin locks, semaphores, and other things to control the system while switching threads. In my research paper I wrote in grad school, I talked about real time systems and how different operating systems provides the means for different threads to interrupt and take control. Merely switching from one thread to another has different solutions. So it’s no surprise that we are still debating whether to lock state and switch threads, pass messages between actors in order to isolate/reduce state in the system, or use some sort of transactional software memory to move state around.

Functional programming is offering some new ideas on how to isolate state to reduce complexity, and thus make it easier for threads to switch at will, reducing how hard it is for us to handle things. But functional programming has been around for 50 years, so is it really new? FP isn’t new, but the problems we are solving today are vastly more complex than what people needed 50 years ago. ENIAC is credited as the first electronic computer, and it one of it’s biggest tasks was to calculate ballistic firing tables for the military. Multi-threaded programming wasn’t a requirement.

Today we are dealing with data warehouses that host terabytes of information. Facebook recently moved several pedabytes of data from one location to another, because they needed more space. Heck, the “settled” concept of using SQL to store and retrieve data is up for grabs when it comes to managing “Big Data”. Our problems are growing in size and complexity, and it’s up to us to craft better and better paradigms of writing code. That is why things like scala and clojure open new doors to help us solve this tough issue of programming. We have to keep investigating and figure out what is more effective, and who says there is only one solution?