Category Archives: hard

Scala is EJB2? I don’t think so

I just read Stephen Colebourne’s blog entry that is circulating the twittersphere. I’m about to post some comments, so I guess this means that I don’t “tolerate dissent.”

First of all, please don’t pull out the infamous “(0 /: l)(_ + _)” argument okay? I get it that not everyone understands foldLeft’s, but it’s not that hard!?! I hadn’t done this type of thing since college, and after about 5 minutes of reading, I grokked it pretty well. I went on to give a presentation on functional programming at NJUG and even used this as one example of programming that is an option in your toolbox. I can’t really say much more other than a) don’t use it or b) read In Defense of (0 /: l)(_ + _) by Ricky Clarkson.

I think someone else already captured my opinion regarding discussion about loving/hating scala: “In my experience this is completely inaccurate, dissent is constant in the community. What is not tolerated is stupidity, unthink or fud. The Scala community contains amongst it a very bright set of individuals who do not let even confirmed Scala enthusiasts get away with the slightest exaggeration and will passionately debate many a sacred cow.” —jed.

I just don’t understand the “scala is complex” idea. We have to solve complex problems! Get it? Things aren’t easy. We are having to evaluate frameworks and tools to build systems. So many new, hot things are being built. Why was Ruby On Rails so popular? Because it enabled us to build e-commerce sites a lot faster than before. Same thing for Grails and Spring Roo. Neither Java nor Ruby by itself was enough for people to do their jobs. Well, I shouldn’t say that. Lots of people built lots of systems using Java and Ruby, but once these rapid app development tools came out, people ate them up fast! It’s because we have tough jobs and anything than can bend the curve helps us out. The same can be said for scala. We are able to bend the curve and write more functionality that captures what we are thinking with it.

Love it or hate it, I really liked PERL. I built mean systems using it, and that was because it was the first language I met that made regular expressions incredibly accessible. Today, I don’t think of using it, because Python has what PERL had and more. If I didn’t have anything like Python, Ruby, or Groovy to do pattern matching, then I wouldn’t hesitate to grab PERL. Now as scala emerges, it’s ability to do algebraic type matching is incredible. I think it sets a new standard in what other languages should offer in pattern matching abilities. So, just because I can write some bad patterns, I should chuck it and move on? I don’t think so.

Some of the criticisms are dead on and point out where effort must be heavily invested. Lack of test suite is critical in my book. Instead of dumping scala due to a lack of a test suite, how about bolstering it as the thing to improve its quality? How many systems have you dropped or discarded because they just didn’t have enough automated testing? I hope not too many, because it’s only recently that I have seen automated testing really take off as the way to go, despite years of preaching.

To me, learning new languages has always been part of the fun of programming. I studied many languages, worked on lots of projects using several including BASIC, C, C++, Assembly, Ada, Tcl, Tk, Expect, Bash, SQL, Java, XML, JavaScript, Scala, Groovy, Clojure, Prolog, and others. Each of these has some amazing aspects as well as funny quirks. It has always been exciting to move to a new technology and discovered what I can really do. As I’ve learned new languages and new paradigms, I feel it has made me a better programmer because I am more rounded. I’m not confined to my provincial thinking that was the result of tinkering with BASIC as a kid. Now I think of big and complex concepts and immediately start chopping them down to small solutions in my mind before my fingers even hit the keyboard. Scala is another one I would be chomping at the bit to work on for a full blown project.

Programming is hard – Part 3

In this latest installment of Programming is hard, let’s talk about abstractions. (For previous readings, see Part 1 and Part 2).

I read a column by Ted Neward where he points out how software development is unlike any other industry. We have to deal with abstractions ALL the TIME. As he stated so eloquently, you can’t just open the computer and point out where the threads, locks, and classes are. Those are just abstractions we invented to avoid dealing in low level assembly code.

But it doesn’t stop there. Assembly code is another abstraction. I studied Computer Engineering at Auburn University, not Computer Science. Not only did I study programming, but I learned how to build circuits, flip-flops, and use logical gates to create certain outputs. Computer chips are basically a bunch of electrical impulses. We want to get the customer’s spreadsheet of data to translate into a bunch of electrical inputs to a plethora of gates, and then convert the output into some web page. The funny part is that we never deal with the real, physical nature of the computer chips.

This is why software is so different and consequently so hard. Even theoretical physics, thought by many to be the most complex and bizarre field of study, have to bow to real observations at some point before proceeding further.

Einstein wrote a few papers about his theory of how gravity operated. He came up with an assumptive model (elastic space influenced by the indentation of mass), hammered out the mathematics of that model, and derived a prospective outcome. His outcome was shocking, because it suggested that Newton was wrong. Newton’s theory of gravity had stood for 400 years with many observations that made it rock solid. The scientific community wouldn’t listen to Einstein without observed evidence. By “wouldn’t listen,” I mean that his academic career may have been cut short because no one else would proceed to build their research on top of his theory if it couldn’t be proven by some observation.

In a nutshell, Einstein predicted that the light rays from distant stars would be warped when passing around our sun by a certain amount. His calculations produced a different result than what Newton’s laws predicted. The only way to prove this was to observe the positions of these distant stars during a solar eclipse when they weren’t washed out by the sun. An exploration team traveled to deep jungles in order to verify this, and they succeeded. The proverbial inventor of the theory of gravity, Newton, had been corrected, and Einstein is now recognized as one of the greatest physicists of all time.

In software, we don’t have to observe the real, physical electrical signales generated by our code. Instead, we only have to measure if it provides our users with the output they want. This translates to spreadsheets, web pages, pages, or other “real things” that tell the customer their system worked. Why do you think so many you speak to get a glazed look if you talk about anything below the surface of software?

The ophthalmologist I grew up visiting retired at a few years ago, went back to school and got a law degree just for fun. He then tried computer science and gave up because it was too hard. This guy had a medical degree and law degree, but computers were too much! Heck, I knew someone in high school that made it to final round of the televised teen tournament of Jeopardy, but when we were both in computer science class, he needed me to tutor him because he just couldn’t get it. Programming is hard!

Why do you think multi-threaded programming is so hard? It’s because there are so many ways to implement it, and we only have the theories and abstractions invented so far. It’s not a real, physical problem, but instead an abstractive problem we need to tackle. I have read 20-year-old text books that talk about spin locks, semaphores, and other things to control the system while switching threads. In my research paper I wrote in grad school, I talked about real time systems and how different operating systems provides the means for different threads to interrupt and take control. Merely switching from one thread to another has different solutions. So it’s no surprise that we are still debating whether to lock state and switch threads, pass messages between actors in order to isolate/reduce state in the system, or use some sort of transactional software memory to move state around.

Functional programming is offering some new ideas on how to isolate state to reduce complexity, and thus make it easier for threads to switch at will, reducing how hard it is for us to handle things. But functional programming has been around for 50 years, so is it really new? FP isn’t new, but the problems we are solving today are vastly more complex than what people needed 50 years ago. ENIAC is credited as the first electronic computer, and it one of it’s biggest tasks was to calculate ballistic firing tables for the military. Multi-threaded programming wasn’t a requirement.

Today we are dealing with data warehouses that host terabytes of information. Facebook recently moved several pedabytes of data from one location to another, because they needed more space. Heck, the “settled” concept of using SQL to store and retrieve data is up for grabs when it comes to managing “Big Data”. Our problems are growing in size and complexity, and it’s up to us to craft better and better paradigms of writing code. That is why things like scala and clojure open new doors to help us solve this tough issue of programming. We have to keep investigating and figure out what is more effective, and who says there is only one solution?

Programming is hard – Part 2

NOTE: You can also read Part 3 of this blog series.

The debate about the complexity of scala ensues. As I previously wrote, programming is hard! Trying to conclude how easy or hard scala is based on language constructs isn’t feasible.

In a really active thread on the LinkedIn Scala Enthusiasts group, someone posted:

…encapsulation is the most important concept in OO and plays no role in Functnal Programming. So these 2 paradigms are not naturally combinable. I think designing fully and only Functional language compilable to JVM bytecode with ability to call methods of Java classes/instances would be much more natural combination.

I’m sorry but this point is totally wrong. The first part where the author suggests that object oriented and functional programming concepts are not combinable is based on a totally false premise.

This can be demonstrated by the fact that languages like Python allow you to use classes and functions all in the same module. You can model things with classes, but capture important operations used by your objects in functions that aren’t part of your class structure. The class methods can invoke functions and the functions can use the class structures in a symbiotic way. This is because not every problem requires the ceremony of classes. In simple, contrived examples, I usually don’t need to chop things up like this, but when I’m working on complex, real systems I like having options available, and sometimes mixing functions and classes together is alright. Think of it like nouns and verbs (read this side article, it’s awesome).

Scala supports this too by letting you put functions inside objects, essentially namespacing them. You can build an entire module as a collection of functions with no classes at all. Define some classes, and you can them using each other just like Python. This is where the functional aspects can be outside the object oriented nature and hence defies the authors assumption that OO and functional programming don’t mix.

The author is also assuming functions and objects are on the same level. This isn’t necessarily the case either. I’m sure we have all seen one article or another that talks about objects using examples like circles/squares/shapes or cars/boats/James Bond boat-cars and dealing with things like multiple inheritance and the diamond problem. In these situations, we seem to have the solution in place when we reach the right structural hierarchy of classes. In contrived examples, this is often the case, but in the real world, we rarely are done when the class structure is defined.

The classes have to accomplish something, like sort some data, digest an electronic invoice, generate a PDF report, or some other operation. We have the ability to use functional programming for some of these tasks. For example, if we are consuming an electronic spreadsheet of data, it may perfectly fit into a foldLeft operation. Various fold calls can be imbedded in the different concrete subclasses of our hierarchy. We instantiate the object based on the invoice type and then we functionally digest the file. If there is commonly used functionality between each subclass, we can abstract it up the hierarchy. Our fold operation may tap some abstract functionality, like readingTheNextLine. This type of abstraction is what we do when we need to handle physical files as well as virtual ones. So we have crafted a functional construct connected to an object oriented one.

This example demonstrates that functional and object oriented structures can easily be mixed together, QED, defying the author’s assumptions. Because scala makes it easy to mix-and-match code together, it is strongly empowering the user. My favorite synopsis of scala is in the video interview of David Pollack on the FLOSS vidcast.

t

Fast forward to 04:00-04:45 to hear David Pollack give a 10,000-foot summary of the Lift web framework. But more importantly jump to 05:54, where he taps his head saying something like “scala makes it easy to capture what was in my mind’s eye.”

As Ted Neward described in a column, working on software is one of the hardest professions, because everything we deal with is purely abstract. Programming is hard! You can’t crack open a computer and point out the threads, the objects, the functions, or the actors. A computer is a series of electronic pulses that we are trying to manipulate through abstractions. Even theoretical physics must eventually bow to actual observations. Einstein’s theory of gravity and it’s implications for bending light wasn’t accepted by the academic community until a scientific team traveled to a far away location to measure stars during an eclipse.

The complexity of scala is governed by the problems we are solving, how well we can simplify the parameters into constructs that allow us to write a solution, and how well we can communicate with our team. I have never worked on a team where were all perfectly in sync on the same page. Someone was usually more seasoned, and one if not more of the people were green newbies. Saying that scala is too hard because everyone is on different levels is a red herring.

All programming languages suffer this problem. The worse thing you can do is isolate yourself and not have regular team meetings and communicate. The team needs to get together regularly and get on the same page. This takes active work, code reviews, team discussions, brown bag lunch seminars, and more. I hate to break it to you, but that is what teams do. Programming is hard, and we must help each other out, not abandon solutions.

Programming is hard – Part 1

NOTE: You can read Part 2 and Part 3 of this blog series as well.

I have read enough articles debating whether or not scala is hard, and having recently watched a YouTube video clip of a kid presenting his Game Of Life app, I just have to weigh in.

Coding in scala isn’t hard. Watch the clip and think about everything that is happening here. I can hear a 11-year-old talking about looking up neighbors, calculating how many items are nearby, creating new and empty grids, two-dimensional sequences, so on and so forth. This kid is tackling what is really hard: writing computer programs using fundamental constructs found in any language.

How many of you reading this actually started learning programming when you were in college? I doubt any of you did. Instead, you were probably some geek kid when you first dabbled. What did you use? BASIC, Pascal, Logo, Prolog, Smalltalk, something else? Did you know all the constructs at the time? Did you understand things like object oriented programming, design analysis, or refactoring? I’ll venture that you started out learning things like for loops, setting variables, and fiddling with making some sort of output, whether it was text on the screen or pushing some sprite around on a screen. (I miss commanding a turtle to move around with the pen down in Logo on an Apple IIe).

And this is where the rubber hits the road: programming is fundamentally about learning how to make that computer do what we want using a handful of concepts. The basics of programming don’t involve complex analysis nor designing fancy structures. When we start out building programs, it’s kind of like building a dog house. We learn just enough to nail together a few boards and see some quick results. We are really happy when Spot had a nice little home that we nailed together and painted, and can see it everyday in our backyard.

The more complex stuff like object oriented analysis, technical debt, code coverage, and all that other stuff are more elaborate techniques we pick up later when we realize that we can’t use our dog-house building techniques to construct a skyscraper. But whether we are building a dog house or a giant tower, we are still manipulating some simple elements to construct a wonderful structure.

Let me go back to my college days. I once worked at the Agricultural Research Service at Auburn University, and developed a real-time control system used to test tractor tires. I built a program that commands the hydraulics on how much pressure to put on the tire, and whether or not my program decided to stop was based on the sensor feedback I got. Basically, I got a sensor reading, calculated how far it was off from my target load, and sent a followup signal to increase pressure at a certain rate.

This is where I really learned an amazing lesson. The system had so many factors that contributed to what was the right amount of force to apply. For example, if you are very close to the target load to place on the tire, you only want to increase the load by a small amount, but if you are far away, you want to increase the rate that force was applied. The system had delays. The time span to read a sensor and calculate a delta took milliseconds, but in that span of time, the apparatus kept moving, so there was always a certain amount of error in making this calculation. But there was absolutely no value in trying to calculate all these factors or analyze every aspect of the apparatus. Instead, I had to run trial after trial, seeking out two numbers: the immediate correction factor and summing correction factor.

You see, the formular for a real time control system involves two key factors: the first is making corrections based on the current delta between where you are and where you want to be. You find the difference, multiply it by some constant, and then send the actuator signal accordingly. 10 milliseconds later, you calculate the difference again, multiply it by the same constant, and send an updated actuator signal. The bigger the constant, the faster the system moves…and the bigger it will overshoot the target. The smaller the constant, the slower it will reach the target…and the smoother the transition will be.

But all control systems have a resolution that results in lurching back and forth over the target. That is where you have to also sum up the deltas and multiply it by the second constant, which is usually much smaller than the first one. This compensates for when the system starts drifting from either side of the target. It may be a small fraction, but after a few rounds, this drift becomes big enough that a course correction is needed.

In case you aren’t catching this: no amount of analysis will tell you what these two constants need to be to get the most efficient, smooth approach to target. In fact, if you try to analyze instead of testing out and using it, you will waste a lot of time for nothing. Instead, it requires empirical test runs until it works right.

Software is no different. No amount of analysis of language constructs, assessments of built in libraries, nor deductions about complexities can substitute for using the language to build something and seeing how it performs. When I watch an 11-year-old kid take command of scala and build the Game of Life, that is a true empirical anecdote that tells me that scala isn’t too difficult.