Category Archives: scala

Mobilizing your website

I just spent two days adding a mobile front end to a web app a few months ago. It is a reference app called Rabbit Trader, an app that demonstrates our JMS on Rabbit module targeted for vFabric 5.3. To hear more about that module, you’ll have to wait a bit.

All I can say is that it has been a blast! Of course with any good progress there are always lessons learned. Here are the ones I want to discuss in this blog entry:

  • Don’t try to have any one page or template serve up both mobile and non-mobile UI.
  • A framework with continuous editing is a must.
  • Add things one piece at a time.
  • Test it on a mobile device.
  • Think about your favorite mobile site as a source of inspiration
I was working with the Play Framework 2.0 Scala, and had built a rich web app based off their chat app. It’s purpose is to push simulated stock trades out over a message broker, and then let the consumer threads display them in the chat channel. At the same time, users that are logged in can also chat amongst themselves.

I had the rich desktop interface put together pretty quickly, because it was already developed in the websocket-chat example. I decided to take a crack at using jQuery Mobile to add a mobile view. I had already seen one presentation last March at the DevNexus conference and also read a book as well. It looks pretty easy, so why not give it a shot?

Let’s walk through the things I learned while working on this.
Don’t try to have any one page or template serve up both mobile and non-mobile UI
At first, I tried to adjust the existing pages so I could pass in an argument like (mobile: Boolean), and simply have an if-then chunk of code in my templates. That seemed like it would reduce the chance of repeating myself with certain idioms to talk to the back end.
That turned out to be a real hassle. By adding extra arguments to the templates, it seemed to break the API that was setup, and really showed me that I didn’t fully understand Play’s template engine. Perhaps if I understand it inside and out, I could have solved that problem. But since it proved too much to handle, and was causing the normal web pages to not work either, I ended up throwing everything away at the end of the first day and starting from scratch on the second day.
Chalking it up to a learning experience, I started the second day with a clean slate, and instead built the front end without touching the existing pages. By using a single-page template for the login page and a multi-page template for the rest of the UI, I got things moving very quickly.
A framework with continuous editing is a must
Play offers the ability to run your app and then make code edits constantly. All you have to do is reload the page and the server recompiles and restarts things. Only on rare occasion did I have to shut things down. This made it much easier to experiment here and there.
Most of the current web frameworks offer this, like Grails, Play, Rails, Roo, Django, and CherryPy. The SpringSource Tool Suite comes with tc Server installed. You can deploy a Spring MVC app there, run tc Server, and as you make edits, they get synchronized. The lesson here is use it! The feedback is critical to building a usable app.
Add things one piece at a time

There is more than one occasion where I started coding too many things before testing the interface. The places where this would get me the most was when I would tweak the javascript. My app uses websockets to provide a publish-subscribe flow of data from the server to the client. The UI has to connect to a socket, and as events are pushed to the client, generate dynamic HTML to append to a div container.

Basically, all this was already written with Play’s websocket-chat example. But it was coded using Twitter’s bootstrap look-and-feel. I started with their layout of div containers, and tried to edit them, but it turned out simpler to simply start with a clean slate and build the mobile perspective from nothing one component at a time.
This also forced me to switch my mode of thinking from retooling the UI to building it clean. This also let me focus on using jQuery Mobile’s style of application. The more I focused on using jQuery Mobile’s style, the better my app looked.
And by adding one change at a time, I was more efficient at learning one idiom at a time, both jQuery Mobile as well as javascript/jQuery.
Test it on a mobile device
I did most testing on my laptop and just pointed my browser at localhost:9000/mobile. But nothing can take the place of looking at it on an iPhone. That tiny screen really shows you how precious real estate is.
In the rich frontend, it has a sidebar widget showing all the users that are connected to the channel. I had reconfigured it to show them at the bottom. When I looked at it on my phone, there was simply no room for that. I also decided that I didn’t really need that information, so I dropped it.
jQuery Mobile makes it a snap to put buttons anywhere and everywhere. It’s very easy to put a left and right button on the header. You can also add standard icons which make it look even better. I put a “Disconnect” button on the left and a “Compose” button on the right of the header, but when I viewed it on my phone, they took up way too much space, overwriting the header itself. So, I opted to drop the text and only show the icons.
This drove me to add a third button for information. I tried to put it at the top, but it turns out that headers only have room for two buttons, and I had used them up. I wasn’t about to add a navbar just for the sake of an info button. So I created a footer navbar with the info button. It displayed nicely, but the footer itself was as slim as the button. I added class=”ui-bar” and it added just a touch of padding, making it look perfect. Finally, I topped it off with data-position=”fixed” so it wouldn’t get pushed off the screen as new messages arrived from the websocket.
Creating another page for information was easy, since I was already coding the thing as a one-page app. It was critical to keep the page with the websocket handling javascript in the browser, and it also reduced the number of routes I had to add on the backend. From the app’s point of view, this three-page mobile front end was just one big HTML page. I just added another div data-role=”page” and wired it in. I then marked it as a dialog, making it nicely popup with an “Ok” button to close it and return back.
The rich UI had a textarea at the bottom so the user could type messages, and was wired to use return keystrokes to send the message into the websocket, so it would travel to all clients that were hooked in. This was a bit impractical when I tested it on my phone. So I moved the entry of data onto a separate “page” on my multi-page mobile app. I also switched the publishing of events to insert the latest message at the top of the string of messages, the same way that twitter does.
All in all, my phone provided an easy way to make sure my mobile experience was not only working but highly practical.
Think about your favorite mobile site as a source of inspiration
When I was thinking in my mind what the mobile view of things should be, I immediately thought of mobile.twitter.com. It has a top bar as well as secondary navbar. Bellow that is the string of data. I instantly started thinking of how to use this general flow of things, but customized to meet my requirements.
One thing that the rich interface included was marking posts from the owner with a light yellow background. They had a particular CSS class to assign. When I decided to add that, it was time for me to dig in and code some CSS myself.
The next thing I observed on twitter’s layout was how they put each tweet into a separate table structure. The username data is on one row while the message content is on another row. Then they have CSS classes to ensure consistent formatting. jQuery Mobile follows a minimalist approach, so while they can lay things out, they don’t style the content with borders. I fiddled with CSS borders on each entry. I quickly decided to apply every entry with the following CSS class:
.message-container {
 border-left: 1px solid black;
 border-right: 1px solid black;
 border-bottom: 1px solid black;
 width: 100%;
}

Then I coded another CSS class to create the “top” border, and used jQuery to first remove the top class from whatever element had it, and add it to the new element I was creating that would get prepended at the top.

I might replace black with something like grey, but for now, it looks great. And seeing it dynamically update on the iPhone is fantastic. Combined with being able to asked Siri to actually write my messages makes for a great mobile experience.

Conclusions

Simply put, jQuery Mobile makes it incredibly easy to layout, style, and wire up a mobile front end. Given today’s explosion of mobile devices, there is simply no reason to not develop a suitable front end. It requires a little bit more than just adding some extra classes to your screen. But in the end, the people that see your handy work will be blown away. I certainly am.

FYI: All data shown on the screens is mock data and does not represent real stock data.

When testing really matters

This past Tuesday, we had James Ward, principal developer advocate for Heroku, give a presentation on the Play Framework, scala/java, and Heroku at the Nashville JUG. Suffice it to say, it was a really awesome presentation. This was far from a sales pitch, and more like a lets-get-our-hands on this framework and build a web app as fast as possible.

Sadly, I had to leave early, but I was hooked on poking my nose into Play. I have been fiddling with a small toy app using Scalatra. It is really nice where you focus on building the routes and then stir in some templates, with a little scala in there to make things nice and succinct. But I had been running into issues repeatedly with the testing. I decided from the get-go, that I would make it a test-driven project, and had built up automated testing quite nicely. I built all the REST points and had everything working with a nice stubbed out data source with future plans to replace it with a NoSQL database.
Scalatra has some nice test structures. You can either use specs or ScalaTest to check everything out. But something that nagged me for some time was how my tests would periodically fail every 3rd or 4th time. Sometimes more often than that. But since they would occasionally pass, I knew that everything worked right. It was probably just some race condition that had to do with the jetty container it used to run the tests.
When I tried to swap out my in-memory maps with MongoDB, I couldn’t get the tests to pass ever. Comment out MongoDB code; tests pass. Bring back ’em on line; fail again. It drove me nuts. I tried lots of things. This was when I saw the Play demo. Seeing that it was heavily route-based, I thought it probably wouldn’t be hard to transfer my existing test suite to a Play app, modify any API calls, and use this set of tests to rebuild the REST points in Play. Well, I was able to do it in just a few hours. And guess what? Now there are no failures…EVER!
Why did Play succeed where Scalatra failed me? I think it’s because Play has different levels of testing. They build in helpers that let you test from any angle, meaning you don’t have to test the runnable container. Instead, they offer a lot of useful angles to isolate and test things out:
  • test individual units
  • test templates
  • test controllers
  • test routes (see http://www.playframework.org/documentation/2.0.2/ScalaFunctionalTest)
  • test the whole thing with an HTTP server
  • test from a browser with Selenium
All these things give you the option to easily grab your existing test suite no matter how it was structured, carry it over to Play, and rebuild you app back up, one test at a time, until everything passes. Then, with your test harness in place, go back and start refactoring as you see fit!
Right now, the web pages look pretty ugly. They are the basics with no styling at all. There aren’t even links to connect the pages together. That is because I am building this app so that each REST point does what it is supposed to. Then I can start putting a nice look-and-feel on it. I’m already excited because it appears that Play has some helpers to plugin Twitter’s bootstrap HTML/CSS/JavaScript library, something I was planning on when I would get around to the UI.
Given all this, Play has really made it easy to put together a REST-ful web app using Scala, my primary goal. I hope you have similar experiences.

Scala makes it easy to evaluate financial problems

Over the past couple of years, I have been innately fascinated by the emergence of scala. Discovering it’s incredible power of type inference, a very extensive collections API, pattern matching, and many other things have been very alluring.

Recently, I had a question pop into my head regarding financial data. You see, I often hear people cite things like “the average performance of the S&P 500 is 12%”. What? That is crazy. I know that isn’t true, and betting your retirement on it is not good. Where does this come from? I saw a short video clip where two financial advisor explained that many people use the arithmetic mean instead of the geometric mean to calculate this value. But I’m getting ahead of myself.

I went and tracked down a website that listed the performance of the S&P 500 back to 1951. I grabbed the numbers from 2001-2010, and punched them into a spreadsheet. Using SUM, I was easily able to calculate the arithmetic mean. Then I tried to calculate the geometric mean. This meant taking all those percents and multiplying them together. Guess what? Neither LibreOffice nor Google Docs spreadsheet have a MULTIPLY nor a PRODUCT function. Well, at least one that handles a list instead of two values. As my mind wandered away from spreadsheets and into software solutions, I realized this was the perfect thing to write a tiny scala app with!

First, we need to create a simple app. We do this by extending scala’s App trait (NOT the dated Application trait).

Next, let’s load up the data using scala’s List function. In this case, we are storing a tuple using () notation, containing the year and the relative change.

Next, we can write a foldLeft to start with 0.0, and then add each entry’s second item using the ._2 method. At the tail end of our foldLeft, we divide it by the size of the sequence.

See how easy this function was to write? We can tabulate everything, or just a slice but the simplicity of this function shows why many people use it to write simple financial advice. But that’s not enough. For a more accurate evaluation, we need to figure out the geometric mean.

First, we need to convert one of these relative percents into an absolute multiplier. Since I like to read outputs in relative percent format, we will need another function to convert back.

What could be a single function, I decided to break out into two. It’s nice to know total growth as well as average growth per year (also called annualized growth). Given that function, we just need to take the nth root based on the length of the list.

With these functions, it is easy to run our entire list of historical data and find the average growth of the stock market. But that isn’t all I wanted to know. I really wanted to see what would happen if my money was invested in an EIUL, meaning that in negative years, growth would be capped at 0.0%, and during booms, growth would be capped at 15%.
I wrote one solution, but it was clunky. I got some help on stackoverflow, and instead coded a way to basically sort each year’s performance against List(0.0, 15.0), and pick the one in the middle. For negative years, 0.0 is in the middle. For big booms, 15.0 is in the middle. For everything else, the stock market number itself is in the middle.

A cornerstone of functional programming is working with lists of data, and transforming them based on your needs. In this case, we need a function that applies the conversion function above to an list of stock data. We can do this easily with scala’s map function.

So, now I can evaluate the entire performance of my money if it was invested in some 500 index fund and compare it with the growth potential of putting that in an EIUL. And it was pretty simple! Imagine what this would have taken to write in Java. It would be clunky, hard to decipher, and probably littered with many more bugs.

I have heard it said that scala makes hard stuff easy and impossible stuff reachable. So I pushed myself. I remember speaking with one of my financial advisors (I have several to provide multiple sources of input to my plans) and he was talking about the average rolling performance of 15 year windows over the past 30 years. Could I write a little more code, and do that myself? I think you know the answer. Start at the first entry of the list, grab n items, then step to the next one and do the same. Before I could write this myself, I checked scala’s collections API only to find a sliding method call to already do this for me.

To have this analysis carry some statistical weight, let’s write a function to calculate standard deviation.

Given all this, it’s time to crank out some output code. Let’s analyze the performance of the S&P 500 and our EIUL, comparing all 10-year, 15-year, 20-year, 25-year, and 30-year intervals.

Let’s wrap it up with a closing brace.

Let’s run it!

This shows that EIULs should pretty consistently beat any index fund based on the S&P 500. It’s possible, due to the variance, that the S&P 500 can beat an EIUL. But the standard deviation of the EIULs are much slimmer showing you a more consistent expected return on investment. S&P 500 has much wider variability meaning you can strike a much better rate, or be passed up my inflation.

And what’s more important: neither of these solutions will make you rich. To do that, you need to be consistently hitting double digit returns. The only type of investment vehicle that has evidence of doing that is either real estate or something with higher risk, such as running your own business. Funds and EIULs aren’t made to make you rich. EIULs, however, are great ways to store money you make through your investment endeavors and later pay a back to you in a consistent, predictable, and TAX FREE manner.

UPDATE (5/18/2012): Since my blog entry was cross posted at Uncommon Financial Wisdom, I have updated the scala app to also find the minimums and maximums for each window. See https://github.com/gregturn/finance for the code updates as well as the results displayed in the README file. Visit David Shafer’s blog and give him a call if you want to talk to a real wealth building expert.

I am not a licensed financial advisor nor an insurance agent, and cannot give out financial advice. This is strictly wealth building opinion and should be treated as such.

Scala is EJB2? I don’t think so

I just read Stephen Colebourne’s blog entry that is circulating the twittersphere. I’m about to post some comments, so I guess this means that I don’t “tolerate dissent.”

First of all, please don’t pull out the infamous “(0 /: l)(_ + _)” argument okay? I get it that not everyone understands foldLeft’s, but it’s not that hard!?! I hadn’t done this type of thing since college, and after about 5 minutes of reading, I grokked it pretty well. I went on to give a presentation on functional programming at NJUG and even used this as one example of programming that is an option in your toolbox. I can’t really say much more other than a) don’t use it or b) read In Defense of (0 /: l)(_ + _) by Ricky Clarkson.

I think someone else already captured my opinion regarding discussion about loving/hating scala: “In my experience this is completely inaccurate, dissent is constant in the community. What is not tolerated is stupidity, unthink or fud. The Scala community contains amongst it a very bright set of individuals who do not let even confirmed Scala enthusiasts get away with the slightest exaggeration and will passionately debate many a sacred cow.” —jed.

I just don’t understand the “scala is complex” idea. We have to solve complex problems! Get it? Things aren’t easy. We are having to evaluate frameworks and tools to build systems. So many new, hot things are being built. Why was Ruby On Rails so popular? Because it enabled us to build e-commerce sites a lot faster than before. Same thing for Grails and Spring Roo. Neither Java nor Ruby by itself was enough for people to do their jobs. Well, I shouldn’t say that. Lots of people built lots of systems using Java and Ruby, but once these rapid app development tools came out, people ate them up fast! It’s because we have tough jobs and anything than can bend the curve helps us out. The same can be said for scala. We are able to bend the curve and write more functionality that captures what we are thinking with it.

Love it or hate it, I really liked PERL. I built mean systems using it, and that was because it was the first language I met that made regular expressions incredibly accessible. Today, I don’t think of using it, because Python has what PERL had and more. If I didn’t have anything like Python, Ruby, or Groovy to do pattern matching, then I wouldn’t hesitate to grab PERL. Now as scala emerges, it’s ability to do algebraic type matching is incredible. I think it sets a new standard in what other languages should offer in pattern matching abilities. So, just because I can write some bad patterns, I should chuck it and move on? I don’t think so.

Some of the criticisms are dead on and point out where effort must be heavily invested. Lack of test suite is critical in my book. Instead of dumping scala due to a lack of a test suite, how about bolstering it as the thing to improve its quality? How many systems have you dropped or discarded because they just didn’t have enough automated testing? I hope not too many, because it’s only recently that I have seen automated testing really take off as the way to go, despite years of preaching.

To me, learning new languages has always been part of the fun of programming. I studied many languages, worked on lots of projects using several including BASIC, C, C++, Assembly, Ada, Tcl, Tk, Expect, Bash, SQL, Java, XML, JavaScript, Scala, Groovy, Clojure, Prolog, and others. Each of these has some amazing aspects as well as funny quirks. It has always been exciting to move to a new technology and discovered what I can really do. As I’ve learned new languages and new paradigms, I feel it has made me a better programmer because I am more rounded. I’m not confined to my provincial thinking that was the result of tinkering with BASIC as a kid. Now I think of big and complex concepts and immediately start chopping them down to small solutions in my mind before my fingers even hit the keyboard. Scala is another one I would be chomping at the bit to work on for a full blown project.

Programming is hard – Part 3

In this latest installment of Programming is hard, let’s talk about abstractions. (For previous readings, see Part 1 and Part 2).

I read a column by Ted Neward where he points out how software development is unlike any other industry. We have to deal with abstractions ALL the TIME. As he stated so eloquently, you can’t just open the computer and point out where the threads, locks, and classes are. Those are just abstractions we invented to avoid dealing in low level assembly code.

But it doesn’t stop there. Assembly code is another abstraction. I studied Computer Engineering at Auburn University, not Computer Science. Not only did I study programming, but I learned how to build circuits, flip-flops, and use logical gates to create certain outputs. Computer chips are basically a bunch of electrical impulses. We want to get the customer’s spreadsheet of data to translate into a bunch of electrical inputs to a plethora of gates, and then convert the output into some web page. The funny part is that we never deal with the real, physical nature of the computer chips.

This is why software is so different and consequently so hard. Even theoretical physics, thought by many to be the most complex and bizarre field of study, have to bow to real observations at some point before proceeding further.

Einstein wrote a few papers about his theory of how gravity operated. He came up with an assumptive model (elastic space influenced by the indentation of mass), hammered out the mathematics of that model, and derived a prospective outcome. His outcome was shocking, because it suggested that Newton was wrong. Newton’s theory of gravity had stood for 400 years with many observations that made it rock solid. The scientific community wouldn’t listen to Einstein without observed evidence. By “wouldn’t listen,” I mean that his academic career may have been cut short because no one else would proceed to build their research on top of his theory if it couldn’t be proven by some observation.

In a nutshell, Einstein predicted that the light rays from distant stars would be warped when passing around our sun by a certain amount. His calculations produced a different result than what Newton’s laws predicted. The only way to prove this was to observe the positions of these distant stars during a solar eclipse when they weren’t washed out by the sun. An exploration team traveled to deep jungles in order to verify this, and they succeeded. The proverbial inventor of the theory of gravity, Newton, had been corrected, and Einstein is now recognized as one of the greatest physicists of all time.

In software, we don’t have to observe the real, physical electrical signales generated by our code. Instead, we only have to measure if it provides our users with the output they want. This translates to spreadsheets, web pages, pages, or other “real things” that tell the customer their system worked. Why do you think so many you speak to get a glazed look if you talk about anything below the surface of software?

The ophthalmologist I grew up visiting retired at a few years ago, went back to school and got a law degree just for fun. He then tried computer science and gave up because it was too hard. This guy had a medical degree and law degree, but computers were too much! Heck, I knew someone in high school that made it to final round of the televised teen tournament of Jeopardy, but when we were both in computer science class, he needed me to tutor him because he just couldn’t get it. Programming is hard!

Why do you think multi-threaded programming is so hard? It’s because there are so many ways to implement it, and we only have the theories and abstractions invented so far. It’s not a real, physical problem, but instead an abstractive problem we need to tackle. I have read 20-year-old text books that talk about spin locks, semaphores, and other things to control the system while switching threads. In my research paper I wrote in grad school, I talked about real time systems and how different operating systems provides the means for different threads to interrupt and take control. Merely switching from one thread to another has different solutions. So it’s no surprise that we are still debating whether to lock state and switch threads, pass messages between actors in order to isolate/reduce state in the system, or use some sort of transactional software memory to move state around.

Functional programming is offering some new ideas on how to isolate state to reduce complexity, and thus make it easier for threads to switch at will, reducing how hard it is for us to handle things. But functional programming has been around for 50 years, so is it really new? FP isn’t new, but the problems we are solving today are vastly more complex than what people needed 50 years ago. ENIAC is credited as the first electronic computer, and it one of it’s biggest tasks was to calculate ballistic firing tables for the military. Multi-threaded programming wasn’t a requirement.

Today we are dealing with data warehouses that host terabytes of information. Facebook recently moved several pedabytes of data from one location to another, because they needed more space. Heck, the “settled” concept of using SQL to store and retrieve data is up for grabs when it comes to managing “Big Data”. Our problems are growing in size and complexity, and it’s up to us to craft better and better paradigms of writing code. That is why things like scala and clojure open new doors to help us solve this tough issue of programming. We have to keep investigating and figure out what is more effective, and who says there is only one solution?

Programming is hard – Part 2

NOTE: You can also read Part 3 of this blog series.

The debate about the complexity of scala ensues. As I previously wrote, programming is hard! Trying to conclude how easy or hard scala is based on language constructs isn’t feasible.

In a really active thread on the LinkedIn Scala Enthusiasts group, someone posted:

…encapsulation is the most important concept in OO and plays no role in Functnal Programming. So these 2 paradigms are not naturally combinable. I think designing fully and only Functional language compilable to JVM bytecode with ability to call methods of Java classes/instances would be much more natural combination.

I’m sorry but this point is totally wrong. The first part where the author suggests that object oriented and functional programming concepts are not combinable is based on a totally false premise.

This can be demonstrated by the fact that languages like Python allow you to use classes and functions all in the same module. You can model things with classes, but capture important operations used by your objects in functions that aren’t part of your class structure. The class methods can invoke functions and the functions can use the class structures in a symbiotic way. This is because not every problem requires the ceremony of classes. In simple, contrived examples, I usually don’t need to chop things up like this, but when I’m working on complex, real systems I like having options available, and sometimes mixing functions and classes together is alright. Think of it like nouns and verbs (read this side article, it’s awesome).

Scala supports this too by letting you put functions inside objects, essentially namespacing them. You can build an entire module as a collection of functions with no classes at all. Define some classes, and you can them using each other just like Python. This is where the functional aspects can be outside the object oriented nature and hence defies the authors assumption that OO and functional programming don’t mix.

The author is also assuming functions and objects are on the same level. This isn’t necessarily the case either. I’m sure we have all seen one article or another that talks about objects using examples like circles/squares/shapes or cars/boats/James Bond boat-cars and dealing with things like multiple inheritance and the diamond problem. In these situations, we seem to have the solution in place when we reach the right structural hierarchy of classes. In contrived examples, this is often the case, but in the real world, we rarely are done when the class structure is defined.

The classes have to accomplish something, like sort some data, digest an electronic invoice, generate a PDF report, or some other operation. We have the ability to use functional programming for some of these tasks. For example, if we are consuming an electronic spreadsheet of data, it may perfectly fit into a foldLeft operation. Various fold calls can be imbedded in the different concrete subclasses of our hierarchy. We instantiate the object based on the invoice type and then we functionally digest the file. If there is commonly used functionality between each subclass, we can abstract it up the hierarchy. Our fold operation may tap some abstract functionality, like readingTheNextLine. This type of abstraction is what we do when we need to handle physical files as well as virtual ones. So we have crafted a functional construct connected to an object oriented one.

This example demonstrates that functional and object oriented structures can easily be mixed together, QED, defying the author’s assumptions. Because scala makes it easy to mix-and-match code together, it is strongly empowering the user. My favorite synopsis of scala is in the video interview of David Pollack on the FLOSS vidcast.

t

Fast forward to 04:00-04:45 to hear David Pollack give a 10,000-foot summary of the Lift web framework. But more importantly jump to 05:54, where he taps his head saying something like “scala makes it easy to capture what was in my mind’s eye.”

As Ted Neward described in a column, working on software is one of the hardest professions, because everything we deal with is purely abstract. Programming is hard! You can’t crack open a computer and point out the threads, the objects, the functions, or the actors. A computer is a series of electronic pulses that we are trying to manipulate through abstractions. Even theoretical physics must eventually bow to actual observations. Einstein’s theory of gravity and it’s implications for bending light wasn’t accepted by the academic community until a scientific team traveled to a far away location to measure stars during an eclipse.

The complexity of scala is governed by the problems we are solving, how well we can simplify the parameters into constructs that allow us to write a solution, and how well we can communicate with our team. I have never worked on a team where were all perfectly in sync on the same page. Someone was usually more seasoned, and one if not more of the people were green newbies. Saying that scala is too hard because everyone is on different levels is a red herring.

All programming languages suffer this problem. The worse thing you can do is isolate yourself and not have regular team meetings and communicate. The team needs to get together regularly and get on the same page. This takes active work, code reviews, team discussions, brown bag lunch seminars, and more. I hate to break it to you, but that is what teams do. Programming is hard, and we must help each other out, not abandon solutions.

Programming is hard – Part 1

NOTE: You can read Part 2 and Part 3 of this blog series as well.

I have read enough articles debating whether or not scala is hard, and having recently watched a YouTube video clip of a kid presenting his Game Of Life app, I just have to weigh in.

Coding in scala isn’t hard. Watch the clip and think about everything that is happening here. I can hear a 11-year-old talking about looking up neighbors, calculating how many items are nearby, creating new and empty grids, two-dimensional sequences, so on and so forth. This kid is tackling what is really hard: writing computer programs using fundamental constructs found in any language.

How many of you reading this actually started learning programming when you were in college? I doubt any of you did. Instead, you were probably some geek kid when you first dabbled. What did you use? BASIC, Pascal, Logo, Prolog, Smalltalk, something else? Did you know all the constructs at the time? Did you understand things like object oriented programming, design analysis, or refactoring? I’ll venture that you started out learning things like for loops, setting variables, and fiddling with making some sort of output, whether it was text on the screen or pushing some sprite around on a screen. (I miss commanding a turtle to move around with the pen down in Logo on an Apple IIe).

And this is where the rubber hits the road: programming is fundamentally about learning how to make that computer do what we want using a handful of concepts. The basics of programming don’t involve complex analysis nor designing fancy structures. When we start out building programs, it’s kind of like building a dog house. We learn just enough to nail together a few boards and see some quick results. We are really happy when Spot had a nice little home that we nailed together and painted, and can see it everyday in our backyard.

The more complex stuff like object oriented analysis, technical debt, code coverage, and all that other stuff are more elaborate techniques we pick up later when we realize that we can’t use our dog-house building techniques to construct a skyscraper. But whether we are building a dog house or a giant tower, we are still manipulating some simple elements to construct a wonderful structure.

Let me go back to my college days. I once worked at the Agricultural Research Service at Auburn University, and developed a real-time control system used to test tractor tires. I built a program that commands the hydraulics on how much pressure to put on the tire, and whether or not my program decided to stop was based on the sensor feedback I got. Basically, I got a sensor reading, calculated how far it was off from my target load, and sent a followup signal to increase pressure at a certain rate.

This is where I really learned an amazing lesson. The system had so many factors that contributed to what was the right amount of force to apply. For example, if you are very close to the target load to place on the tire, you only want to increase the load by a small amount, but if you are far away, you want to increase the rate that force was applied. The system had delays. The time span to read a sensor and calculate a delta took milliseconds, but in that span of time, the apparatus kept moving, so there was always a certain amount of error in making this calculation. But there was absolutely no value in trying to calculate all these factors or analyze every aspect of the apparatus. Instead, I had to run trial after trial, seeking out two numbers: the immediate correction factor and summing correction factor.

You see, the formular for a real time control system involves two key factors: the first is making corrections based on the current delta between where you are and where you want to be. You find the difference, multiply it by some constant, and then send the actuator signal accordingly. 10 milliseconds later, you calculate the difference again, multiply it by the same constant, and send an updated actuator signal. The bigger the constant, the faster the system moves…and the bigger it will overshoot the target. The smaller the constant, the slower it will reach the target…and the smoother the transition will be.

But all control systems have a resolution that results in lurching back and forth over the target. That is where you have to also sum up the deltas and multiply it by the second constant, which is usually much smaller than the first one. This compensates for when the system starts drifting from either side of the target. It may be a small fraction, but after a few rounds, this drift becomes big enough that a course correction is needed.

In case you aren’t catching this: no amount of analysis will tell you what these two constants need to be to get the most efficient, smooth approach to target. In fact, if you try to analyze instead of testing out and using it, you will waste a lot of time for nothing. Instead, it requires empirical test runs until it works right.

Software is no different. No amount of analysis of language constructs, assessments of built in libraries, nor deductions about complexities can substitute for using the language to build something and seeing how it performs. When I watch an 11-year-old kid take command of scala and build the Game of Life, that is a true empirical anecdote that tells me that scala isn’t too difficult.

Please don’t say “don’t do that”

I have been digging for a couple of days on how to build a Scala List inside Java. The blog entries are hard to find. Most talk about converting a java.util.List into a Scala List from inside Scala. That is not what I’m looking for, but what really infuriated me was reading a blog entry that told me to just not do it.

I’ll show you how to do what you want in a second, but first, and importantly, I need to explain that what you actually want to do is NOT convert a Java List to a Scala List. Why? It’s because, despite the name similarity, they’re actually very different things…http://grahamhackingscala.blogspot.com/2010/02/how-to-convert-java-list-to-scala-list.html

That is really annoying. Please don’t tell me what to do and not do, okay?

  • I know that there are gaping differences between java.util.List’s interface scala.collections.immutable.List’s concrete implementation.
  • I know that they are backed by different structures. 

That doesn’t change the fact that to embed Scala’s compiler into my Java project, I need to access scala.tools.nsc.Settings which requires that I enter certain parameters as a scala.collections.immutable.List.

This was really annoying, and sounded very similar to the answer I often see when people are seeking help on writing multi-threaded coding in Python. “Don’t write multi-threaded. You are probably doing it wrong.” Sorry, but eventually, most people have to write some multi-threaded code. That answer in the sounds a lot like saying “you will hit the GIL and have all kinds of problems, because Python wasn’t built for multi-threading. And we aren’t removing the GIL.” I find that approach equally annoying. It’s one reason I watch the blogs from PyPy, since it seems the only viable way to remove the GIL from Python.

To cut to the chase, it’s not that hard to build an immutable list in Java. Look below.

The key part is Nil.$colon$colon(“dce”). That is the same as saying “dce” :: Nil in pure Scala. The whole block is showing how to embed the Scala compiler (2.9.1) inside Java, configured with –usejavacp -Ystop-after:dce -debug, and then compiling a string of code (not a file).

All you need to remember is that 1 :: 2 :: 3 :: Nil translates to Nil.::(3.::(2.::(1))). Given that, you can probably figure out many Scala interop issues.