Learning valuable lessons in the trenches of production

computer_coffeeA key turning point in my career was when I was able to take over a non-functional Java Swing app. I had worked for years on a mission critical, 24×7 system written in a now non-existent language and runtime. Frankly, I was a bit tired of it. I had managed a total of seven employees over the years. Then I found one who was perfect to take over. He didn’t suffer from the burn out I was approaching, and he had new ideas. So I put him in charge. When one of our other devs announced he was leaving the company, I asked my manager if I could take over his app. It was a tiny, Java Swing application that the target users simply didn’t use to complete their daily job.

I found out why. This application had been under development for two years. Six devs had worked on it. I heard how one program manager had complained in strong language how too much money had been invested into that app, and it STILL didn’t accomplish anything. Let’s see:

  • non-functional: check!
  • Java Swing (who wants THAT?): check!
  • the current developer would be gone and not available to answer questions in two weeks: check!

A 100% basket case. And I was hungry to take it over.

Before I lifted a finger, I pledged that I wouldn’t commit a line of code unless that code was covered by an automated test. A simple pledge, and the only thing I went in with. I didn’t know how difficult it would be to uphold such a commitment, let alone how gratifying its outcome would be.

Current state of things

The app was used by the finance team to ingest itemized invoices in the form of spreadsheets from our subcontractors. While we had over three hundred, there were about six different formats. The first test case I wrote involved reading in an Excel spreadsheet with something like five thousand rows of data, storing it’s contents in the database, and doing some other post processing. That process probably took a minute. I knew that down the road, this would get costly.

But it was through this painstaking process that I started weeding out what was needed and what was a waste of time. The code had a layer of abstraction around the spreadsheet that frankly was useless. It was like having the ability to abstractly fetch a column of data, only this wasn’t needed. So once I had wrapped invoice processing in a handful of test cases, I went in and started ripping this out.

And then I discovered other patterns and anti-patterns at play. I started ripping out other things. Simplifying. Reducing. The app was beginning to function. I was interacting with the users and had begun implementing features they needed. I had only been on the code for about three months when the finance team leader decided it was time for a demonstration to program management

Acceptance by management

I could do mission-critical-never-goes-down in my sleep. The side effect was nothing would faze me. I wanted to demo this app in front of managers that had seen this app in the past, and worried that too much money had been spent. It was important to make it work right.

I knew what would be perfect. I wrote a single, length JUnit test case that basically walked through the app like a movie script. I spent an entire month on this script. (I have since learned this would be viewed as an acceptance test). Every day, I walked through the script and tried to add one step. And another. And another.

What this script was doing was forcing me to decouple the Swing UI layer from all business logic. My automated movie script was essentially driving the app in a headless fashion, and it was so cool.

Then came the day of demo. I printed my movie script on paper, and walked into the conference room. I would indeed drive the GUI using the movie script. Essentially, it told me each and every step. And I had 100% it would work flawlessly.

Suffice it to say, management was awestruck. I didn’t care that this made me look good. I just loved that the power of test automation had allowed me to build a flawless demo. I also loved that all this test driven development was empowering me to rip out junk, rewrite flawed patterns, and basically Do The Right Thing. It was liberating and sold me on the power of TDD.

Sure my entire coverage was something like 60%. But something I learned was that 60% vs. 80% is a really worthless comparison. If the team is spending 99% of their time using the 60% that works, then it’s effective coverage.

Welcome Spring and other OSS tools

As I grew the app, I knew I had the ability to pull in open source tools as needed. I reached for Spring, something I had read about previously in “Spring: A Developer’s Notebook”. I started to use it, but thanks to my TDD approach, I only used the parts from this bag-o-tricks that I needed. The feeling was nothing less than glorious.

After about six months of work, things were sizzling. I was having weekly meetings with the users. They would describe features they wanted. I would write a test case, and sometimes have another released out that week, or even the very next day. They had never seen anything like that.

Less is more

One day, my manager called me into his office. By “manager”, I mean the guy that I used to share a cubicle with, write code side-by-side, and whom was a victim of his own success. I wasn’t worried. Just curious.

One of his fun jobs is that he has to count lines-of-code for every app once a month and present it as part of his monthly report to senior management. He showed me the graph of my app going back to before I had touched it, all the way up to today. It had risen steadily, peaked a few months ago, and began to go down.

“None of the others apps is doing this,” he said. “No one else has actually reduced their lines of code.”

I hadn’t realized this until now. I was actually shrinking the application, and yet it did more than ever. It was loaded with new features. But I had managed to yank out useless abstractions, features nobody used, and simplified its architecture. The net effect was less code despite having added gobs of automated test code.

This is when I discovered that less is truly more. And the only way to discover this is through test automation. I even learned that less testing is more. The test cases were built up on the flow of “clean database, ingest invoice into database, examine database”. At one time, my test suite took an hour and a half to run. When I had to throw away an entire day’s worth of changes because I hadn’t run the test suite and broken half the system, I paused and went to clean up the tests.

I honed things to get them back down to thirty minutes. A decent price to pay for a solid application.

Users and developers, working together

I reached a point where we had a “summit” of sorts between myself and the users. They had some new features they needed that really went off on a tagent. We met in a conference room, and I fired up PowerPoint. We captured each use case as a separate slide. They could understand that. I would then convert each slide into a separate test method, using the language of the slides in my code. We spent a week honing the slides so there was concurrence.

I then spent may be a couple weeks implementing the features. And we were done with that go around! I showed the actually coded tests to the user’s team leader. She wasn’t technically savvy. But she could easily read my automated test because it was written in the identical language of the slides.

When I listened to a recent episode of the Java Posse where they were discussing Domain Driven Design, I was connecting with so many points being made.

  • Common set of terms
  • Agreed upon vocabulary

I had brought the system to a point where the users had an active participation in what was built. I perceived myself not as the focal point the way I had in the past. Instead, the business these people were doing was the focal point, and I was the support guy merely making it possible with a bit of software.

The future

It was at this point, when things were humming along solidly, that I received my offer from SpringSource. I told the financial team leader that I would be departing in a couple weeks. He smiled and said, “well the timing couldn’t be better.”

I asked him what that meant. He indicated that my work had made things great for the users, but they still had a terrible inefficiency. The labor cost of a half dozen accounts receivable staff was way over budget. There are companies that specialize in this exact work and even get bonus checks for finding overcharges. And were cheaper. They were going to decommission my application and subcontract the work to someone else.

I chuckled. All my effort would be scrapped because doing this in house just wouldn’t work.

I wasn’t upset. Instead, I was excited! Maybe that sounds strange, but the past year had been like a playground where I got to test out a lot of ideas I had been reading about in books, blogs, and listened to in podcasts.

I had grown so much. And it really prepped me for my new position. I felt like the past year combined with my mission critical experience gave me a great set of skills. I was more excited than ever to join my new team.

Leave a Reply

Your email address will not be published. Required fields are marked *