Learning valuable lessons in the trenches of production

computer_coffeeA key turning point in my career was when I was able to take over a non-functional Java Swing app. I had worked for years on a mission critical, 24×7 system written in a now non-existent language and runtime. Frankly, I was a bit tired of it. I had managed a total of seven employees over the years. Then I found one who was perfect to take over. He didn’t suffer from the burn out I was approaching, and he had new ideas. So I put him in charge. When one of our other devs announced he was leaving the company, I asked my manager if I could take over his app. It was a tiny, Java Swing application that the target users simply didn’t use to complete their daily job.

I found out why. This application had been under development for two years. Six devs had worked on it. I heard how one program manager had complained in strong language how too much money had been invested into that app, and it STILL didn’t accomplish anything. Let’s see:

  • non-functional: check!
  • Java Swing (who wants THAT?): check!
  • the current developer would be gone and not available to answer questions in two weeks: check!

A 100% basket case. And I was hungry to take it over.

Before I lifted a finger, I pledged that I wouldn’t commit a line of code unless that code was covered by an automated test. A simple pledge, and the only thing I went in with. I didn’t know how difficult it would be to uphold such a commitment, let alone how gratifying its outcome would be.

Current state of things

The app was used by the finance team to ingest itemized invoices in the form of spreadsheets from our subcontractors. While we had over three hundred, there were about six different formats. The first test case I wrote involved reading in an Excel spreadsheet with something like five thousand rows of data, storing it’s contents in the database, and doing some other post processing. That process probably took a minute. I knew that down the road, this would get costly.

But it was through this painstaking process that I started weeding out what was needed and what was a waste of time. The code had a layer of abstraction around the spreadsheet that frankly was useless. It was like having the ability to abstractly fetch a column of data, only this wasn’t needed. So once I had wrapped invoice processing in a handful of test cases, I went in and started ripping this out.

And then I discovered other patterns and anti-patterns at play. I started ripping out other things. Simplifying. Reducing. The app was beginning to function. I was interacting with the users and had begun implementing features they needed. I had only been on the code for about three months when the finance team leader decided it was time for a demonstration to program management

Acceptance by management

I could do mission-critical-never-goes-down in my sleep. The side effect was nothing would faze me. I wanted to demo this app in front of managers that had seen this app in the past, and worried that too much money had been spent. It was important to make it work right.

I knew what would be perfect. I wrote a single, length JUnit test case that basically walked through the app like a movie script. I spent an entire month on this script. (I have since learned this would be viewed as an acceptance test). Every day, I walked through the script and tried to add one step. And another. And another.

What this script was doing was forcing me to decouple the Swing UI layer from all business logic. My automated movie script was essentially driving the app in a headless fashion, and it was so cool.

Then came the day of demo. I printed my movie script on paper, and walked into the conference room. I would indeed drive the GUI using the movie script. Essentially, it told me each and every step. And I had 100% it would work flawlessly.

Suffice it to say, management was awestruck. I didn’t care that this made me look good. I just loved that the power of test automation had allowed me to build a flawless demo. I also loved that all this test driven development was empowering me to rip out junk, rewrite flawed patterns, and basically Do The Right Thing. It was liberating and sold me on the power of TDD.

Sure my entire coverage was something like 60%. But something I learned was that 60% vs. 80% is a really worthless comparison. If the team is spending 99% of their time using the 60% that works, then it’s effective coverage.

Welcome Spring and other OSS tools

As I grew the app, I knew I had the ability to pull in open source tools as needed. I reached for Spring, something I had read about previously in “Spring: A Developer’s Notebook”. I started to use it, but thanks to my TDD approach, I only used the parts from this bag-o-tricks that I needed. The feeling was nothing less than glorious.

After about six months of work, things were sizzling. I was having weekly meetings with the users. They would describe features they wanted. I would write a test case, and sometimes have another released out that week, or even the very next day. They had never seen anything like that.

Less is more

One day, my manager called me into his office. By “manager”, I mean the guy that I used to share a cubicle with, write code side-by-side, and whom was a victim of his own success. I wasn’t worried. Just curious.

One of his fun jobs is that he has to count lines-of-code for every app once a month and present it as part of his monthly report to senior management. He showed me the graph of my app going back to before I had touched it, all the way up to today. It had risen steadily, peaked a few months ago, and began to go down.

“None of the others apps is doing this,” he said. “No one else has actually reduced their lines of code.”

I hadn’t realized this until now. I was actually shrinking the application, and yet it did more than ever. It was loaded with new features. But I had managed to yank out useless abstractions, features nobody used, and simplified its architecture. The net effect was less code despite having added gobs of automated test code.

This is when I discovered that less is truly more. And the only way to discover this is through test automation. I even learned that less testing is more. The test cases were built up on the flow of “clean database, ingest invoice into database, examine database”. At one time, my test suite took an hour and a half to run. When I had to throw away an entire day’s worth of changes because I hadn’t run the test suite and broken half the system, I paused and went to clean up the tests.

I honed things to get them back down to thirty minutes. A decent price to pay for a solid application.

Users and developers, working together

I reached a point where we had a “summit” of sorts between myself and the users. They had some new features they needed that really went off on a tagent. We met in a conference room, and I fired up PowerPoint. We captured each use case as a separate slide. They could understand that. I would then convert each slide into a separate test method, using the language of the slides in my code. We spent a week honing the slides so there was concurrence.

I then spent may be a couple weeks implementing the features. And we were done with that go around! I showed the actually coded tests to the user’s team leader. She wasn’t technically savvy. But she could easily read my automated test because it was written in the identical language of the slides.

When I listened to a recent episode of the Java Posse where they were discussing Domain Driven Design, I was connecting with so many points being made.

  • Common set of terms
  • Agreed upon vocabulary

I had brought the system to a point where the users had an active participation in what was built. I perceived myself not as the focal point the way I had in the past. Instead, the business these people were doing was the focal point, and I was the support guy merely making it possible with a bit of software.

The future

It was at this point, when things were humming along solidly, that I received my offer from SpringSource. I told the financial team leader that I would be departing in a couple weeks. He smiled and said, “well the timing couldn’t be better.”

I asked him what that meant. He indicated that my work had made things great for the users, but they still had a terrible inefficiency. The labor cost of a half dozen accounts receivable staff was way over budget. There are companies that specialize in this exact work and even get bonus checks for finding overcharges. And were cheaper. They were going to decommission my application and subcontract the work to someone else.

I chuckled. All my effort would be scrapped because doing this in house just wouldn’t work.

I wasn’t upset. Instead, I was excited! Maybe that sounds strange, but the past year had been like a playground where I got to test out a lot of ideas I had been reading about in books, blogs, and listened to in podcasts.

I had grown so much. And it really prepped me for my new position. I felt like the past year combined with my mission critical experience gave me a great set of skills. I was more excited than ever to join my new team.

Old public keys revoked; new keys generated

heartbleedRight now, I’ve added a widget to the right with some key links. Due to the heartbleed bug that has compromised an estimated 500,000 servers, most secrets are considered unsafe. Hence, I have revoked all past gpg keys and generated a new one.

I would sign the new key with the old one to maintain my trust linkage with the others, but that isn’t safe. Basically, I have to rebuild from scratch. I joined the strong set web of trust years ago. Now I need to email some of the people I knew that were in it, and see if we can sign the new keys.

If you have a public-private key pair, I suggest you do the same.

Scripting does the trick!

Last night I finally googled around and found http://python-wordpress-xmlrpc.readthedocs.org/en/latest/. This library uses WordPress’s XML-RPC API hook to let you connect, fetch content, and make edits. It was fabulous!

Having migrated from Blogger, I still had a huge number of links that were pointed at Picasaweb, where I have images stored. I had wanted to somehow update all my old blog posts and point them at the images I pulled down from Picasaweb and uploaded here.

This library was perfect. Their intro docs show you the basics of how to get started. It wasn’t long before I figured out how to scan every post, look for particular patterns I wanted to clean up, and then replace them properly. For example, my blog used to be greg-turnquist.blogspot.com. Several of my posts were still linked as such. Blogger kindly would forward uses hitting those links, and I left sufficient settings behind so they would keep getting redirected here. But to finally erase that old domain from all cross posted blog entries was perfect.

The other thing that I need to wrap up is how I still have some blog entries that hyperlink to the old label system of blogger (/search/labels/foobar). In WordPress, I use /category/foobar, so I need to scan for them all and make updates. I also need to replace things like real%20estate with real-estate.

The library even had the ability to pull down information on every image uploaded to this site. I naturally checked the name of each image here and on Blogger, and they all have the same root filename. I just needed to update the link and URI path.

To top things off, as usual, I dug up Python’s formidable re library. I always have to pull up the reference docs. Something about regular expressions always requires me to reboot my brain on using it. I am quite skilled and have been writing regular expressions since I learned PERL probably twenty years ago. But to really sink your teeth into a problem, write the correct pattern, and extract the groups through their API…well, let’s just say it doesn’t stick to me quite like certain other things. No doubt, this site helped me craft the pattern I needed.

After all was said and done, I finally pulled the trigger and issued a handful of edits. In no time, I had updated both this blog and my financial one, ensuring that I was now using local images.

Having a ton of fun with Spring Data REST + Javascript

Today has been 100% fun coding with Spring Data #REST, javascript, and pics of cats. :)https://twitter.com/gregturn/status/448219842056052736

Since last fall, I have shifted to working full time on Spring Data REST. This is an amazing project. It takes Spring HATEOAS & Spring Data and combines them together. This way, your Spring Data repositories are exported with an automatically generated hypermedia interface.

What does this mean? Maybe it’s not obvious, but data stored in a persistent repository is now available to desktop web apps, smart phone hybrid apps (those that basically wrap a browser), and native iOS/Android apps.

I’ve spent several months getting up to speed on this project by fixing bugs, adding new features, and adding to the test suite. This has slowly but surely helped me understand how it works. I don’t have it all down, but when we reached 2.0.0.GA, and had the basic CRUD functionality working, I proceeded to write a handful of getting started guides. This way, other people could quickly see through simple examples how to add a hypermedia interface to their existing Spring Data-powered apps.

But for at least six months, I’ve had a vision for where I could take this project. I finally felt I had enough know how to start working on a sample application that would heavily use this hypermedia + some javascript to build a dynamic web app. (NOTE: There are pictures of cats!) That’s what I’ve been working on for the past couple weeks.

Thankfully, we have some super sharp javascript coders on our team. I’ve been able to get solid feedback on my newbie javascript code. And it’s been fun! Javascript has a neat, function-oriented perspective. Instead of focusing on classes, i.e. nouns, I have been writing in terms of verbs. When you click this, do that, then that. It’s a big shift. But I ran into familiar territory when I needed to splice some data together. One of my javascript colleagues told me about reduce(), and when I read about it, I recognized it was the same as foldLeft.

Ultimately, I didn’t use it, because it didn’t fit what I needed, but nonetheless, it was nice to run into javascript’s nice functional APIs.

I have been having an extra dose of fun this past week. My colleague commented, “hey @Greg. looks like you’re having a ton of fun! :)” That was the truth! I just didn’t realize it was so obvious.

I’ve always been a language junky. While I enjoy writing Java full time and am really good at it, I love the opportunity to write code in any language. (Okay maybe not COBOL, but I think you get the idea.)

Hopefully I’ll be able to offer a presentation at SpringOne on this exact topic. It’s not guaranteed, but I’ll see if I can get a talk approved. Now…back to coding javascript!

Show don’t tell and character POV

undergroundersAmong the great feedback I’m getting from a couple generous beta readers, I’m getting a real primer on show-don’t-tell.

In case you haven’t heard, there is an old mantra in writing. Instead of having the characters enunciate what they are seeing, hearing, and feeling, i.e. telling, you instead show what’s happening. There are many reasons for this approach. It often connects the reader more intimately with what’s happening. They are more plugged in as if they were there, instead of hearing it retold through the character after the fact.

I’ve worked hard to shoot for that, but being so close to the work, I didn’t realize how many times I veered back into telling and got away from showing the action. Look at this example.

She heard some scurrying motions. She looked around, trying to see what their unit leader was going to do. “Are we going to back them up? How many are there?” As she looked to the front of their wave, she could see three of the men communicating with hand signals. “Blast, I wish I had paid attention better so I could know what they were saying.” One of them had pulled out a small tablet and was scribbling marks on it.

“She heard” puts distance between the reader and what’s happening. Try this instead.

Booted feet scraped the tunnel floors, scurrying to the front of the group. “Are we going to back them up? How many are there?” Three of the men at the front of their wave furiously exchanged hand signals. “Blast, I wish I had paid attention better so I could know what they were saying.” One had drawn out a darktablet and hastily scratched on it with a stylus while pointing back towards Base. The second man nodded. The third shook his head.

Which one draws you in and makes you feel like your really there?

Character POV

In my experience, a side effect of telling and not showing is that I was letting the POV (point of view) slip in various scenes. There are different ways to write such as 1st Person POV, 3rd Person POV, Omniscient POV, and others. 1st Person is “I saw. I walked. I whatever.” 3rd Person is “Clarel saw. Clarel walked. Clarel whatever.”

Whether you write in 1st Person or 3rd Person, it’s quite common that in any given scene, you are only able to know what one character knows, observes, hears, etc. Omniscient lets you know everything, but few people seem to use this one. It can get confusing as you bounce around between different characters. In a few of my scenes, I errantly floated between two or three characters. It was hard for my beta reader to track that.

Why am I talking about POV and show-don’t-tell? Because by focusing on show-don’t-tell, it forces me to only show what can be seen from the scene’s POV. It intensifies that character’s POV which tightens things up for the reader. I feel this leads to better action and suspense.

Now as I skim certain scenes, I can see how various parts are more sloppy than others. As part of the my follow up to these critiques, I am going through an tagging each scene with the character’s POV. It’s obvious that certain scenes were written across a few days, and I lost focus of whose POV it was. My goal is that by sifting through the manuscript and cleaning up POV with show-don’t-tell, it will improve the story tenfold.

Verb tenses

Another smaller item I’ll have to evaluate is past present and past perfect tenses. In general you need to pick the tense of your story and stick to it. Books are commonly written in past tense. But there’s more than one past tense. “She was pausing” vs. “She paused”. This is past progressive vs. simple past tense. Simple past slims things down and doesn’t sound so tedious.

I try to avoid past perfect (like “She had paused.”) because it’s not only tedious, it also denotes something that happened in the past before something else in the past. The only time I use it is for flashbacks in my story. This way, the flashbacks are clear and easy to spot. It also fits in with the concept that a flashback occurred BEFORE the “current” past tense.

On a side note, I have been studying German for about three months. While exercising my usage of English past tense, I am also learning about German past tense. Here are a couple types. “Ich habe das Buch gelesen” literally means “I have the book read”. Rearranged for Enlglish grammar, it translates to “I have read the book”.

An alternative is “Ich las das Buch” which means “I read the book. These are past perfect and simple past. In German, they are often described as spoken past and written past (or story past according to my native German high school teacher), because when you are talking to someone you use the former, and when writing a story you use the latter.

Suffice it to say, using the right tense is important in any language and if you mix tenses willy nilly in your manscript, any potential agent or editor will cringe and dismiss you. You will appear amateur if you don’t focus on using tenses correctly and consistently. Scrubbing your manuscript can wear you out, but it’s important work.

As I get more and more feedback, I’m deeply grateful. Happy writing!

GregLTurnquist moves to WordPress!

wordpress-300x300I recently moved my financial blog to WordPress and the experience has been fantastic. So I asked, why not do the same here?

The features are fantastic including: nice mobile view, easy management of old and new links, very professional look-and-feel, and superior options and features.

I’ve been using this blog to write about software, writing, and a host of other things. Hopefully it’s been enjoyable for you.


How important is the title of your book?

I have found titles the first step in my desire of whether or not to read a particular book. It isn’t the whole choice, but I can remember strong titles like “Foundation” and “Dune”. I came to the Jack Reacher series by way of the movie, so the title had less impact on me there. Nonetheless, I’m not the only one saying this.

It’s the reason I have latched onto Darklight as the title of my first book. The word sounds strange and intriguing, and a bit like an oxymoron. It also seems to inviting a potential reader to open the book and found out WHAT that means.

When I finished Darklight and sent it out to my beta readers, I began forming thoughts about a sequel. At the time I started Darklight, I had the vision of a world with some characters sprinkled in it. By the end, my focus had strongly shifted towards the characters. I realized that investing in characters was way more important to myself and my readers than the backdrop where they would interact.

And one character in particular began to stand out as the pivotal one. This was evident as I hammered out the beat sheet. Naturally, this character is the one to start in the sequel. And I can easily envision a whole series of books with this character having to face new struggles.

So what would I call my second book? I was attracted to the idea of having a recurrent theme across the books, but frankly it didn’t seem to fit the bill. What I have observed is that I’m not writing multiple books chaining together episodes of one story. Instead, it’s like each book is a different adventure. The Jack Reacher series has completely different titles, so I basically set that presumption aside and focused on the one-sentence summary for my second work and came up with Neophyte. I think it summarizes the circumstances my character is having to deal with. And also the potential ramifications that will have for others.

The other tasks before me, after having just drafted the hook, is to work on the beat sheet and also draw up more details of each character. I want good, solid characters. Because then the scenes begin to write themselves. Having a good, strong design for my story will make it easier to share it with others.

Happy writing.!

Nose to the grindstone writing Neophyte’s beat sheet

I previously talked about first thoughts of a sequel to Darklight. So far, I drafted the opening hook. This is basically the part of the book where you set the tone and try to rope in a potential reader by “hooking” them such that they can’t put it down. You want someone to pick up your book at the store, read two or three pages, and then HAVE to buy it.

Well, I had an entire sequence of action in my head, and so I went and captured it. It was delicious, except it was missing something. I thought about what I had read in the past year and what “hooked” me. Nothing stood out more so than the intro to Killing Floor, the first of the Jack Reacher series. That book has a bold introduction where you are thrown into the action. I reviewed what it was about that book that made me want to read every one after it.

It didn’t waste time with boring monologues, but instead dove straight into action. You didn’t know everything about Jack Reacher, but from what he was saying, I was getting hints left and right. But the thing that really added spice to the sauce was its first person narrative.

So I took what I originally wrote for an opening hook and rewrote it in first person. That seemed hard at first. It forced me to really avoid dumb monologues and instead communicate voice and action. Once I got into the groove, it almost seemed easier. Almost. When I finished, the whole section seemed more exciting, which was what I wanted. Every time I open the manuscript to work on it, I debate whether I can write the whole novel this way. And I keep saying yes!

But a hook isn’t a novel. I have an idea. I know where point A is as well as point Z, but how will my character get there from here? This where I have to put my head down and start knocking out a beat sheet. This is hard work, because I have to think out lots of parts. I need to stitch things together. I have some big plot points, but I haven’t figured out how it all connects together.

I can easily see this being the point where many potential authors fizzle. It just isn’t fun. Writing action and dialog is fun. But thinking about what to write is flat out boring. But without this structure, my novel won’t come into being. Some authors really can write from the seat of their pants. I can’t. It doesn’t work that way for me. But hopefully the excitement of that hook will provide motivation to finish out the beat sheet and then get cracking on the writing.

Gradle or Maven? I really don’t care

The project I’m working on, Spring Data REST, recently rolled out a big update: they changed the build system from gradle to maven!

This kind of shocked me at first. I’ve been hearing that gradle is hip and cool and maven is old and stagnant. All hail gradle!

And yet, in just a few hours, I realized that I really didn’t care. I didn’t care because both gradle and maven are GREAT build tools. That automate the entire process. Since 99% of my time spent on a given project is in the source code and support scripts and NOT the build process, I don’t really care a lot about what build tool is in use.

I understand people run into issues with maven. If you try to deviate too much from the convention you will pay for it dearly. Guess what; it’s not that hard to follow the convention. It’s not a bad price to pay for being able to jump to other projects and see the same layout. It’s also handy that IntelliJ IDEA can easily suck in a multi-module, maven-based project without batting an eyelash.

What about gradle? It’s pretty neat as well. I sure enjoy updating the build system when needed without having to bear all the XML cognitive overload. Gradle’s groovy-based system is simpler to read. I appreciate that IntelliJ IDEA can also crack open a gradle-based app just as well.

But then again, it can run away. I’ve seen REALLY complex build systems (like the Spring Framework). It becomes an art to discern all the stuff involved to build that hunk of software. Why? Because Spring has a lot of stuff going on! No build system can make it super simple.

Pros and cons for each?

  • Gradle it easier to read. No XML overload. It’s easy to use, declarative, and comes with gobs of plugins. I have even had people tell me that crafting your own plugin is pretty easy. But it doesn’t have the concept of parent projects that can provide inheritable version numbers
  • Maven is widely adopted. Even when we wanted to primarily use gradle for Spring’s Getting Started Guides, we decided that maven support could not be ignored. Maven has more than one way to inherit a set of dependencies and properties. This feature is REALLY useful. Spring Boot leverages it widely, effectively declaring a stack of dependencies it uses.

Bottom line: either one of these are better than some ivy+ant solution. I have worked with ant in the past. It is so customizable, so detail oriented, so extensible, that you can ONLY interact with it at the low level. I don’t imagine IntelliJ IDEA being able to suck in an ant project and immediately fetch all it’s dependencies.

You are dragged down to that horrendous level of detail every time you want to dig into an ant-based build. Most of the time, when I interact with the build system, it’s because I need to roll some versions and upgrade things. Maybe add a new module. That’s easy.

In the end, trading in such detailed complexity has a huge productivity payoff. I have to use both because of the proliferation of projects, hence, it’s not a steep job for me to switch gears.

A tale of two git workflows

My company has gone full on with using github as the place to host our code. It’s great! I love git and I love github. And lately, in the past couple of months, I have taken on using hub, and git + hub tool.

So what’s the difference in workflow? The standard approach suggested by github is fork-a-repo. In a nutshell, you fork the repo, clone it to your box, and add a remote to the “official” project named “upstream”. The problem I ran into all the time that was annoying was that I had to keep updating my own master branch to keep it in sync with the official project. Sometimes I would accidentally forget that my own master branch wasn’t up-to-date and have to back up to do that extra bookkeeping.

But then came hub. That tool has a different workflow that is superb. You create a fork of the official project, but that is not the one you clone to your work machine. Instead, you clone the official project and then add a remote to your own fork. Create a new branch, push it to your fork, and submit a pull request back to the official repo.

By taking away the need to sync my own repo, it has lifted this burden of remembering that step. Instead, I simply perform periodic “git pull”s against the official repo, and things stay up-to-date.

The tradeoff? Visit my own fork, and you won’t see an up-to-date master branch there. But so what? I don’t build anything against my own master branch. Instead, I’m more interesting in all the contributions going into the official master branch.

The confusion? Why does github (the producer of both of these workflows) suggest two flows? It’s confusing. I have already run into a co-worker that uses the classic one vs. hub’s. I had to chat him through an issue when it was time to deploy and I didn’t have the proper commit rights.

NOTE: It has come to my attention that “git pull-request” has been deprecated. That is simply because github might be deprecating the ability to convert issues into pull requests. But after I found the issue on the repo, it turns out that if github indeed removes that feature, hub will simply replace it with another command to issue a new pull request from the command line.