Category Archives: software

Tuning Reactor Flows

I previously wrote a post about Reactively talking to Cloud Foundry with Groovy. In this post, I want to discuss something of keen interest: tuning reactor flows.

When you use Project Reactor to build an application, is the style a bit new? Just trying to keep your head above water? Perhaps you haven’t even thought about performance. Well at some point you will. Because something big will happen. Like a 20,000 req/hour rate limit getting dropped on your head.

Yup. My development system mysteriously stopped working two weeks ago. I spotted some message about “rate limit exceeded” and rang up one of my friends in the Ops department to discover my app was making 43,000 req/hour. Yikes!

As I poured over the code (big thanks to the Ops team giving me a spreadsheet showing the biggest-to-smallest calls), I started to spot patterns that seemed like things I had seen before.

Reactor tuning a lot like SQL tuning

Long long ago, I learned SQL. As the saying goes, SQL isn’t rocket science. But understanding what is REALLY happening is the difference between a query taking twenty minutes vs. sub-second time to run.

So let’s back up and refresh things. In SQL, when you join two tables, it produces a cartesian product. Essentially, a table with n rows + a table with m rows, will produce a table with n x m rows, combining every possible pair. From there, you slim it down based on either relationships or based on filtering the data. What DBMS engines have had decades is learning is how to read your query and figure out the BEST order to do all these operations. For example, many queries will apply filtering BEFORE building the cartesian product.

In Reactor, when you generate a flux of data and then flatmap it to another flux, you’re doing the same thing. My reactor flow, meant to cache up a list of apps for Spinnaker, would scan a list of eighty existing apps and then perform a domain lookup…eighty times! Funny thing is, they were looking up the same domain EIGHTY TIMES! (SQL engines have caching…Reactor doesn’t…yet).

So ringing up my most experienced Reactor geek, he told me that it’s more performant to simply fetch all the domains in one call, first, and THEN do the flatmap against this in memory data structure.

Indexing vs. full table scans

When I learned how to do EXPLAIN PLANs in SQL, I was ecstatic. That tool showed me exactly what was happening in what order. And I would be SHOCKED at how many of my queries performed full table scans. FYI: they’re expensive. Sometimes it’s the right thing to do, but often it isn’t. Usually, searching every book in the library is NOT as effective as looking in the card catalog.

So I yanked the code that did a flatmap way at the end of my flow. Instead, I looked up ALL domains in a CF space up front and passed along this little nugget of data hop-to-hop. Then when it came time to deploy this knowledge, I just flatmapped against this collection of in memory of data. Gone were all those individual calls to find each domain.

.then(apps ->
	apps.stream()
		.findFirst()
		.map(function((org, app, environments) -> Mono.when(
			Mono.just(apps),
			CloudFoundryJavaClientUtils.getAllDomains(client, org))))
		.orElse(Mono.when(Mono.just(apps), Mono.empty())))

This code block, done right after fetching application details, pauses to getAllDomains(). Since it should only be done once, we only need one instance from our passed along data structure. The collection is gathered, wrapped up in a nice Mono, and passed along with the original apps. Optionally, if there are no domains, an empty is passed along.

(NOTE: Pay it no mind that after all this tweaking, the Ops guy pointed out that routes were ALREADY included in the original application details call, hence eliminating the need for this. The lesson on fetching a whole collection up front can be useful.)

To filter or not to filter, that is the question

Filtering is an art form. Simply put, a filter is a function to reduce rows. Being a part of both Java 8’s Stream API as well as Reactor’s Flux API, it’s pretty well known.

The thing is to watch out for if the filter operation is expensive and if it’s inside a tight loop.

Loop? Reactor flows don’t use loops, right? Actually, that’s what flatmaps really are. When you flatmap something, you are embedding a loop to go over every incoming entry and possibly generating a totally different collection. If this internal operation inside the flapmap involves a filter that makes an expensive call, you might be repeating that call too many times.

I used to gather application details and THEN apply a filter to find out whether or not this was a Spinnaker application vs. someone else’s non-Spinnaker app in the same space. Turns out, finding all those details was expensive. So I moved the filter inward so that it would be applied BEFORE looking up the expensive details.

Look at the following code from getApplications(client, space, apps):

return requestApplications(cloudFoundryClient, apps, spaceId)
	.filter(applicationResource ->
		applicationResource.getEntity().getEnvironmentJsons() != null &&
		applicationResource.getEntity().getEnvironmentJsons().containsKey(CloudFoundryConstants.getLOAD_BALANCERS())
	)
	.map(resource -> Tuples.of(cloudFoundryClient, resource))
	.switchIfEmpty(t -> ExceptionUtils.illegalArgument("Applications %s do not exist", apps));

The code above is right AFTER fetching application information, but BEFORE going to related tables to find things such as usage, statistics, etc. That way, we only go for the ones we need.

Sometimes it’s better to fetch all the data, fetch all the potential filter criteria, and merge the two together. It requires a little more handling to gather this together, but again this is what we must do to tailor such flows.

Individual vs. collective fetching

Something I discovered was that several of the Cloud Foundry APIs have an “IN” clause. This means you can feed it a collection of values to look up. Up until that point, I was flatmapping my way into these queries, meaning that for each application name in my flux, it was making a separate REST call for one.

Peeking at the lower level APIs, I spotted where I could give it a list of application ids vs. a single one. To do that, I had to write my flow. Again. By putting together a collection of ids, by NOT flatmapping against them (which would unpack them), but instead using collectList, I was able to fetch the next hop of data in one REST call (not eight), shown below:

return PaginationUtils
	.requestClientV2Resources(page -> client.spaces()
		.listApplications(ListSpaceApplicationsRequest.builder()
			.names(applications)
			.spaceId(spaceId)
			.page(page)
			.build()))
	.map(OperationUtils.<ApplicationResource, AbstractApplicationResource>cast());

cf-java-client has an handy utility to wrap paged result sets, iterating and gathering the results…reactively. Wrapped inside is the gold: client.spaces().listApplications(). There is a higher level API, the operations API, but it’s focus is replicating the CF CLI experience. The CF CLI isn’t built to do bulk operations, but instead operate on one application at a time.

While nice, it doesn’t scale. At some point, it can a be a jump to move to the lower level APIs, but the payoff is HUGE. Anyhoo, by altering this invocation to pass in a list of application names, and following all the mods up the stack, I was able to collapse eighty calls into one. (Well, two, since the page size is fifty).

You reap what you sow

By spending about two weeks working on this, I was able to replace a polling cycle that perform over seven hundred REST calls with less than fifty. That’s basically a 95% reduction in network traffic, and nicely put my app in the safe zone for the newly imposed rate limit.

I remember the Ops guy peeking at the new state of things and commenting, “I’m having a hard time spotting a polling cycle” to which the lead for Cloud Foundry Java Client replied, “sounds like a good thing.”

Yes it was. A VERY good thing.

Reactively talking to Cloud Foundry with Groovy

I’ve been working on this Spinnaker thing for over a year. I’ve coded support so Spinnaker can make continuous deployments to Cloud Foundry. And the whole thing is written in Groovy. I recently upgraded to that I can now talk reactively to Cloud Foundry with Groovy.

And it’s been a nightmare.

Why?

Groovy is pretty darn wicked. Coding Spring Boot apps mixed with Spring MVC controllers in the terse language of Groovy is nothing short of gnarly. But it turns out there’s a couple things where Groovy actually gets in your way.

Reactor + Cloud Foundry

Want a taste? The code fragment below shows part of a flow used to look up Spinnaker-deployed apps in Cloud Foundry:

operations.applications()
  .list()
  .flatMap({ ApplicationSummary appSummary ->
    operations.applications()
      .getEnvironments(GetApplicationEnvironmentsRequest.builder()
        .name(appSummary.name)
        .build())
      .and(Mono.just(appSummary))
  })
  .log('mapAppToEnv')
  .filter(predicate({ ApplicationEnvironments environments, ApplicationSummary application ->
    environments?.userProvided?.containsKey(CloudFoundryConstants.LOAD_BALANCERS) ?: false
  } as Predicate2))
  .log('filterForLoadBalancers')
  .flatMap(function({ ApplicationEnvironments environments, ApplicationSummary application ->
    operations.applications()
      .get(GetApplicationRequest.builder()
        .name(application.name)
        .build())
      .and(Mono.just(environments))
  } as Function2))

This is the new and vastly improved Cloud Foundry Java SDK built on top of Project Reactor’s async, non-blocking constructs (Mono and Flux with their operations). Every function call is an async, non-blocking operation fed to the next function call when the results arrive.

What does this code do? It looks up a list of Cloud Foundry apps. Iterating over the list, it weeds anything that doesn’t have a LOAD_BALANCER environment variable, a tell for Spinnaker-deployed apps. Finally it looks up the detailed record for each application.

The heart of the issue

What’s nestled inside several of these “hops” in this flow is a tuple structure. In functional flows like where each hop gets a single return, we often need to pass along more than one piece of data to the next hop. It’s the side effect of not using the imperative style of building up a set of variables, but instead passing along the bits in each subsequent funtion call.

cf-java-client has TupleUtils, a collection of functions meant to pack and unpack data, hop to hop. It’s elegant and nicely overloaded to support up to eight items passed between hops.

And that’s where Groovy falls flat. Groovy has this nice feature where it can coerce objects. However, with all the overloading, Groovy gets lost and can’t tell which TupleUtils function to target.

So we must help it by coercing it into the right structure. See those “as Function2”  and “as Predicate2” calls? That helps Groovy figure out the type of lambda expression to slide things into.

And it’s dragging me down!

The solution

So I finally threw in the towel and converted this one class into pure Java.

Yes, I ditched hip and cool Groovy in favor of the old warhorse Java.

You see, when something is so dependent on every character being in the right place, we need all the static support from the IDE we can get. Never fear; I’m not dropping Groovy everywhere. Just this one class.

And here is where Groovy’s interoperability with Java shines. Change the suffix of one file. Make the changes I need. And both the IDE and the compiler is happy, giving me an operational chunk of code.

I had to rewrite a handful of collections, but it wasn’t the worse thing in the world. In half a day, I had successfully moved the code. And now as I’m working on another flow, the pain of Groovy’s need for coercion specification is no longer wreaking havoc.

Cheers!

 

Have you crossed the midpoint in your career?

There is something that has snuck up on me. When I stopped to think about it, it became clear. There is a point in your career when you cross this “midpoint.” I remember Day One of my first job as a professional software engineer.

I had already written little scripts, apps, and other hobby projects. But this was the day I was building stuff for my livelihood. The day when I had to start steadfastly listening to others, and doing what I was told. Sounds scary, right? Well, not really. I guess I was too gung ho. And the people that hired me did a great job at hand holding. Nevertheless, what I know today and what I knew back then are starkly different.

What is the midpoint? I like to think of every person out there writing code on a spectrum. People on their first day start at the bottom. Then slowly, but surely your knowledge and experience causes you to rise. On your Day Two, there is someone else having their Day One. At a certain point, you cross the halfway mark, or the “midpoint.” Congratulations, you arguably know more than half of the other people in the your field.

This isn’t an article about arrogance, or how I’m better than you. No. This is recognizing that the things you learned in college have benefit, but we all gain new talents and experience every day. And your responsibilities change.

On Day One, your responsibility may be to test someone else’s code. (That was mine!) But at a certain point, perhaps after you’ve crossed the midpoint, your responsibility may be to help others. To teach, lead, form communities. Or to take the reins of a project where there is zero management and guidance. Those that are on Day One aren’t equipped to do that.

An honest recognition of where you “are” can make you realize that signing up to speak at a conference, hosting a JUG group, leading an open source project, or offering to run a project with little more than a single sentence for a high ranking manager in your company.

It’s also a keen time to visit places like Stack Overflow and answer questions. And remember to extend grace to those that are new and seeking to grow themselves. Some people call this giving back. I’ve never been a fan of this expression. I prefer to think of it as sharing what you have learned with others.

The magic of software development

People think I have a magical talent. It’s funny seeing the difference between what I know and what others think I know when it comes to debugging stuff.

Simply put, I know how to tinker. I have clues, hunches, and insights. But that stuff is useless to a developer not willing to roll up their sleeves and engage in ten solid hours of trial and error. Often times, that is what makes or breaks a demo or a customer feature.

clock-with-a-questionDid I mention ten hours? Sorry, I meant ten days! Or sometimes it takes ten weeks to push your way through a mountain of integration, setup, and labor. Whatever it takes. But it isn’t solved by “knowing the answer” when you sit down at the keyboard.

We software devs work in the strangest field. First, we pursue a degree in either Computer Science or Computer Engineering. Ever seen a scientist and an engineer talk? Polar opposites. So what is software development? A science or an engineering trade?

My buddy Russ Miles has likened this to R&D, since there is no real sense of “can you have that ready in a month?” Building a bridge between two fixed endpoints is much easier to estimate than “can we have this website operational by next year?”

Experiment: Find your most non-technical friend, and ask them if they have a clue what you do. Then try to clarify when they say “no.” And try, and try, and try.

Solving software problems can seem foreign to both science AND engineering. On many days, it’s just tinker, tinker, tinker. And yet, our world’s dependency on it is growing.

HamletSoftware is really just organizing a set of patterns to respond to information. We need certain behaviors triggered automatically, and software is a giant stack of rules meant to handle that. As we collect more information and link more things together, the need to manage this grows.

A good visual of what software really entails is a hologram. Remember that limited-response hologram from I, Robot? We are just tinkering to put together a set of rules to better manage everything around us.

This requires tinkering, not magic. Good ole elbow grease is sometimes the best tool.

The value of backwards compatibility

springI was listening to Episode 6 of the Nash Dev Cast as they spoke about bitrot, of how stuff degrades as the whole Internet upgrades around you. It really reminded me of the inestimable value of Spring’s approach to backwards compatibility.

Something that is almost unnoticeable is that the Spring Framework and its entire portfolio has backwards compatibility. Bumping up the version in your project when a new release comes out is a no-brainer. They just don’t break stuff. You may get alerted to deprecated features, but the JavaDocs will kindly direct you along the upgrade path. Moving from Spring 1.2.8 -> 2.5 -> 3.0 -> 4.0 has such little cost, that people are eager to jump to the latest.

junior-devAs a junior developer, this kind of thing doesn’t really resonate. And not because you are somehow bad. No. It’s simply because you don’t have as many war stories to share over upgrading other libraries, and having major breakage. Senior developers at conferences will share a beer and discuss some crazy upgrade that got started, and a breaking change they were forced into turned a 2-day upgrade into a 3-week effort.

Even as a senior developer who recognized this feature of the Spring portfolio, it wasn’t until I joined the team and proposed changes, that I learned of the technical challenges this team faces to enforce backwards compatibility. It really rose to the surface when I heard Juergen speak about Spring 4 being “Java 8 ready” but still working with Java 6. With scrunched eyebrows, I leaned closer as he spoke during the 2013 Birds of a Feather at SpringOne how the framework would certainly support Java 8 features, but still operate on Java 6.

That perplexed me. How could a framework take advantage of things like Optional yet blink “no problem” if you used Java 6? Digging in, I learned not only how to build a JAR file with Java 6 specs but neatly keep Java 8-based API calls inside Java 8-flagged classes that were only loaded when Java 8 was the loading engine. And I learned how hard this effort was.

I already respected the Spring team and the programming model they had developed. But this level of technical expertise combined with forward thinking plans to start ten years ago and maintain to this day, was quite impressive.

So, listening to podcasts about JavaScript open source modules, sifting through and evaluating products, and getting “left-padded” at times, made me snug and comfortable in my decision to bring onboard the Spring Framework years ago. Their decisions to invest so highly in the community have paid dividends I have no idea how to tabulate.

In defense of leftpad

Given the time I’ve had to think about the whole leftpad controversy, I have to come to the conclusion that leftpad was PERFECTLY FINE with its decision to build a module with only eleven lines. (As to the brooha over unpublishing and then being forcibly republished by npm, that is a different topic.)

leftpad wrote code anyone with half a brain should be able to write

laptop_chimpThat must be the biggest sentiment I’ve heard. Blogs, podcasts, and other forums have thrown out this strong criticism of leftpad’s code. It’s best summarized as “have people forgotten how to code?”

Gee, leftpad getting unpublished broke React.js, Babel.js, and how many other systems? If those impressive projects saw value in leftpad, sounds pretty big of me to judge them all for making such a foolish decision. After all, that’s what I’m doing, right? I’m ascribing judgment for using it.

Let me be clear: I haven’t actually READ leftpad’s code. I’m not defending the physical lines of code in that module. The market spoke and accepted leftpad. I am defending the choice others made because I myself haven’t invested the time to make a better choice.

leftpad wasn’t REALLY used by React.js, etc. It was just a transitive dependency

2001This one doesn’t sit right. It’s kind of like saying, “React.js is good. The people writing it would never need something like leftpad. They just got caught by some intermediate dependency.” Can you smell the disdain?

If leftpad was used by an intermedia dependency, and that is what all the big weights used, then the transitive property applies. Not only has the market said leftpad met needs, but the intermediate dependencies are just as valuable. See what I did there? Transitive dependency, transitive value.

leftpad isn’t actually the best way to do what it did

Kaypro_in_Israel_1984Here is where it gets interesting. This is what I hear from those that have spotted the transitive link. Can’t criticize React.js, because they didn’t REALLY use leftpad. Let’s look elsewhere.

People have been singing the UNIX tune of “do one thing and do it well” as microservices arise. Defenders of leftpad point it out as a composable function, adhering to the UNIX philosophy. But if you can tear out “do it well”, then leftpad doesn’t qualify for this concept. Hence, leftpad was a badly written composable function.

Sorry, maties, but as software engineers, we seek out solutions on a daily basis. And in the land of open source, we constantly make “build or buy” decisions. Whenever someone asked me, “what is an engineer”, my answer has always been, “we find a good enough solution for the problem at hand.” We don’t seek perfect answers, because we don’t have an infinite budget.

For a LOT of people, leftpad was good enough. They could have written a better solution, but they could also have written a better solution for LOTS of other modules. Imagine the cost of perfecting every single thing you use. Or how about this: imagine the cost of tens of thousands of developers writing an equivalent to leftpad. Perhaps 15 minutes tops, but multiplied by 10,000 and your talking 2500 hours of effort. Doesn’t that defy the concept of open source?

What’s next?

imageI know people that don’t like certain tools out there. They have written their own version of things. At first glance, there’s underscore and lodash. These two libraries essentially do the same thing. But one project was not satisfied with the other project. Hence, they built a fork and whole community around it.

These are people I respect, because they took the time to evaluate the current state of affairs and write their own solution. When I go pick from one of these two solutions, I can find out the basis of one or the others. Has the bulk of the JavaScript community picked one of the other? Or is it a big split? These are factors in choosing.

I can spend all my time building everything from scratch and live under the haughty label “I know how to program. I’ll build it myself.” But we already have a term for developers like that: “Not Invented Here“. Some people will only touch code they themselves wrote.

If you actually spent the time to read leftpad’s code, then used it or decided otherwise, that’s fine. But don’t go around criticizing others for using leftpad. After all, you’re criticizing the choices of MANY others.

Good developers take breaks

Boring PresentationSomething that has become crystal clear since I joined the Spring team is how important it is to take a break. Good code happens when developers take breaks.

Of course there are times when I find myself working solid until 8:00 pm or later. But I try to make that the exception rather than the norm. Most of the time, I’m pulled away by my kids coming home from school. In fact, I’m often annoyed at stopping a minute early.

STTNG_pinballAnd yet I’m astonished at how many perplexing problems I’ve solved by simply stopping, doing something else, and **BAM**, the answer pops into my head. Sometimes within minutes, sometimes the next morning while putting shoes on one of my kids. Those are the moments when I remind myself that developers take breaks. I consider that payoff for having stopped early.

This phenomena is well know. In fact, you’ve probably heard of it. I also so it addressed keenly at a writer’s conference last year. Your subconscious is still working on the problem whether you are or not. Sometimes, when you aren’t actively trying to fix it, your noodle is freed up to look back at what you did, what you’ve read, and other things.

devnexusWhile chatting with a teammate of mine at DevNexus, he expressed that if he hadn’t taken various breaks, gone for walks and thought about the architecture he was designing, the project he was striving to build would never have happened.

HamletReflection is a critical component. My martial instructor often taught “visualize, visualize, visualize.” It’s a mechanism to put your mind on the subject, even when you’re not actively pursuing it.

The key thing is telling yourself to take that break. To pause. To stop and not burn the candle at both ends. If you work that hard, your subconscious will be on the ropes. Those extra ideas that can often succeed when you may have failed all day, may not come at all. It’s a big leap I know, but give it a try.

#opensource is not a charity

clock-with-a-questionLogging onto my laptop this morning, I have already seen two tickets opened by different people clamoring for SOMEONE to address their stackoverflow question. They appeared to want an answer to their question NOW. The humor in all this is that the issue itself is only seven hours old, with the person begging for a response when their question is barely three hours old. Sorry, but open source is not a charity.

gordonBatPhoneIf you have a critical issue, perhaps you should think about paying for support. It’s what other customers need when they want a priority channel. It definitely isn’t free as in no-cost. Something that doesn’t work is opening a ticket with nothing more than a link to your question.

question-not-answeredOpen source has swept the world. If you don’t get onboard to using it, you risk being left in the dust. But too many think that open source is free, free, FREE. That is not the case. Open source means you can access the source code. Optimally, you have the ability to tweak, edit, refine, and possibly send back patches. But nowhere in there is no-cost support.

pivotal-ossIn a company committed to open source, we focus on building relationships with various communities. The Spring Framework has grown hand over fist in adoption and driven much of how the Java community builds apps today. Pivotal Cloud Foundry frequently has other companies sending in people to pair with us. It’s a balancing act when trying to coach users to not assume their question will be answered instantly.

helpingI frequent twitter, github, stackoverflow, and other forums to try and interact with the community. If at all possible, I shoot to push something through. Many times, if we’re talking about a one-line change, it’s even easier. But at the end of the day, I have to draw a line and focus on priorities. This can irk some members not aware of everything I’m working on. That is a natural consequence.

Hopefully, as open source continues to grow, we can also mature people’s expectations between paid and un-paid support. Cheers!

P.S. For a little while longer, there is a coupon code to Learning Spring Boot for 50% off (Python Testing Cookbook as well!)

Banner (LSPT50)

Why software development is not for everyone

Have you ever had gobs of fun hacking away on a computer? Noodled with a piece of code that you discovered in the afternoon, and here it is, 2:00 a.m.? That’s a sign you may be a computer geek. That’s all and good, but the question that may come before you is, do you want to make this your dream job? Your career? Watch out, though. Software development is not for everyone.

mr_potatohead-300x190I fear that some people may be getting into software development because computers are now hip and cool. And the money is good! I remember a time, several years ago, when I saw the corner turn in how computer geeks were no longer the goof bots found in Wargames, but instead, totally righteous dudes.

What was that tip off? A TV ad for liquor which showed “Jim from IT” partying just like everyone else. TV ads are a bellwether for trends, because that write them are constantly polling groups of people to see what will resonate. When IT people had entered the main fray of the party-going crowd and were seen a force to be reckoned with (err…a force to be advertised with), then game over.

The side effect of anything going mainstream is that others, who have no deep seated desire to submit and review pull requests on vacation (for the record, I’ve actually done both at Disney World. Top that!), will still flock to the field since its cool, and there’s money to be made. Am I saying that to get ahead, you must be a workaholic and eschew your family? Not at all. What I’m saying is that computer geeks that are in it for the challenge and not just cuz it’s cool can’t help themselves but do what i just described.

So far, I’ve talked about hacking on bits of code at odd hours in odd places. But here is the real test, the true gambit to see if deep down you really are a totally righteous hacker dude: do you go after the most boring, challenging, mind numbingly difficult problems until the code surrenders its secrets to you?

Do you go after the most boring, challenging, mind numbingly difficult problems until the code surrenders its secrets to you?  –sign of a righteous hacker

I have been working on something for several months that is coming to light in less than two weeks. (Stay tuned!) For MANY hours/days/weeks, every time I tweaked one line of code, I had to restart a microservice. That microservice would take, on average, two minutes to load up its data set from a remote system. Next, I went to another tab and clicked on a button to launch a process, and wait to see if my system detected it and processed it correctly. Got that? Edit one line of code, wait five minutes for the results, then debug the outcome. Rinse and repeat.

aaarrrggghhhAaaarrgggghhh!!!! <— that was my reaction on many days. And my followup was, “thou shalt not defeat me!” My street cred was at stake, and my own internal desire to beat this system into submission were there as well.

Suffice it to say, trudging through this task is not for the faint of heart. Software hobbyists need not apply.

I’ve always hated the expression “9-to-5 coder.” It’s meant to imply people that only work on the clock, and don’t work late hours, etc. In general, I do that as well. At times, I have to alert my family that I’ll be putting in extra hours (especially the month before SpringOne each year), but in general, I have found that working smarter, not harder, and thinking out solutions before slinging code can make things more efficient.

So I prefer to think of developers not based on the hours they work, but rather, are you a hobbyist or a diehard coder? Diehard coders will take the most ruthless problems and dig, dig, dig, dig, and dig until the answer is found. They are willing to take many hits, submit faulty patches, and screw things up big time, because the end game is important. Version control is our tool to save us from screwing things up, not timidity. Are you a hobbyist that fears making a big change? Or are you a diehard hacker ready to show your handy work to others?

I remember this topic coming up when a website popped up offering bounties for people to solve problems. This was rightly criticized as encouraging people to go after easy–to-tackle issues, or selecting issues that carried much public praise, like UI mods. But serious, professional hacks will take some of the most invisible things (like a 300ms slowdown) and go to extreme ends to unearth what’s wrong. Imagine trying to get ANYONE to pick up that issue for a bounty.

If such tasks sound outrageous or tedious, no problem. I understand. You are hobbyist. Nothing wrong there. But if you’re ready to dive into the slimiest mud bath of code, smile. You are rare.

We won’t own our own information until we do

Not a week goes by when I hear some story about people displeased with how their own content has been misused. People gripe that entering stuff into Facebook no longer belongs to them. Other people copy-and-paste such stuff into other places. Things go “viral”.

I chuckle at how so many politicians act like we are still in a pre-YouTube era. They say and do stupid things in front of small crowds. Someone inevitably grabs a clip with their phone, posts it to YouTube, and it blazes across the twittersphere. Or someone makes an accusation or promise, and in minutes, a video interview from fifteen years surfaces and gets re-posted to the interwebs.

The truth is, there should be some sense of ownership of our content. If I post something, somewhere, I understand people’s desire to hold onto it and have the last word. Unless you speak it in front of someone else, and they make an independent copy, it makes sense that you should be able to effectively delete your posting. What you said and the repercussions thereof are your own issue to deal with.

The truth is, we can’t “own” anything we write anywhere unless we can wrap it using encryption technology. Essentially, if every blurb you posted was an encrypted bundle, people would have to come to you to decrypt and read it. Throw away the key, and that blurb is gone forever. For certain avenues, like everything I write on Facebook, I can understand retaining a hold on it. And if someone wanted to copy-and-paste it, it would be really nice is the encryption traveled with it. Copy it 1000 times, as long as the wrapper is in place, and you still control it.

But there in lies the rub. Encryption technology has proven far too difficult for mass consumer adoption. Just now, the web is headed towards moving all web traffic from HTTP to HTTPS to prevent intermediate snooping. This should have happened years ago. But the next big leap would be encrypting all email traffic. With people losing laptops and thumbdrives, lots of security breaches have happened. If ALL email was wrapped in encryption, privacy would be a much stronger concept. But the process of doing that is arduous. Exchanging keys, keeping your private key secure, and then entering passphrases all the time is a hassle.

Take that concept and apply it to every other medium in which you write something. Even this blog entry! It’s safe to say, such a concept won’t come to pass until it becomes effortless to prove on a terminal you are who you say you are, and to lock and unlock keys suitably. Seeing the web move to SSL is a good sign. I just hope we can migrate along these paths faster than we can figure out to integrate stuff together smoother. We must own the pipes and the traffic our data rides along.