Category Archives: rest

HTTP + REST + OAuth = say what???

In the past couple of weeks, things have really gotten hopping on the Nashville Java community’s #java slack channel. A recent topic topic of interest is how do we take something like REST, which talks about clean URIs and stateless services, and stir in this crazy OAuth stuff?

I threw in my own $0.02 given I’ve done a bit of work on projects like Spring HATEOAS and Spring Data REST. And my $0.02 is this:

HTTP + REST + OAuth should work hunky dory without compromising any of these principles.

What does this even mean?

HTTP is a spec that goes back twenty years. Or should I say, HTTP is an amalgamation of specs that goes back that long. HTTP includes the request/response protocol, the verbs (GET, POST, etc.), the concept of media types, content negotiation, and more. At the heart of MANY of these specs is none other than Roy Fielding. Yes, the man that wrote his doctoral dissertation on REST had his fingers in a dozen specs that governs how the Internet operates.

REST is doctoral dissertation that was published in 2001. REST not a spec, not an API, not a standard. It’s an idea. An idea that if similar constraints that existed for the web plus a few others were adopted when building Internet-based services, then they too could enjoy the scalability and fault tolerance the web has come to enjoy.

OAuth is an 11-year-old security protocol driven by the explosion of 3rd party social media applications that were requiring users enter their credentials so the apps could log into social media networks on their behalf. It introduces a flow whereby the app, instead of gathering credentials, would take the user over to the source site, have them login there, be issued a token, and thereafter use the token in lieu of actual credentials.

All of these concepts have grown supreme in applications today. Enterprise shops want big apps to support millions of customers, and they want them yesterday. Hence the desire for scale.

We’re all aware of how the web was won the war of the UI. I remember coding Swing apps in data centers, and side stepping the growth of the web, but the web has beaten thick client apps hands down. The only REAL thick client app are the apps found on mobile devices. The standards of web apps is front and center.

EVERYONE wants OAuth. I’ve often stated “it isn’t real until it’s secured” indicating that frameworks without a security solution will get passed over by production shops. That’s because no customer will talk to you unless your solution can be secured. And with the rise of mobile apps and multiple clients talking to a backend, the need for OAuth is gigantic.

So how does this mishmash of ideas all fit together without tragic compromise? Let’s take things apart.

Fundamentals of REST

One of the most fundamental concepts behind REST is to include not just data in the payload, but links to resources from which you can DO SOMETHING with the payload. Or links to a state you can transition toward. For example, a Coffee Shop API that let’s you place orders should include a link to cancel the order WHEN IT’S POSSIBLE. By pushing the link to the human user, they can see when they can/can’t do that. Hence the rise of media types that support including links. For example, the JSON document below shows HAL, with both data and links.

{
  "firstName" : "Frodo",
  "lastName" : "Baggins",
  "_links" : {
    "self" : {
      "href" : "http://localhost:8080/persons/1"
    },
    "address" : {
      "href" : "http://localhost:8080/persons/1/address"
    }
  }
}

That’s nice, but what do we do when this data needs to be wrapped with OAuth?

OAuth, as briefly stated, is a flow where the website redirects the client to a login page, potentially at a different site altogether. A token is issued, and then we go back to the original site. Now we have a token in hand.

This token nicely fits into an HTTP Authorization header like this:

Authorization: Bearer 0b79bab50daca910b000d4f1a2b675d604257e42

If you’ll notice, I said header. HTTP headers don’t infringe upon payloads designed to support REST. HAL doesn’t have to know that the service is protected by OAuth + whatever other security protocols are layered on. Security, as they say, is orthogonal.

That’s because HTTP includes options to extend itself. HTTP has room for new media types (application/hal+json and application/hal+xml), new headers (Authorization), and new payloads.

And why shouldn’t we expect this? The man involved with designing the web also designed REST. Someone trying to take lessons learned from the web and apply them to APIs would surely work to fit it into the nature of the web.

To approach a secured service, I make the call, and if I get a 302 redirect to a login page, my client can come back to the human user and indicate that there is security required. At this stage, it would be completely normal to SHOW the human the login page. Let the user supply what’s needed, and my client should be able to harvest the token, and press on.

Should security be a part of the REST representation?

Yes and no. Security protocols aren’t required to be there. That would clutter up the nature of data and navigation. And clients shouldn’t be giving away any secrets, whether that is a bearer token or credentials.

But hypermedia links should be VERY sensitive to the nature of the client and serve different sets of links based on what the client is allowed to do. If the client will be declined attempting to perform a DELETE, then don’t show them the link to do it.

REST and stateless services

People like to regurgitate this REST-ism about “no state no state no state”. Yet when we get into the subject of security and the STATE of the user’s logged in nature, some people go nuts. Being stateless means that my next REST request should be able to get routed to any server on the backend. In a cloud native environment, there could be twenty copies right now, fifty five minutes from now, and two copies ten minutes after that.

The state of my next REST operation shouldn’t require I go back to the same server. If the server is feverishly trying to push session details around between gobs of servers, that is crazy. That’s why toolkits like Spring Session offer the option to use either cookies or headers. It’s possible to log into a system, get some form of session, offload the session state to a 3rd pary service, like Redis, and then bounce off of that using a header.

As Rob Winch states in the video below, a little bit of state is alright. When done like this, where you security state is managed properly in a data store and NOT on the server itself, you’ll do fine.

So wrap things up, serving up things over the web, securing them with modern practices, while also having scalable APIs should fit together nicely. Because the web was built for flexibility and extensibility, if we just embrace all of its paradigms.

REST, SOAP, and CORBA, i.e. how we got here

I keep running into ideas, thoughts, and decisions swirling around REST. So many things keep popping up that make me want to scream, “Just read the history, and you’ll understand it!!!”

wayback-machineSo I thought I might pull out the good ole Wayback Machine to an early part of my career and discuss a little bit about how we all got here.

In the good ole days of CORBA

This is an ironic expression, given computer science can easily trace its roots back to WWII and Alan Turing, which is way before I was born. But let’s step back to somewhere around 1999-2000 when CORBA was all the rage. This is even more ironic, because the CORBA spec goes back to 1991. Let’s just say, this is where I come in.

First of all, do you even know what CORBA is? It is the Common Object Request Broker Architecture. To simplifiy, it was an RPC protocol based on the Proxy pattern. You define a language neutral interface, and CORBA tools compile client and server code into your language of choice.

The gold in all this? Clients and servers could be completely different languages. C++ clients talking to Java servers. Ada clients talking to Python servers. Everything from the interface definition language to the wire protocol was covered. You get the idea.

Up until this point, clients and servers spoke binary protocols bound up in the language. Let us also not forget, that open source wasn’t as prevalent as it is today. Hessian RPC 1.0 came out in 2004. If you’re thinking of Java RMI, too bad. CORBA preceded RMI. Two systems talking to each other were plagued by a lack of open mechanisms and tech agreements. C++ just didn’t talk to Java.

CORBA is a cooking!

advanced-corbaWith the rise of CORBA, things started cooking. I loved it! In fact, I was once known as Captain Corba at my old job, due to being really up to snuff on its ins and outs. In a rare fit of nerd nirvana, I purchased Steve Vinoski’s book Advanced CORBA Programming with C++, and had it autographed by the man himself when he came onsite for a talk.

Having written a mixture of Ada and C++ at the beginning of my career, it was super cool watching another team build a separate subsystem on a different stack. Some parts were legacy Ada code, wrapped with an Ada-Java-CORBA bridge. Fresh systems were built in Java. All systems spoke smoothly.

The cost of CORBA

Boring PresentationThis was nevertheless RPC. Talking to each other required meeting and agreeing on interfaces. Updates to interfaces required updates on both sides. The process to make updates was costly, since it involved multiple people meeting in a room and hammering out these changes.

The high specificity of these interfaces also made the interface brittle. Rolling out a new version required ALL clients upgrade at once. It was an all or nothing proposition.

At the time, I was involved with perhaps half a dozen teams and the actual users was quite small. So the cost wasn’t that huge like today’s web scale problems.

Anybody need a little SOAP?

soapAfter moving off that project, I worked on another system that required integrate remote systems. I rubbed my hands together, ready to my polished CORBA talents to good use again, but our chief software engineer duly informed me a new technology being evaluted: SOAP.

“Huh?”

The thought of chucking all this CORBA talent did not excite me. A couple of factors transpired FAST that allowed SOAP to break onto the scene.

First of all, this was Microsoft’s response to the widely popular CORBA standard. Fight standards with standards, ehh? In that day and age, Microsoft fought valiantly to own any stack, end-to-end (and they aren’t today???? Wow!) It was built up around XML (another new acronym to me). At the time of its emergence, you could argue it was functionally equivalent to CORBA. Define your interface, generate client-side and server-side code, and its off the races, right?

But another issue was brewing in CORBA land. The OMG, the consortium responsible for the CORBA spec, had gaps not covered by the spec. Kind of like trying to ONLY write SQL queries with ANSI SQL. Simply not good enough. To cover these gaps, very vendor had proprietary extensions. The biggest one was Iona, an Irish company that at one time held 80% of the CORBA marketshare. We knew them as “I-own-ya'” given their steep price.

CORBA was supposed to cross vendor supported, but it wasn’t. You bought all middleware from the same vendor. Something clicked, and LOTS of customers dropped Iona. This galvanized the rise of SOAP.

But there was a problem

SOAP took off and CORBA stumbled. To this day, we have enterprise customers avidly using Spring Web Services, our SOAP integration library. I haven’t seen a CORBA client in years. Doesn’t mean CORBA is dead. But SOAP moved into the strong position.

Yet SOAP still had the same fundamental issue: fixed, brittle interfaces that required agreement between all parties. Slight changes required upgrading everyone.

When you build interfaces designed for machines, you usually need a high degree of specification. Precise types, fields, all that. Change one tiny piece of that contract, and clients and servers are no longer talking. Things were highly brittle. But people had to chug along, so they started working around the specs anyway they could.

I worked with a CORBA-based off the shelf ticketing system. It had four versions of its CORBA API to talk to. A clear problem when using pure RPC (CORBA or SOAP).

Cue the rise of the web

Around-the-webWhile “rise of the web” sounds like some fancy Terminator sequel, the rampant increase in the web being the platform of choice for e-commerce, email, and so many other things caught the attention of many including Roy Fielding.

Roy Fielding was a computer scientist that had been involved in more than a dozen RFC specs that governed how the web operated, the biggest arguably being the HTTP spec. He understood how the web worked.

The web had responded to what I like to call brute economy. If literally millions of e-commerce sites were based on the paradigm of brittle RPC interfaces, the web would never have succeeded. Instead, the web was built up on lots of tiny standards: exchanging information and requests via HTTP, formatting data with media types, a strict set of operations known as the HTTP verbs, hypermedia links, and more.

But there was something else in the web that was quite different. Flexibility. By constraining the actual HTML elements and operations that were available, browsers and web servers became points of communication that didn’t require coordination when a website was updated. Moving HTML forms around on a page didn’t break consumers. Changing the text of a button didn’t break anything. If the backend moved, it was fine as long as the link in the page’s HTML button was updated.

The REST of the story

The-Rest-of-the-StoryIn his doctoral dissertation published in 2000, Roy Fielding attempted to take the lessons learned from building a resilient web, and apply them to APIs. He dubbed this Representational Transfer of State or REST.

So far, things like CORBA, SOAP, and other RPC protocols were based on the faulty premise of defining with high precision the bits of data sent over the wire and back. Things that are highly precise are the easiest to break.

REST is based on the idea that you should send data but also information on how to consume the data. And by adopting some basic constraints, clients and servers can work out a lot of details through a more symbiotic set of machine + user interactions.

For example, sending a record for an order is valuable, but it’s even handier to send over related links, like the customer that ordered it, links to the catalog for each item, and links to the delivery tracking system.

Clients don’t have to use all of this extra data, but by providing enough self discovery, clients can adapt without suffering brittle updates.

The format of data can be dictated by media types, something that made it easy for browsers to handle HTML, image files, PDFs, etc. Browsers were coded once, long ago, to render a PDF document inline including a button to optionally save. Done and done. HTML pages are run through a different parser. Image files are similarly rendered without needing more and more upgrades to the browser. With a rich suite of standardized media types, web sites can evolve rapidly without requiring an update to the browser.

Did I mention machine + user interaction? Instead of requiring the client to consume links, it can instead display the links to the end user and let he or she actually click on them. We call this well known technique: hypermedia.

To version or not to version, that is the question!

HamletA question I get anytime I discuss Spring Data REST or Spring HATEOAS is versioning APIs. To quote Roy Fielding, don’t do it! People don’t version websites. Instead, they add new elements, and gradually implement the means to redirect old links to new pages. A better summary can be found in this interview with Roy Fielding on InfoQ.

When working on REST APIs and hypermedia, your probing question should be, “if this was a website viewed by a browser, would I handle it the same way?” If it sounds crazy in that context, then you’re probably going down the wrong path.

Imagine a record that includes both firstName and lastName, but you want to add fullName. Don’t rip out the old fields. Simply add new ones. You might have to implement some conversions and handlers to help older clients not yet using fullName, but that is worth the cost of avoiding brittle changes to existing clients. It reduces the friction.

In the event you need to REALLY make a big change to things, a simple version number doesn’t cut it. On the web, it’s called a new website. So release a new API at a new path and move on.

People clamor HARD for getting the super secret “id” field from a data record instead of using the “self” link. HINT: If you are pasting together URIs to talk to a REST service, something is wrong. It’s either your approach to consuming the API, or the service itself isn’t giving you any/enough links to navigate it.

When you get a URI, THAT is what you put into your web page, so the user can see the control and pick it. Your code doesn’t have to click it. Links are for users.

Fighting REST

fight1To this day, people are still fighting the concept of REST. Some have fallen in love with URIs that look like http://example.com/orders/523/lineitem/14 and http://example.com/orders/124/customer, thinking that these pretty URLs are the be-all/end-all of REST. Yet they code with RPC patterns.

In truth, formatting URLs this way, instead of as http://example.com/orders?q=523&li=14 or http://example.com/orders?q=124&mode=customer is to take advantage of HTTP caching when possible. A Good Idea(tm), but not a core tenet.

As a side effect, handing out JSON records with {orderId: 523} has forced clients to paste together links by hand. These links, not formed by the server, are brittle and just as bad as SOAP and CORBA, violating the whole reason REST was created. Does Amazon hand you the ISBN code for a book and expect you to enter into the “Buy It Now” button? No.

Many JavaScript frameworks have arisen, some quite popular. They claim to have REST support, yet people are coming on to chat channels asking how to get the “id” for a record so they can parse or assemble a URI.

BAD DESIGN SMELL! URIs are built by the server along with application state. If you clone state in the UI, you may end up replicating functionality and hence coupling things you never intended to.

Hopefully, I’ve laid out some the history and reasons that REST is what it is, and why learning what it’s meant to solve can help us all not reinvent the wheel of RPC.