Pete Lacey’s Weblog

As promised, in today’s post I will put forth my answer to Mike Champion’s question that he left in the comments section of my Day of the Dove post. To wit: “Finally, my suspicion is that if people start thinking hard about REST-like multi-hop or multi-protocol reliability / security / etc., they will end up reinventing something that looks an awful lot like the SOAP stack. I presume you disagree, but why?”

Executive Summary

I do disagree, but not dogmatically. The executive summary is as follows: People have been thinking hard about REST-like systems for quite some time. Roy’s dissertation was written in 2000, and though awareness of REST for machine-to-machine communication is still relatively new, the Web itself is largely RESTful, and in the dozen or so years that people have been industriously hacking away at these systems, the need for what is currently realized in the WS stack has either been addressed in a different manner or deemed not significant enough to address at all. And this is true even though any number of intermediaries may be positioned between client and server.

Should machine-to-machine, multi-hop, RESTful communications expose a need for additional functionality, then, and only then, will the need be addressed. This is opposed to the WS style of standards creation where solutions are created that go in search of problems. And when new solutions are created, they will be minimalist; the simplest thing that will possibly work and nothing more. They will leverage and extend existing standards whenever possible, and maintain and possibly add to the constraints imposed by the REST architectural style.

Protocol Jumping

Mike’s assertion is based on the premise that messages that flow across intermediaries and protocols will need message-level semantics in order to maintain integrity from end to end. On the face of it, this seems like a perfectly reasonable assertion, but is it?

Let’s start with the problem of crossing protocols. This is simply not an issue for HTTP-based, REST-styled systems (and now that I’ve proven I know the different between REST as architectural style and HTTP as an application protocol that is RESTful, I feel free to conflate the two as desired) for the simple reason that REST, as implemented in HTTP, does not attempt to be protocol agnostic. HTTP is the application protocol used today for RESTful applications, and it embodies many of the constraints that make REST work. In other words, if a system is pure REST, it doesn’t cross protocols.

Now, before you cry foul, let me expand. “Crossing protocols” is another way of saying “integration,” and integration is a problem all of its own. A problem we’ve been actively addressing since the second piece of code went live, and that we will be working on until the machines revolt and enslave humanity—and maybe even then. REST systems will need to integrate with legacy systems.

When integrating with existing systems, it is safe to assume that systems leveraging protocols other than HTTP are not RESTful systems, and that they do not anticipate that the inbound messages contain meta-information. Instead such a system would expect to find messages explicitly tailored to its needs, with meta-information, if any, bound up in the protocol. In this case, there’s no extra information to pass on: no message-level security, no message-level reliability, etc.

Newly designed systems (for the purposes of argument) are designed RESTfully. Today, that means HTTP. Thus, there are no protocols to bridge. If requirements dictate a messaging style that HTTP doesn’t address naturally, e.g. pub/sub, then you should use a MOM-based approach to this system’s design and not HTTP. Remember, use the right tool for the job.

Now the tricky part: What if you design a hybrid system: REST when desired, MOM when desired? For instance, suppose you want your partners to be able to use HTTP to send you a purchase order. On receipt of the PO, your design calls for it to be put onto a MOM-based topic to which are subscribed any number of other processes: for fulfillment, accounts payable, business activity monitoring, etc. Now, what if you want to maintain message-level information across that HTTP/MOM boundary? Do we now need to recreate any part of the WS stack?

The Web Services Framework

Well, lets take a look at the WS stack. First, for the purposes of this discussion, we only need to concern ourselves with those specs that have a representation in a SOAP message, meaning we can ignore things like WS-Policy (and family), BPEL, and WS-Management.

Of those specifications that impact an actual message, there is a class of specifications that define various messaging models:

• WS-Addressing: asynchronous messaging
• WS-ReliableMessaging: Point to point reliability
• WS-Eventing: Event driven, non-brokered.
• WS-Notification: Event driven, brokered pub/sub

(Of these, only WSA has been finalized, though WSRM is close. WS-Eventing and WS-Notification are likely to be combined into WS-EventNotification some time in the future.)

For the use case of protocol bridging, none of these specs needs to be recreated in a RESTful world. Remember, HTTP is the application (not transport) protocol used for RESTful systems today, to extend the messaging model used by HTTP by putting messaging semantics in the message is a non-starter as that simply relegates HTTP (and the MOM, in this case) to a dumb transport. The MOM, however, already supports all of these messaging styles, which is indeed the reason we are moving the message from HTTP to the MOM in the first place.

Now, should the REST/HTTP community desire to make messaging styles such as these available, for instance to make pub/sub available across the firewall, they will do so by leveraging HTTP’s existing features and/or extending HTTP as needed (possibly to the point of creating an entirely new protocol. For instance, HTTP can already do fire-and-forget and asynchronous messaging. The latter can be accomplished by having the server returns a 202-Accepted response and a URL in the Location: header that the client can poll until a response arrives. Alternately, to avoid polling, HTTP can be extended to introduce a “Reply-To” request header, that the client can send with its initial request.

Reliability is also addressable with ease. First, we’ll note that GET, PUT, and DELETE are idempotent (GET is also safe), meaning the client can call them repeatedly without fear, and thus they are intrinsically reliable; just call them until you get a response. POST, of course, is not idempotent, but if there is a desire to make it so you can, either by designing the resource to be idempotent anyway (and telling the client as much) or by minimally-and officially-extending HTTP. Here are two of such proposals: HTTPLR and POE.

Event-driven messaging is also trivial. In fact, if polling is sufficient, I would say that the millions of RSS/ATOM feeds out there are demonstrating the largest application of publish and subscribe ever seen. Something more formal might look like this: A resource is a topic or a queue, an event generator POSTs messages to a resource and event sinks GET them back off (or GET and DELETE them). PUT can be used to create topics and queues. POST to specified URLs can be used to subscribe and unsubscribe, or, if desired, HTTP can be extended with formal SUBSCRIBE and UNSUBSCRIBE methods.

Note: I am not advocating HTTP as a MOM replacement, only that the messaging styles are easy enough to implement if desired.

Now, I fully admit that all of the above messaging styles, while possible, currently exist by agreement between individual clients and servers. They are not standardized, and therefore not subject to common tooling. Today, it is not possible to simply tell a client that your resource is asynchronous, you have to describe how you’ve approached asynchrony so that client developers can write the appropriate code on their end. As such, should there be a desire to formally support these messaging styles in a common way, there will need to be standards created. This seems to bolster Mike’s point that the REST camp will need to eventually recreate the WS stack. It looks that way, but there are critical differences. Any new standards will be:

• Created only when needed (if not enough people want these messaging styles, they won’t be created/accepted)
• Broadly agreed upon, if not standardized
• Succinctly specified
• Work with HTTP, not against it (any attempt to put messaging semantics into the message itself subverts the underlying transport or application, rendering it dumb and suitable only for tunneling)

I will return to these messaging standards when I address the multi-hop issue.

Next on the stack of WS standards are those that actually affect the message without trying to implement messaging or other control functionality. In other words, they augment a message in such a way that is meaningful to keep this information with the message as it moves across transports. The poster child here is obviously WS-Security. It is entirely reasonable to design an application such that message-level encryption, digital signatures, or credentials are a requirement. Experience has shown that most people are content with transport-level security, but support for message-level security is still reasonable. Now, messages can be secured today in absence of any standard (as Amazon does it), but standards are always good. I concede that a REST-styled application can benefit from something like WS-Security and the various token profiles. But WSS won’t be reinvented, we’ll simply leverage WSS as it exists and thank the authors for all their hard work. Using WSS implies the use of an envelope as well, for which the SOAP envelope (and just the envelope) will do just fine. An Atom entry will do equally well. Note, that if you wanted message-level security and weren’t planning on crossing protocols, then HTTPSec is an approach to adding message level security right into HTTP.

The following specs I’m dismissing out of hand: WS-Transaction (and family) and WS-Transfer, WS-Enumeration, and the WSRF family of resource management specifications. The former because no one really wants the tight binding and state management associated with distributed transactions. And the latter because these specifications effectively re-implement HTTP (and I don’t know anything about grids).

Multi-Hop

With this background, the multi-hop problem is simple to address. Lets imagine we want to take advantage of one or more of the hypothetical messaging standards mentioned above, say asynchronous HTTP using a Reply-To: header, how can we be sure that the Reply-To information actually makes it through any intermediaries and on to the ultimate resource?

Simple, HTTP already has the notion of hop-by-hop and end-to-end headers. If the Reply-To header is end-to-end, then the (mythical) Asynchronous HTTP specification should mention this. But even if it doesn’t, HTTP denotes how proxies should forward header information: There’s a short list of hop-by-hop headers, all other headers are end-to-end. If HTTP is extended, and the header should be a hop-by-hop header, then it must be listed in the Connection: header as well. If a proxy doesn’t recognize a header field it must forward it on. Should an HTTP extension introduce new methods, e.g. SUBSCRIBE or UNSUBSCRIBE, then an existing proxy is likely to reject the request—heck some proxies in place today reject PUT and DELETE. In this case, the proxy will have to be configured to let the request through (or not). Finally, in the case of meta-information one actually wants to travel with the message, e.g. WS-Security, well, that’s already been addressed—we may want WSS, but we won’t reinvent it.

Summary

In summary, the REST community may or may not desire to implement some of the functionality found in the WS stack. But the multi-hop/multi-protocol issue is not going to be the impetus. Most of the WS stack is either outside of the message itself (e.g., WS-Policy), implementing various messaging styles (e.g. WSRM) or simply not of interest (WS-Transfer). In fact, this discourse has discovered only one specification, WS-Security, that describes functionality that a designer might want to travel with a message over protocol boundaries. And when it comes to messaging styles other than unreliable request/response, these are easily simulated with HTTP as it exists today, and HTTP is extensible and designed to be intermediated should that path prove desirable.

HTTP and REST Assholes, please correct anything I got wrong.