Skip to content

Reinventing the WS Stack

As promised, in today’s post I will put forth my answer to Mike Champion’s question that he left in the comments section of my Day of the Dove post. To wit: “Finally, my suspicion is that if people start thinking hard about REST-like multi-hop or multi-protocol reliability / security / etc., they will end up reinventing something that looks an awful lot like the SOAP stack. I presume you disagree, but why?”

Executive Summary

I do disagree, but not dogmatically. The executive summary is as follows: People have been thinking hard about REST-like systems for quite some time. Roy’s dissertation was written in 2000, and though awareness of REST for machine-to-machine communication is still relatively new, the Web itself is largely RESTful, and in the dozen or so years that people have been industriously hacking away at these systems, the need for what is currently realized in the WS stack has either been addressed in a different manner or deemed not significant enough to address at all. And this is true even though any number of intermediaries may be positioned between client and server.

Should machine-to-machine, multi-hop, RESTful communications expose a need for additional functionality, then, and only then, will the need be addressed. This is opposed to the WS style of standards creation where solutions are created that go in search of problems. And when new solutions are created, they will be minimalist; the simplest thing that will possibly work and nothing more. They will leverage and extend existing standards whenever possible, and maintain and possibly add to the constraints imposed by the REST architectural style.

Protocol Jumping

Mike’s assertion is based on the premise that messages that flow across intermediaries and protocols will need message-level semantics in order to maintain integrity from end to end. On the face of it, this seems like a perfectly reasonable assertion, but is it?

Let’s start with the problem of crossing protocols. This is simply not an issue for HTTP-based, REST-styled systems (and now that I’ve proven I know the different between REST as architectural style and HTTP as an application protocol that is RESTful, I feel free to conflate the two as desired) for the simple reason that REST, as implemented in HTTP, does not attempt to be protocol agnostic. HTTP is the application protocol used today for RESTful applications, and it embodies many of the constraints that make REST work. In other words, if a system is pure REST, it doesn’t cross protocols.

Now, before you cry foul, let me expand. “Crossing protocols” is another way of saying “integration,” and integration is a problem all of its own. A problem we’ve been actively addressing since the second piece of code went live, and that we will be working on until the machines revolt and enslave humanity—and maybe even then. REST systems will need to integrate with legacy systems.

When integrating with existing systems, it is safe to assume that systems leveraging protocols other than HTTP are not RESTful systems, and that they do not anticipate that the inbound messages contain meta-information. Instead such a system would expect to find messages explicitly tailored to its needs, with meta-information, if any, bound up in the protocol. In this case, there’s no extra information to pass on: no message-level security, no message-level reliability, etc.

Newly designed systems (for the purposes of argument) are designed RESTfully. Today, that means HTTP. Thus, there are no protocols to bridge. If requirements dictate a messaging style that HTTP doesn’t address naturally, e.g. pub/sub, then you should use a MOM-based approach to this system’s design and not HTTP. Remember, use the right tool for the job.

Now the tricky part: What if you design a hybrid system: REST when desired, MOM when desired? For instance, suppose you want your partners to be able to use HTTP to send you a purchase order. On receipt of the PO, your design calls for it to be put onto a MOM-based topic to which are subscribed any number of other processes: for fulfillment, accounts payable, business activity monitoring, etc. Now, what if you want to maintain message-level information across that HTTP/MOM boundary? Do we now need to recreate any part of the WS stack?

The Web Services Framework

Well, lets take a look at the WS stack. First, for the purposes of this discussion, we only need to concern ourselves with those specs that have a representation in a SOAP message, meaning we can ignore things like WS-Policy (and family), BPEL, and WS-Management.

Of those specifications that impact an actual message, there is a class of specifications that define various messaging models:

  • WS-Addressing: asynchronous messaging
  • WS-ReliableMessaging: Point to point reliability
  • WS-Eventing: Event driven, non-brokered.
  • WS-Notification: Event driven, brokered pub/sub

(Of these, only WSA has been finalized, though WSRM is close. WS-Eventing and WS-Notification are likely to be combined into WS-EventNotification some time in the future.)

For the use case of protocol bridging, none of these specs needs to be recreated in a RESTful world. Remember, HTTP is the application (not transport) protocol used for RESTful systems today, to extend the messaging model used by HTTP by putting messaging semantics in the message is a non-starter as that simply relegates HTTP (and the MOM, in this case) to a dumb transport. The MOM, however, already supports all of these messaging styles, which is indeed the reason we are moving the message from HTTP to the MOM in the first place.

Now, should the REST/HTTP community desire to make messaging styles such as these available, for instance to make pub/sub available across the firewall, they will do so by leveraging HTTP’s existing features and/or extending HTTP as needed (possibly to the point of creating an entirely new protocol. For instance, HTTP can already do fire-and-forget and asynchronous messaging. The latter can be accomplished by having the server returns a 202-Accepted response and a URL in the Location: header that the client can poll until a response arrives. Alternately, to avoid polling, HTTP can be extended to introduce a “Reply-To” request header, that the client can send with its initial request.

Reliability is also addressable with ease. First, we’ll note that GET, PUT, and DELETE are idempotent (GET is also safe), meaning the client can call them repeatedly without fear, and thus they are intrinsically reliable; just call them until you get a response. POST, of course, is not idempotent, but if there is a desire to make it so you can, either by designing the resource to be idempotent anyway (and telling the client as much) or by minimally-and officially-extending HTTP. Here are two of such proposals: HTTPLR and POE.

Event-driven messaging is also trivial. In fact, if polling is sufficient, I would say that the millions of RSS/ATOM feeds out there are demonstrating the largest application of publish and subscribe ever seen. Something more formal might look like this: A resource is a topic or a queue, an event generator POSTs messages to a resource and event sinks GET them back off (or GET and DELETE them). PUT can be used to create topics and queues. POST to specified URLs can be used to subscribe and unsubscribe, or, if desired, HTTP can be extended with formal SUBSCRIBE and UNSUBSCRIBE methods.

Note: I am not advocating HTTP as a MOM replacement, only that the messaging styles are easy enough to implement if desired.

Now, I fully admit that all of the above messaging styles, while possible, currently exist by agreement between individual clients and servers. They are not standardized, and therefore not subject to common tooling. Today, it is not possible to simply tell a client that your resource is asynchronous, you have to describe how you’ve approached asynchrony so that client developers can write the appropriate code on their end. As such, should there be a desire to formally support these messaging styles in a common way, there will need to be standards created. This seems to bolster Mike’s point that the REST camp will need to eventually recreate the WS stack. It looks that way, but there are critical differences. Any new standards will be:

  • Created only when needed (if not enough people want these messaging styles, they won’t be created/accepted)
  • Broadly agreed upon, if not standardized
  • Succinctly specified
  • Work with HTTP, not against it (any attempt to put messaging semantics into the message itself subverts the underlying transport or application, rendering it dumb and suitable only for tunneling)

I will return to these messaging standards when I address the multi-hop issue.

Next on the stack of WS standards are those that actually affect the message without trying to implement messaging or other control functionality. In other words, they augment a message in such a way that is meaningful to keep this information with the message as it moves across transports. The poster child here is obviously WS-Security. It is entirely reasonable to design an application such that message-level encryption, digital signatures, or credentials are a requirement. Experience has shown that most people are content with transport-level security, but support for message-level security is still reasonable. Now, messages can be secured today in absence of any standard (as Amazon does it), but standards are always good. I concede that a REST-styled application can benefit from something like WS-Security and the various token profiles. But WSS won’t be reinvented, we’ll simply leverage WSS as it exists and thank the authors for all their hard work. Using WSS implies the use of an envelope as well, for which the SOAP envelope (and just the envelope) will do just fine. An Atom entry will do equally well. Note, that if you wanted message-level security and weren’t planning on crossing protocols, then HTTPSec is an approach to adding message level security right into HTTP.

The following specs I’m dismissing out of hand: WS-Transaction (and family) and WS-Transfer, WS-Enumeration, and the WSRF family of resource management specifications. The former because no one really wants the tight binding and state management associated with distributed transactions. And the latter because these specifications effectively re-implement HTTP (and I don’t know anything about grids).


With this background, the multi-hop problem is simple to address. Lets imagine we want to take advantage of one or more of the hypothetical messaging standards mentioned above, say asynchronous HTTP using a Reply-To: header, how can we be sure that the Reply-To information actually makes it through any intermediaries and on to the ultimate resource?

Simple, HTTP already has the notion of hop-by-hop and end-to-end headers. If the Reply-To header is end-to-end, then the (mythical) Asynchronous HTTP specification should mention this. But even if it doesn’t, HTTP denotes how proxies should forward header information: There’s a short list of hop-by-hop headers, all other headers are end-to-end. If HTTP is extended, and the header should be a hop-by-hop header, then it must be listed in the Connection: header as well. If a proxy doesn’t recognize a header field it must forward it on. Should an HTTP extension introduce new methods, e.g. SUBSCRIBE or UNSUBSCRIBE, then an existing proxy is likely to reject the request—heck some proxies in place today reject PUT and DELETE. In this case, the proxy will have to be configured to let the request through (or not). Finally, in the case of meta-information one actually wants to travel with the message, e.g. WS-Security, well, that’s already been addressed—we may want WSS, but we won’t reinvent it.


In summary, the REST community may or may not desire to implement some of the functionality found in the WS stack. But the multi-hop/multi-protocol issue is not going to be the impetus. Most of the WS stack is either outside of the message itself (e.g., WS-Policy), implementing various messaging styles (e.g. WSRM) or simply not of interest (WS-Transfer). In fact, this discourse has discovered only one specification, WS-Security, that describes functionality that a designer might want to travel with a message over protocol boundaries. And when it comes to messaging styles other than unreliable request/response, these are easily simulated with HTTP as it exists today, and HTTP is extensible and designed to be intermediated should that path prove desirable.

HTTP and REST Assholes, please correct anything I got wrong.

{ 7 } Comments

  1. Mark Baker | January 3, 2007 at 5:46 pm | Permalink

    Just wanted to point out that HTTP is already multi-hop, and there’s a ton of experience with that viz a viz caches, firewalls, surrogates; basically every intermediary configuration dreamed up by SOAP advocates and then some. HTTP doesn’t have a processing model that includes intermediary targetting, nor are intermediaries first class (not identified by URI, but by IP/port), but I’ll take feature-deficient-and-widely-interoperable over untested specware anyday. Anyhow, Waka’s supposed to address those deficiencies IIRC.

  2. Michael Champion | January 3, 2007 at 7:56 pm | Permalink

    Hmmm … I come away feeling that you more or less agree that a lot of the basic ideas of the WS stack (such as envelopes and the WS-Security model) would be preserved if it were rewritten by RESTifarians.

    I think the main way various parties to this discussion see the world differently is that some start from the Web and ask “why can’t the enterprisey people just do it the Web Way?” The WS-* folks start from the enterprisey world today(where reliable messaging Just Works, even across multiple nodes and across oceans, gigabytes a day) and ask “how can we achieve these service levels over the internet?” So, the hybrid system is the only one of interest, AFAIK, since you can indeed do things better with a MOM if you don’t have to worry about the Web.

    So I don’t understand your conclusion “But the multi-hop/multi-protocol issue is not going to be the impetus”. HTTP may be multihop already, but the people who have bet the farm on MQ for the last 30 years are *not* going to rip out their MOM and replace it with HTTP, so fuggidaboudit. (I know Pete isn’t advocating this).

    The nub of the issue is “all of the above messaging styles, while possible, currently exist by agreement between individual clients and servers. They are not standardized, and therefore not subject to common tooling.” SOAP, like XML, may not be elegant or optimal for much of anything, the question is whether it is good enough to provide a standardized tool platform that is better than all those ad hoc agreements that would be needed without a standard. I don’t work in that world anymore, so I don’t really know. But the enterprisey people that do know seem to be ignoring this whole debate and just using the WS stuff. I don’t believe that’s just because Gartner and IBM and Microsoft et al. tell them what to think, I suspect that many enterprisey types share the sentiments in “Pornography, gambling, lies, theft and terrorism: The Internet sucks”. They do NOT trust the Web, they spend their days trying to keep the mess out of their nice clean machine rooms.
    So, a lot of the points about how one might use HTTP in a reliable, secure, multi-hop way without WS-* seem technically valid, but pragmatically implausible.

    The bit I like best from this post is “WSS won’t be reinvented, we’ll simply leverage WSS as it exists and thank the authors for all their hard work.” That’s more or less the “Dove” answer that I was hoping for; people are more likely to agree on what to do than why to do it. Use what works, ignore the rest, let Father Darwin sort it all out in the long run. The real world Web isn’t purely RESTful, the real world enterprise needs to accommodate the Web to some degree, let’s quit arguing about abstract principles and discuss what works in which scenarios.

  3. Manuzhai | January 4, 2007 at 3:55 am | Permalink

    As an aside, I think the open XMPP standard (the basis for Jabber, standardized by the IETF in RFCs 3920 and 3921) should be(come) the de facto standard when thinking about MOM. It’s very easy to understand, yet supports a lot of the desired characteristics (such as PubSub, through XEP-0060).

  4. Stu Charlton | January 8, 2007 at 1:15 am | Permalink

    The issue with reusing WSS is that it’s for XML infosets only. MTOM extends this to other media types, but we still lack message-level integrity and encryption for entire HTTP representations. Since HTTP headers are the “authoratative” metadata, they too will need integrity protections, with something like HTTPSec.

    As for ripping out one’s MOM and replacing it with HTTP — I don’t see vendors rushing to commoditize their MOM implementations with highly reliable and scalable implementations of WS-RM. I think fronting a MOM with HTTP for better interoperability with certain consumer classes, is absolutely a pragmatic and sensible thing to do.

    IT folks are using SOAP happily, and will continue to (myself included)… but the wall likely is to occur when true IT investment begins on the next round of WS-* specs: WSDL 2.0, WS-ReliableMessaging, WS-Addressing, and WS-Policy (specifically WS-SecurityPolicy, and WS-ReliableMessagingPolicy). The explosion in complexity is staggering - particularly the mess of combining HTTP, WS-A, and WSDL 2.0, and for questionable benefit until we actually start seeing alternative transports take off, like SOAP/UDP or SOAP/MQ.

    We, as an industry of technologists, don’t have a good understanding of how hypermedia has transformed systems integration. Its organizational and social consequences are difficult to accept. It’s sort of like the advent of the microcomputer confusing the heck out of minicomputer & mainframers.

  5. fakelouisvuittonpurs | June 21, 2013 at 11:11 am | Permalink

    What’s up, I just wanted to say, you’re dead wrong. Your point Pete Lacey’s Weblog : Reinventing the WS Stack doesn’t make any sense. fake louis vuitton purses

  6. shoes toms for girls | June 21, 2013 at 5:34 pm | Permalink

    This is my first time pay a visit at here and i am in fact pleassant to read Pete Lacey’s Weblog : Reinventing the WS Stackeverthing at alone place. shoes toms for girls

  7. nasza strona | March 11, 2014 at 4:27 pm | Permalink

    I’m really enjoying the design and layout of your site.
    It’s a very easy on the eyes which makes it much more enjoyable for me to come here and visit more often.
    Did you hire out a developer to create your theme?

    Exceptional work!

{ 3 } Trackbacks

  1. Stefan Tilkov's Random Stuff | January 3, 2007 at 2:25 pm | Permalink

    (Not) Reinventing the WS Stack…

    Yet another excellent post by Pete Lacey. I’m in violent agreement and don’t have much to add; one thing I’ll note is that Mark has convinced me long ago that protocol independence is a bug, not a feature. I assume this is a sentiment…

  2. [...] I wasn’t looking for yet another “REST vs. Web Services” thread but Pete Lacey sucked me in (and many others) by hooking us with a hilarious bait post and since then he’s been pulling strongly on the line with very serious discussions on the topic so we haven’t been able to let go. The latest one left me a little puzzled though. In the security section Pete writes that it would make sense to use WS-Security indeed (and the SOAP envelope as a wrapper) if there was a need for message-level security rather than simply transport-level security. And then, barely catching his breath, he dismisses WS-Transfer and WS-Eventing in the following paragraph on the basis that “these specifications effectively re-implement HTTP” (not really true for WS-Enumeration but let’s leave that aside). More importantly, how am I to reconcile this with the previous paragraph? Once I use WS-Security and the SOAP envelope, I can’t use pure HTTP anymore. But the patterns supported by HTTP are still very useful. That’s what WS-Transfer is for. That’s what SOAP is for more generally, providing a hook-up point for things like WS-Security that composes with the rest of the message. I don’t understand how Pete can concede that in some cases message-level security is useful but then take away the possibility to do a GET in these circumstances. Is he saying that for some reason the scenarios that justify message-level security are scenarios in which REST-style interactions don’t apply? [...]

  3. [...] Reinventing the WS Stack (tags: rest webservices SOAP soa architecture by:pete_lacey) [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *