Pete Lacey’s Weblog

Oh man, what I wouldn’t give to attend this W3C Workshop. Nick Gall’s position paper is incendiary (via Paul):

It is my position that the W3C should extricate itself from further direct work on SOAP, WDSL, or any other WS-* specifications and redirect its resources into evangelizing and standardizing identifiers, formats, and protocols that exemplify Web architectural principles.

Wow. And then there’s Paul’s paper (who will actually be able to present):

We believe that rather than impose Web architecture on Web services, the W3C should strongly highlight the distinction between “Web Services” and “The Web” which whilst not being very compatible, may continue to be complementary.

So, who can get me in?

A few weeks ago I finished Seth Lloyd’s book Programming the Universe, which argues that the universe is one big quantum computer. Here is an especially apt paragraph:

… complexity is a key issue in engineering. How can we engineer complex systems that are still robust in their behavior? The maxim we teach engineering undergraduates at MIT goes by the acronym KISS: Keep It Simple, Stupid! But what if the system you’re engineering is itself complex, like an airplane? …. One promising technique for engineering complex systems is known as axiomatic design. …. The idea of axiomatic design is to minimize the information content of the engineered system while maintaining its ability to carry out its functional requirements. Properly applied, axiomatic design results in airplanes, software, and toasters all just complex enough, and no more, to attain their design goals. Axiomatic design minimizes the effective complexity of the engineered system while maintaining the system’s effectiveness. Keep It Simple, Stupid—but not too simple.

And if you follow the link I supplied for axiomatic design, you find this intriguing statement:

The Information Axiom requires the minimization of the information content. According to the Information Axiom, the design that has the least information content is the best design.

Take away from this what you want.

Oh, and a capsule book review: Interesting idea, could’ve been better argued.

Comments continue to trickle in on “The S Stands for Simple” post. Just yesterday someone named Yamamoto Yohei dropped by to say:

I’ve translated this great post into Japanese.

http://yohei-y.blogspot.com/2007/01/s-s.html

Now, I know this kind of thing happens all the time, but still, I’m tickled pink. Is this a minor example of what Sam was talking about yesterday: manufactured serendipity?

Yamamoto’s comment also made me realize that I have no explicit licensing clause on this site. So I’ve added that to the bottom of each page. I chose to use the Creative Commons Attribution License. This is their most liberal of licenses and basically says “feel free to do whatever you want with this copyrighted work, but don’t try and pass it off as your own.” Not for a second did I consider any more restrictive license. I mean, it’s a blog. But then Mark Pilgrim has an undue influence over the way I think.

Just in case these went unnoticed:

Duncan Cragg has put up part 3 of  “The REST Dialogues.” I can’t say I’m in 100% agreement with Duncan. In fact, I have difficulty with the same things that “ocean” over at discipline and punish does.

Paul Downey has an absolutely perfect REST presentation available: “Web APIs Are Just Web Sites.”

Dave Orchard’s most recent post states that I mistakingly conflated SOAP and XML Schema in my InfoQ interview. Specifically where I note that SOAP endpoints tend to blow up when the messages aren’t constructed exactly as expected. Dave says that what I call a SOAP issue is really an XML Schema issue. I tried to comment on his blog, but after more than 24 hours the comment hasn’t made it through moderation, so I’ll comment here instead.

First, I don’t think I conflated anything. At that point in the interview I was talking about XML Schema (as used by WSDL). More accurately I was talking about how SOAP toolkits make use of WSDL and Schema information. But even if I had confused SOAP and XSD, what difference would it make when the WS-I Basic Profile “mandates the use of XML Schema as the type system for WSDL descriptions of Web Services?” (Emphasis mine.) And, yes, I know that SOAP doesn’t mandate the use of WSDL, but the Basic Profile does.

But really, this is not a failing of SOAP or WSDL or XML Schema. I can have a WSDL/XSD described SOAP service that doesn’t suffer from the problem of being too brittle. All I have to do is not use any of the SOAP platforms currently available Update: WCF excepted. (Axis 2 might be excepted too, must check). It’s the tools, that on receipt of a message try to serialize it into an object, that cause this problem. As I said in the interview, “this style of SOAP development reinforces the SOAP as RPC/distributed-component/serialized-object perspective and relegates XML to just another serialization framework.

Dave goes on to state that I’m also mistaken in my assertion that SOAP web services are too tightly coupled, and links to Dan Pritchett’s post along the same lines. But I’ve already answered that, and I stand by my claim.

I’ve been otherwise occupied for the last two weeks or so, but there’s been some interesting feedback on my Day of the Dove post and it’s related successors, Strategic Direction and Reinventing the WS Stack. My apologies for letting them linger unanswered for so long. I’ll respond to some of that feedback here and now, and update the positioning accordingly.

First, there is this fantastic post from Don Box (and a follow up) regarding how WCF is not promoting WS-* as a strategic direction for Microsoft, but that it is an open and extensible platform. Since this echoes what Mike Champion has to say, and recognizing the authority of these two Microsoft engineers, I concede this point. I’m not sure this jives with Microsoft’s more public presence, but my thinking is probably stuck in the early Indigo days. It’s more difficult to say the same for IBM given the plethora of WS/SOA related products they offer, but I will strike the whole point from my positioning statement: it is not necessary to adopt SOAP/SOA/WS in order to stay in step with the product direction of many large vendors.

Next comes Mike, who had a long response to “Reinventing the WS stack.” To be honest, I’m not sure if Mike has a final conclusion, so I’ll excerpt the interesting bits piecemeal.

Hmmm … I come away feeling that you more or less agree that a lot of the basic ideas of the WS stack (such as envelopes and the WS-Security model) would be preserved if it were rewritten by RESTifarians.

This is the bit I have the most difficulty parsing. I certainly don’t speak for the REST community (if there is such a thing) but I imagine that no REST proponent is even remotely interested in rewriting the WS stack. What I was getting at, in the context of the original question, is that if you examine the entire stack, there’s potential value in WS-Security, and since that requires (?) an envelope, the SOAP envelope will work. But helping oneself to a couple of good ideas does not constitute agreement with the basic idea of the WS stack. In fact, I’m not sure I can say what the basic ideas of the WS stack are. It seems to be a dumping ground for every conceivable technology. By Burton Group’s latest count there are more than 50 WS-* specifications. Some of the security related specs may be useful to the REST community, but not much else.

I think the main way various parties to this discussion see the world differently is that some start from the Web and ask “why can’t the enterprisey people just do it the Web Way?” The WS-* folks start from the enterprisey world today (where reliable messaging Just Works, even across multiple nodes and across oceans, gigabytes a day) and ask “how can we achieve these service levels over the internet?” So, the hybrid system is the only one of interest, AFAIK, since you can indeed do things better with a MOM if you don’t have to worry about the Web. (Emphasis mine.)

Exactly! For an example of the first view of the world see this recent post by Mark Baker.

To restate: The Web has reach and MOMs are enterprisey. What then does WS-* have? Today, not much. Basic SOAP processing doesn’t offer much, and the advanced WS functionality is not widely implemented, nor does it have much, if any, production use. Also note that implementing, say, WS-ReliableMessaging, does not necessarily mean that the implementation itself is robust - it might implement the spec perfectly, but still drop every other message on the floor. WebSphere MQ, on the other hand has been around in one form or another for 30-odd years. It’s proven, high performing, fault tolerant, transactional, etc. It would be madness to toss it aside. Thus, one can imagine a pure REST system, a pure MQ system, or a hybrid of both.

Though I have a hard time imagining what WS-* offers, WS proponents will argue that WS-* offers the following:

Platform neutrality: For the first time ever, all major vendors are backing a single “remoting” technology. No more DCOM over here and CORBA over there. That’s great! However, this is really the second time. The first was when they embraced HTTP.

Common messaging standard: MOMs use binary messaging. MOMs don’t interoperate. With WS-* you have a messaging/middleware solution that is open, interoperable, and easily intermediated. You can put a series of intermediaries in the message path that apply organizational policies and processes centrally. This is a very compelling argument that looks great on paper. Today, though, even basic SOAP/WSDL interoperability remains challenging, let alone the rest of the WS stack. And the practice of consistently applying corporate and technology policies via an intermediary is still in its infancy. Furthermore, to me anyway, mediation on this scale fails the sniff test. For one thing it smacks of moving intelligence into the network, generally a bad thing. For another, it reeks of inefficient, top-down control that will either be ignored in practice or stifle innovation. And for a third, these intermediaries are anything but transparent (they can’t be without a uniform interface) to either the client and the server, burdening developers on both sides with intimate knowledge of what the intermediaries are doing.

However, the jury’s still out, and I’ll put this down as a possible future win for WS. Of course, HTTP is also designed for intermediation, and there is a wealth of experience building proxy servers and caches and security solutions. Stuff you likely already own. An HTTP proxy can’t intermediate an MQ message transfer, true, but I’m pretty sure I don’t want it to.

So I don’t understand your conclusion “But the multi-hop/multi-protocol issue is not going to be the impetus”. HTTP may be multihop already, but the people who have bet the farm on MQ for the last 30 years are *not* going to rip out their MOM and replace it with HTTP, so fuggidaboudit. (I know Pete isn’t advocating this).

I’m certainly not advocating ripping out a MOM. What I meant by the “impetus” statement is that the RESTafarians may want to mimic some of the messaging styles found in MOMs and WS-*, e.g. async or pub/sub. But if they do, it won’t be because of challenges presented crossing protocol or server boundaries, but rather because these messaging styles are attractive. For clarity’s sake, note that implementing a messaging style like pub/sub, does not imply implementing a robust, fault-tolerant, etc. MOM, it’s just another messaging style alongside HTTP’s existing synchronous, request/response style.

SOAP, like XML, may not be elegant or optimal for much of anything, the question is whether it is good enough to provide a standardized tool platform that is better than all those ad hoc agreements that would be needed without a standard.

No, the question is whether these standards are needed at all. Remember, the REST folks have not implemented any of the mythical messaging styles I put down, and they may never do so. Whereas the WS camp has gone ahead and created specs (and very few implementations) for just about everything, whether people wanted it or not. The same is true when you go beyond messaging and into management, configuration, discovery, metadata, orchestration, choreography, transactions, federation, trust, and so on. Who asked for all this stuff?

“Pornography, gambling, lies, theft and terrorism: The Internet sucks”. They do NOT trust the Web, they spend their days trying to keep the mess out of their nice clean machine rooms. So, a lot of the points about how one might use HTTP in a reliable, secure, multi-hop way without WS-* seem technically valid, but pragmatically implausible.They do NOT trust the Web, they spend their days trying to keep the mess out of their nice clean machine rooms.

I’m sorry, I can’t buy this one at all. To conflate HTTP with a number of dubious web sites is just plain wrong. If there’s a jenna_jameson resource whose representation happens to be a jpeg, how is that the fault of HTTP? What if there’s a SOAP service for gambling online? Besides, what protocol do you think SOAP is tunneling over?

Use what works, ignore the rest, let Father Darwin sort it all out in the long run. The real world Web isn’t purely RESTful, the real world enterprise needs to accommodate the Web to some degree, let’s quit arguing about abstract principles and discuss what works in which scenarios.

Another point of agreement and the whole point of the original dovish post. Alas, we now have one less reason to use SOAP/WS today.

I also need to highlight Stu Charlton’s comment, if only to say that I am in 100% agreement.

Incorporating this feedback the new positioning is as follows :

• Your default messaging style should be REST. This allows for data to be created and consumed by the largest possible audience. Before considering any other distributed architecture, one should have valid reasons for discounting REST.
• If asynchronous, reliable messaging is absolutely required, use a MOM.
• If event driven or broadcast messaging is absolutely required, use a MOM.
• If pub/sub is absolutely required, use a MOM.
• If you have to integrate with existing WS interfaces, use SOAP/WS-*.

That doesn’t leave much for WS-* to do to except integrate with other, existing WS-* systems.  Of course, standards-based, interoperable, readily intermediated, advanced messaging is still the WS-WildCard.  We’ll see if it pans out.  Though I remain dubious.

As promised, in today’s post I will put forth my answer to Mike Champion’s question that he left in the comments section of my Day of the Dove post. To wit: “Finally, my suspicion is that if people start thinking hard about REST-like multi-hop or multi-protocol reliability / security / etc., they will end up reinventing something that looks an awful lot like the SOAP stack. I presume you disagree, but why?”

Executive Summary

I do disagree, but not dogmatically. The executive summary is as follows: People have been thinking hard about REST-like systems for quite some time. Roy’s dissertation was written in 2000, and though awareness of REST for machine-to-machine communication is still relatively new, the Web itself is largely RESTful, and in the dozen or so years that people have been industriously hacking away at these systems, the need for what is currently realized in the WS stack has either been addressed in a different manner or deemed not significant enough to address at all. And this is true even though any number of intermediaries may be positioned between client and server.

Should machine-to-machine, multi-hop, RESTful communications expose a need for additional functionality, then, and only then, will the need be addressed. This is opposed to the WS style of standards creation where solutions are created that go in search of problems. And when new solutions are created, they will be minimalist; the simplest thing that will possibly work and nothing more. They will leverage and extend existing standards whenever possible, and maintain and possibly add to the constraints imposed by the REST architectural style.

Protocol Jumping

Mike’s assertion is based on the premise that messages that flow across intermediaries and protocols will need message-level semantics in order to maintain integrity from end to end. On the face of it, this seems like a perfectly reasonable assertion, but is it?

Let’s start with the problem of crossing protocols. This is simply not an issue for HTTP-based, REST-styled systems (and now that I’ve proven I know the different between REST as architectural style and HTTP as an application protocol that is RESTful, I feel free to conflate the two as desired) for the simple reason that REST, as implemented in HTTP, does not attempt to be protocol agnostic. HTTP is the application protocol used today for RESTful applications, and it embodies many of the constraints that make REST work. In other words, if a system is pure REST, it doesn’t cross protocols.

Now, before you cry foul, let me expand. “Crossing protocols” is another way of saying “integration,” and integration is a problem all of its own. A problem we’ve been actively addressing since the second piece of code went live, and that we will be working on until the machines revolt and enslave humanity—and maybe even then. REST systems will need to integrate with legacy systems.

When integrating with existing systems, it is safe to assume that systems leveraging protocols other than HTTP are not RESTful systems, and that they do not anticipate that the inbound messages contain meta-information. Instead such a system would expect to find messages explicitly tailored to its needs, with meta-information, if any, bound up in the protocol. In this case, there’s no extra information to pass on: no message-level security, no message-level reliability, etc.

Newly designed systems (for the purposes of argument) are designed RESTfully. Today, that means HTTP. Thus, there are no protocols to bridge. If requirements dictate a messaging style that HTTP doesn’t address naturally, e.g. pub/sub, then you should use a MOM-based approach to this system’s design and not HTTP. Remember, use the right tool for the job.

Now the tricky part: What if you design a hybrid system: REST when desired, MOM when desired? For instance, suppose you want your partners to be able to use HTTP to send you a purchase order. On receipt of the PO, your design calls for it to be put onto a MOM-based topic to which are subscribed any number of other processes: for fulfillment, accounts payable, business activity monitoring, etc. Now, what if you want to maintain message-level information across that HTTP/MOM boundary? Do we now need to recreate any part of the WS stack?

The Web Services Framework

Well, lets take a look at the WS stack. First, for the purposes of this discussion, we only need to concern ourselves with those specs that have a representation in a SOAP message, meaning we can ignore things like WS-Policy (and family), BPEL, and WS-Management.

Of those specifications that impact an actual message, there is a class of specifications that define various messaging models:

• WS-Addressing: asynchronous messaging
• WS-ReliableMessaging: Point to point reliability
• WS-Eventing: Event driven, non-brokered.
• WS-Notification: Event driven, brokered pub/sub

(Of these, only WSA has been finalized, though WSRM is close. WS-Eventing and WS-Notification are likely to be combined into WS-EventNotification some time in the future.)

For the use case of protocol bridging, none of these specs needs to be recreated in a RESTful world. Remember, HTTP is the application (not transport) protocol used for RESTful systems today, to extend the messaging model used by HTTP by putting messaging semantics in the message is a non-starter as that simply relegates HTTP (and the MOM, in this case) to a dumb transport. The MOM, however, already supports all of these messaging styles, which is indeed the reason we are moving the message from HTTP to the MOM in the first place.

Now, should the REST/HTTP community desire to make messaging styles such as these available, for instance to make pub/sub available across the firewall, they will do so by leveraging HTTP’s existing features and/or extending HTTP as needed (possibly to the point of creating an entirely new protocol. For instance, HTTP can already do fire-and-forget and asynchronous messaging. The latter can be accomplished by having the server returns a 202-Accepted response and a URL in the Location: header that the client can poll until a response arrives. Alternately, to avoid polling, HTTP can be extended to introduce a “Reply-To” request header, that the client can send with its initial request.

Reliability is also addressable with ease. First, we’ll note that GET, PUT, and DELETE are idempotent (GET is also safe), meaning the client can call them repeatedly without fear, and thus they are intrinsically reliable; just call them until you get a response. POST, of course, is not idempotent, but if there is a desire to make it so you can, either by designing the resource to be idempotent anyway (and telling the client as much) or by minimally-and officially-extending HTTP. Here are two of such proposals: HTTPLR and POE.

Event-driven messaging is also trivial. In fact, if polling is sufficient, I would say that the millions of RSS/ATOM feeds out there are demonstrating the largest application of publish and subscribe ever seen. Something more formal might look like this: A resource is a topic or a queue, an event generator POSTs messages to a resource and event sinks GET them back off (or GET and DELETE them). PUT can be used to create topics and queues. POST to specified URLs can be used to subscribe and unsubscribe, or, if desired, HTTP can be extended with formal SUBSCRIBE and UNSUBSCRIBE methods.

Note: I am not advocating HTTP as a MOM replacement, only that the messaging styles are easy enough to implement if desired.

Now, I fully admit that all of the above messaging styles, while possible, currently exist by agreement between individual clients and servers. They are not standardized, and therefore not subject to common tooling. Today, it is not possible to simply tell a client that your resource is asynchronous, you have to describe how you’ve approached asynchrony so that client developers can write the appropriate code on their end. As such, should there be a desire to formally support these messaging styles in a common way, there will need to be standards created. This seems to bolster Mike’s point that the REST camp will need to eventually recreate the WS stack. It looks that way, but there are critical differences. Any new standards will be:

• Created only when needed (if not enough people want these messaging styles, they won’t be created/accepted)
• Broadly agreed upon, if not standardized
• Succinctly specified
• Work with HTTP, not against it (any attempt to put messaging semantics into the message itself subverts the underlying transport or application, rendering it dumb and suitable only for tunneling)

I will return to these messaging standards when I address the multi-hop issue.

Next on the stack of WS standards are those that actually affect the message without trying to implement messaging or other control functionality. In other words, they augment a message in such a way that is meaningful to keep this information with the message as it moves across transports. The poster child here is obviously WS-Security. It is entirely reasonable to design an application such that message-level encryption, digital signatures, or credentials are a requirement. Experience has shown that most people are content with transport-level security, but support for message-level security is still reasonable. Now, messages can be secured today in absence of any standard (as Amazon does it), but standards are always good. I concede that a REST-styled application can benefit from something like WS-Security and the various token profiles. But WSS won’t be reinvented, we’ll simply leverage WSS as it exists and thank the authors for all their hard work. Using WSS implies the use of an envelope as well, for which the SOAP envelope (and just the envelope) will do just fine. An Atom entry will do equally well. Note, that if you wanted message-level security and weren’t planning on crossing protocols, then HTTPSec is an approach to adding message level security right into HTTP.

The following specs I’m dismissing out of hand: WS-Transaction (and family) and WS-Transfer, WS-Enumeration, and the WSRF family of resource management specifications. The former because no one really wants the tight binding and state management associated with distributed transactions. And the latter because these specifications effectively re-implement HTTP (and I don’t know anything about grids).

Multi-Hop

With this background, the multi-hop problem is simple to address. Lets imagine we want to take advantage of one or more of the hypothetical messaging standards mentioned above, say asynchronous HTTP using a Reply-To: header, how can we be sure that the Reply-To information actually makes it through any intermediaries and on to the ultimate resource?

Simple, HTTP already has the notion of hop-by-hop and end-to-end headers. If the Reply-To header is end-to-end, then the (mythical) Asynchronous HTTP specification should mention this. But even if it doesn’t, HTTP denotes how proxies should forward header information: There’s a short list of hop-by-hop headers, all other headers are end-to-end. If HTTP is extended, and the header should be a hop-by-hop header, then it must be listed in the Connection: header as well. If a proxy doesn’t recognize a header field it must forward it on. Should an HTTP extension introduce new methods, e.g. SUBSCRIBE or UNSUBSCRIBE, then an existing proxy is likely to reject the request—heck some proxies in place today reject PUT and DELETE. In this case, the proxy will have to be configured to let the request through (or not). Finally, in the case of meta-information one actually wants to travel with the message, e.g. WS-Security, well, that’s already been addressed—we may want WSS, but we won’t reinvent it.

Summary

In summary, the REST community may or may not desire to implement some of the functionality found in the WS stack. But the multi-hop/multi-protocol issue is not going to be the impetus. Most of the WS stack is either outside of the message itself (e.g., WS-Policy), implementing various messaging styles (e.g. WSRM) or simply not of interest (WS-Transfer). In fact, this discourse has discovered only one specification, WS-Security, that describes functionality that a designer might want to travel with a message over protocol boundaries. And when it comes to messaging styles other than unreliable request/response, these are easily simulated with HTTP as it exists today, and HTTP is extensible and designed to be intermediated should that path prove desirable.

HTTP and REST Assholes, please correct anything I got wrong.