Pete Lacey’s Weblog

Maybe posting just before the holiday break wasn’t such a good idea. It meant ignoring some excellent comments for far too long. Mike Champion, in particular, had some interesting things to say, which I’m surfacing in this post and the next.

In my previous post on positioning messaging technologies I stated that one reason to use SOAP/WS was to stay in step with the product direction of certain large vendors such as Microsoft or IBM. Mike, a Microsoft employee, says, “Speaking only for my little corner of Microsoft, I don’t think the company as a whole takes sides [in the REST vs SOAP debate]…” (Read the full comment).

That is good to hear, and I’ve seen Don Box and Dare Obasanjo make similar claims. In fact, I would love to hear that MS has fully embraced both technologies and is not biased towards one or the other, but, to be honest, I’m not convinced.

Yes, Microsoft is a big company working on many different products. And the product teams probably have the leeway to make pragmatic choices, such as supporting REST instead of (or as well as) SOAP. But from the outside looking in, and ignoring what Microsoft is doing with its online properties, all I can see is the Windows Communication Foundation! Its name says it all. The WCF is the preferred mechanism for remote communication. Indeed it is the very foundation of Windows communication. And it is SOAP/WS. Sure, I’m aware that you can create RESTful apps with WCF, and I appreciate that this is even possible, but it’s hardly front and center. For instance, here’s how Microsoft describes the WCF . Now look at the very last paragraph. To me that shows a definite bias.

I want to be proved wrong. I want to be pointed to the web pages that show REST support (and not just HTTP verbs and URIs) in Longhorn Server and Visual Studio and BizTalk and SQL Server. I want to see high-level support for content negotiation and ETags and permanent redirects (as Sam said in his comment on Dare’s post). The fact is, though, that Microsoft has made a huge bet on SOAP web services, even co-authoring nearly every spec that matters. And while the HTTP stack may be accessible through WCF, it hardly seems that REST is part of Microsoft’s-strategic direction.

Not to pick on Microsoft; the same is true for IBM and others. Heck, IBM employs many of the leading proponents of REST: Sam Ruby (or have you moved on, Sam?), James Snell, Mark Pilgrim, Joe Gregorio, and more. Yet, based on IBM’s product releases and sales and marketing efforts, I think it’s safe to sat that they are backing SOAP and SOA all the way. And this is not necessarily a critique either; some very smart people believe that SOAP/SOA is the way of the future, so there’s no reason that companies like MS and IBM shouldn’t place a huge bet there. I don’t believe it will pay off all that handsomely, but there’s no denying there’s a lot of money on the table.

After the new year I’ll address Mike’s other point: “Finally, my suspicion is that if people start thinking hard about REST-like multi-hop or multi-protocol reliability / security / etc., they will end up reinventing something that looks an awful lot like the SOAP stack. I presume you disagree, but why?”

Update: In the comments, Frank Greco correctly points out that I have miscategorized Tuple Spaces technology. On his web log, Dan Creswell implies the same. This was pure laziness on my part. I wanted to give Tuple Spaces a mention because of the renewed interest it seems to be receiving, and I added it hastily to the message-oriented middleware category, when, in fact, Tuple Spaces implement a distributed shared memory model. In this light, it seems to me that Tuple Spaces are not designed to address the problem set up in this post: making business logic available over the network, and I have removed this technology from the list of potential solutions. Note to self; download Blitz and get smart.

Do you remember that Star Trek TOS episode where a hyper-intelligent shade of the color red locks a Klingon crew and the Enterprise crew in unending violent combat so that it can feed off their heightened emotions? Remind you of anything?

If the RESTafarians are the Klingons (I’m being nice) and the SOAPies are the Federation, who is the alien? Who the heck is getting off on this? I suspect, that we all are. Recently, however, a number of voices have been trying to rise above the din and laugh the alien off the Enterprise. Or, as Kirk put it, “Captain’s log, stardate… Armageddon. We must find a way to defeat the alien force of hate that has taken over the Enterprise, stop the war now, or spend eternity in futile, bloody violence.”

What all of these voices have in common is the notion of “using the right tool for the job.” A phrase I’ve used myself from time to time, both in print and in conversation, so I’m cool with this. Neither SOAP nor REST are universal solutions. They both address similar problem sets, but there are times when one or the other is the more appropriate solution. There are also times when a different technology altogether is the appropriate solution. In the interest, then, of genuine rapprochement lets do some positioning.

First, we’ll note the general problem we’re trying to solve: Making business logic available over the network.

Second, we’ll enumerate the technologies that are commonly used to address this problem (not all inclusive):

1. Binary RPC Style
• CORBA
• DCOM
• RMI / EJB
2. Message-Oriented Middleware
• Topics
• Queues
• Tuple Spaces
3. SOAP / WS-*
4. REST / HTTP

Third, let’s try and establish some axioms:

1. All of these technologies work, i.e. you can build systems today using any of these technologies.
2. Binary RPC-styled technologies are no longer considered viable for underpinning large scale distributed systems, but remain viable for tightly coupled systems—in particular RMI/EJB
• There is no need to rip and replace existing systems that use these technologies.
3. Message-Oriented Middleware (MOM) is a proven technology with wide deployment.
• Tuple Space oriented technologies are not as widely deployed.
• MOMs are not interoperable.
• Deploying APIs and JMS providers is an issue.
4. Most ISVs have lined up behind SOAP/WS-* as the “remoting” technology of the future.
• SOAP/WS-* is complex, and large swaths of WS-* remain unstandardized and unimplemented.
• Vendors will eventually make most of WS-* work, but interoperability among implementations will remain challenging.
5. REST-styled systems based on HTTP have the broadest reach.
• REST is based on proven, scalable, interoperable, standards.
• Awareness of REST outside of common Web applications is lacking, and so too are codified REST best practices.

And finally, some positioning:

• Your default messaging style should be REST. This allows for data to be created and consumed by the largest possible audience. Before considering any other distributed architecture, one should have valid reasons for discounting REST.
• If asynchronous, reliable messaging is absolutely required, use a MOM¹.
• If event driven or broadcast messaging is absolutely required, use a MOM.
• If pub/sub is absolutely required, use a MOM.
• If you have to integrate with existing WS interfaces, use SOAP/WS-*.
• if you desire to stay in step with the product direction of certain large vendors (e.g. Microsoft or IBM) use SOAP/WS-*.

The above, of course, is my opinion, a considered opinion, but biased nonetheless. It is also rather superficial, but it’s a start. I think it may be valuable to flesh this out, and I’m seeking your opinion. If there’s enough feedback I’ll create a wiki to manage this permanently. I’m especially interested in the use cases that Jini and JavaSpaces (Tuple Spaces) are particularly well suited for.

Also, this positioning is likely to change in the future. As WSA, WSRM, WS-Eventing, and WS-Notification mature, implementations will be able to address some of what is solved today by MOMs. Not all of it, but some. And while I doubt that such implementations will be easily interoperable, if you stick with a single vendor you will gain the benefit of having this functionality available with a standard JEE/.NET deployment.

Another point of interest, especially to the SOA people, is intermediation. This is not a concern for REST, as it too is designed to have intermediaries between endpoints, but it’s not so easy for MOMs. In the SOA world it is possible to put something like a web service management system or an XML appliance (and sometimes an ESB) in the message path. It is then possible to use these systems to do what is often done on an app server today, like message transformation and security processing. This provides the advantage of centralizing certain corporate policies and alleviating the service developer and app server from having to do this work (possibly erroneously). I admit that this is attractive, though I’m dubious about how broadly it can be put into practice. In any event, it is a compelling argument for moving away from MOMs as soon as possible, so long as WS-* messaging really delivers on its promise. Like I said, though, REST can be intermediated as easily as SOAP, though it may need some more standardization, for instance around message-level security.

So, am I hanging up my Claymore? No, no I’m not. I think the SOAP/REST debate is different from other techno-religious wars in that a fair number of people (myself included) believe that SOAP/SOA is being oversold. You see, systems vendors large and small are pushing web services as a cure for all your IT woes and advocating a stepwise re-architecture of IT around SOAP-based service orientation. Whereas a comparatively small but vocal minority have been advocating a REST-styled approach to some of these same problem domains. So long as SOAP is positioned outside of the boundaries listed above, it will be important to educate users and vendors alike on how REST is the better tool for the job.

1. Plain old HTTP can do asynchronous communication and there are simple extensions to HTTP to allow reliable messaging. However, there is no agreement that spells out the one true way (or more) to do asynchronous messaging and the HTTP reliability extensions have not been standardized.

Dan Pritchett takes me to task for something I said in my infoQ interview. In particular, this:

The second reason is that web services are too tightly coupled. Each participant needs to know a great deal about each of the others: the service and operation names, the messages they exchange, the datatypes of the message elements, the security context, the messaging semantics, etc. Thusly are non-scalable (and again I don’t mean transactional scalability) systems designed. One could argue that WSDL and the WS-Policy family of specifications addresses all this, but that’s not what I mean. Whether the tool knows or the programmer knows, the point is that one side of the conversation possesses intimate knowledge of the other, and should either side change, everything breaks.

Dan says:

What Pete is calling coupling, I call interfaces. Short of some magic that nobody has yet to explain, software components that interact need to have a contract and both sides have to know the contract. The client may not know about every detail of the contact, but it does know about the parts it needs. This is true whether the contract is exposed in SOAP or REST. And yes Peter, REST also exposes contracts that the other side needs to know. Move your resource to a new URI and see what happens to the clients.

Now I don’t believe in magic either, of course. But what Dan misses in his analysis is the word “too.” Read what I wrote again; “web services are too tightly coupled.” Yes, a REST client needs to know certain things about a service. And, yes, you can break a client by changing some of these things. But the difference between REST and SOAP is the degree of coupling.

To take Dan’s example, what happens if I do move a resource to a new URI? Do the clients break? Not if they leverage a proper implementation of HTTP. If I move my service, I set up the old URI to simply return a 301-Moved Permanently response code and put the new URI of the service in the Location field. Furthermore, in a RESTful application, which I’m defining here as an application that exposes multiple related resources, it’s not the client’s responsibility to orchestrate among multiple resources. Instead the server should guide the client through the application by supplying it the available hypermedia links at any particular stage. A client, then, need only know a single URI, that of the application’s introspection document, in order to access all of the resources available.

As is well known, REST applications also constrain the operations that are available, whereas each and every web service ever written publishes a unique set of operations that the client must know and know how to orchestrate. In a perfect world (and I know it’s far from perfect), RESTful applications don’t even need to agree ahead of time on the resource representation, instead they negotiate which well-known media type will be exchanged via HTTP’s accept and Content-Type headers. Properly RESTful applications push all state out to the client, therefore reducing what the server needs to know about past interactions. REST has a cache constraint, as well, allowing the client or server to cache a response without the client application even being aware that a response was retrieved from cache. And, so long as transport level security is sufficient, a RESTful client does not need to know the security semantics of the service ahead of time either. Nor does it need to worry about breakage if GET, PUT, and DELETE are called over and over again; it knows these operations are idempotent. REST applications exchanging XML documents should not have to worry about the data types of the message’s elements and attributes. They shouldn’t care if elements they weren’t expecting suddenly appear in these messages either.

Aside: I am fully aware that much of the overly-tight coupling that is evident in web service systems is attributable to the tooling and not to the specifications. However, the practice of web services is such that one cannot reasonably divorce web services from the tools.

It’s all a matter of degree. The more things that can be universally agreed upon (i.e., standardized) up front and the more constraints that are placed on the system, the looser the coupling. In the web services world not even the transport is constrained. Not that REST is perfect, mind you: there are quite a few best practices that can be made more formal, but new standards that are themselves RESTful (e.g. APP) can feel free to introduce additional constraints, and the community should be working on such standards. As Andrew Townley put it recently:

If the hypermedia constructs being used to describe the interaction between the clients and servers are not rich enough to abstract these things so the client needs to know that it’s supposed to POST data to URI x rather than being able to simply traverse hypermedia provided by the server (meaning the operation, data and location are provided to the client in a way it can understand rather than having any of this hard-coded as client implementation logic), then the hypermedia application being used (and not the information service) has not been sufficiently defined.

Dan goes on to say that the contract between client and service extends beyond the call interface and into the realm of service level agreements (response time, availability, etc.). He further notes that neither SOAP nor REST address this formally. No argument here, both of these statements are very true. Whether or not SLAs need to be specified via some standard, I haven’t put much thought into. But certainly, if a client is dependent on a service, then the client should have a formal agreement from the service owner to honor agreed upon SLAs. Otherwise, the service owner might just pull a Google on you.

“Slowly, all over the world, the lights on the SOAP endpoints are going out.”

So says Steve Loughran (with a nod towards Sir Edward Grey), in reaction to Google shutting down its SOAP search API.

Update: Dave Megginson supplies an interesting read on this, and it’s not a SOAP v REST thing.

Update 2: James Snell calls bullshit.

Update 3: I hesitant to keep this post alive, but you aboslutely have to read what Leonard Richardson has to say.

Not ten minutes after noticing the tag meme going around, I was tagged by my colleague Richard Monson-Haefel.

Five facts you don’t know about me (unless you know me)

1. I have two sons, ages 3 and 11 months, adopted from South Korea.
2. I have 90% of an LDAP-oriented Active Record clone completed, but haven’t touched the code in seven months.
3. My wife is smarter than I am.
4. I’m from Brooklyn, N.Y originally, now live on the North Shore of Massachusetts, but want to move back to Chicago.
5. I’m an atheist.

And now I have to tag five more bloggers.  I’ll tag my wife, of course, so that this meme goes off into the mommysphere, and then, just for grins, the first four people in my blog roll: Dan Pritchett, Anil John, Bill Higgins, and Joe Gregorio.

Last week Stefan Tilkov and I conducted an email interview that’s now published over on infoQ. I’m not crazy about the title, but I guess it’s accurate. Here’s a taste:

InfoQ: What’s your opinion on UDDI, the third of the core web service specs?

PL: I find it grossly over-engineered, what with tModels and bindingTemplates and categoryBags. Succinctly, anything that requires a design pattern to achieve the predominate use case for which it’s deployed has failed.

Comments welcome here or there.

A little clarity around my previous post of transport-level security and message-level security.

I’ll first note that neither security approach is bound to the REST or SOAP style of component interaction. There is nothing in REST that mandates transport-level security or precludes message-level security. The same is true of SOAP. That said, secure systems implemented via REST or SOAP today tend to leverage the transport-level security mechanisms inherent to HTTP/HTTPS, and a few SOAP systems leverage some subset of WS-Security. But lets take the terms “REST” and “SOAP” off the table, they’re immaterial here.

The point of the previous post was that transport-level security (TLS) and message-level security (MLS) (in their purest forms) are equally credible, and the existence of MLS does not suddenly make TLS worthless. Nor is MLS any more or less better suited to addressing the problems of phishing, identity theft, or trust than TLS is. I do contend that TLS, in the form of SSL/TLS, while not perfect, is mature and of continued value. I further contend that MLS, in the form of WSS, is also imperfect and comparatively immature — not bad, mind you, just young. MLS security provides value that TLS doesn’t, and I’m glad we have it, but the inverse is also true.

But the biggest point I need to make is that the concerns addressed by TLS and MLS are only a small subset of the total number of security concerns. Furthermore, the total universe of security threats, from organized crime and information warfare to shoulder surfing and lost laptops, is far larger than the total universe of available countermeasures, which in turn is larger than the authentication, authorization, encryption, and integrity solutions enabled by TLS and MLS. You cannot secure against every conceivable threat. Organizations have finite resources, and thus must apply threat mitigation under the rubric of risk analysis. When designing a new system, security considerations should be held paramount. A proper risk analysis in the context of the application’s function, design, and user-base must be performed, and it may indicate, for instance, that TLS or MLS or some aspects of both should be used (again, both techniques have value). But the application design must also account for a seemingly infinite number of other attack vectors (XSS, SQL injection, buffer overflows, etc.), and a proper risk analysis is the only means to determine how many resources to allocate to each conceivable threat. And, of course, security concerns extend beyond the application boundaries to securing the physical data, securing private keys, password management practices, network security, physical security, and on and on and on.

As an aside, I find the The eBankingSecurity PDF referenced in Tim’s Web Application Security post of only marginal value. It outlines the many ways that malicious code can steal user credentials or make an untrustworthy site look trustworthy. The fundamental premise is that if a client machine is compromised, then the game’s up. This is very true, but what to do about it? The paper doesn’t say. And therein lies the flaw, there’s no risk assessment or recommendations for how to proceed. It’s good that banks have their consciousness raised and are made aware that some of the security features they’ve come up with are not 100% effective. But what should they do? Fold up shop and go home?

And “the never-mentioned weakness of SSL” is anything but. It requires that the black hats rewrite your host file and install a trusted root certificate of their own making. This is not a weakness of SSL, it’s a weakness of any system that is based on trust, which, ultimately, they all are, including WSS and PGP. I fail to see how any TLS or MLS system can protect against this. I mean once your machine is 0wn3d, its 0wn3d. It’s up to the vendors of operating systems and browsers and security software to make it as difficult as possible for this to happen. But ultimately it’s up to the user to be educated enough to keep his or her machine secure.

Over in the comments for Tim Bray’s “S for Simple” post and again over on his personal blog, Gunnar Peterson claims that the standard REST security approach is inadequate and that something like WS-Security is required for any truly secure architecture. Gunnar has a shoot-from-the-hip tone, but we’ve communicated briefly via email and he states that he would like to buy into the REST vision if only security was adequately addressed. So lets compare and contrast approaches.

The standard RESTful security approach is, of course, HTTP Basic or HTTP Digest or SSL certificate-based mutual authentication for propagating identity credentials (the actual act of authentication and authorization being the purview of the server) and SSL for data integrity (digital signatures) and data protection (encryption). Gunnar notes that these technologies were state of the art in 1995, with the implication they are no longer sufficient for meeting the security needs of the 21st century. This is an example of what I call the “Gilligan, you can’t fly” argument; that is stating that something can’t be done even as somebody else is doing it. The fact is that billions of dollars of worth of business is conducted every year over because of SSL.

SSL has shown itself to be amazingly versatile. Not only do I do my online banking and shopping over SSL, I VPN into the office over SSL and check and send email over SSL. That the first version of SSL is now some 11 years old is a boon. In fact, that’s exactly what I want in a standard; stability for the long term. More so in a security related standard, as that provides the time to shake out any bugs and ambiguities. Of course, SSL hasn’t been static; there’s been SSL 2, and SSL 3, and now TLS 1.1. In fact, given the relative youth of WS-Security, what do you think the odds are that it is immune to attack and that all of the implementations are free from bugs? Close to zero I would imagine. That’s not a slam on WSS–all standards are young once–but praise for the tried and true.

Gunnar notes that the world has moved past SSL etc., and cites as examples identity theft, phishing/pharming, and malware. But these security threats are completely orhtogonal to the security concerns SSL addresses. Ditto, I might add, WS-Security. Both of these standards address identity propagation, message encryption, and message integrity only, and neither will protect you from the threats just mentioned. Security is a BIG subject and the areas covered by SSL and WS-Security are just one small part of it. We also need good practices around securing persisted data (and what data to persist); education to prevent social engineering attacks; properly designed operating systems that won’t run anything with a .exe extension or run needless services; developers who are cognizant of buffer overflows, SQL injection, and cross-site scripting attacks; properly managed perimeter defenses; and so on and so on.

Gunnar then notes that HTTP Basic is not secure, and that, of course, is absolutely true. The password passed via HTTP Basic is base64 encoded, which is practically the same thing as being sent in clear text. In fact, it is the same thing, as encoding is not encryption. However, nobody is claiming otherwise. For those use cases where identity is all that’s required, then HTTP Basic is all that’s needed. For the larger number of use cases where credentials need to be encrypted, then use HTTP Basic in conjunction with SSL or use HTTP Digest or use SSL mutual auth. I’ll note that WSS is similar in design. The standard identity token is the Username token, the password element of which is sent in the clear by default. You can hash the password, but it should be noted that the recommendations specified in the Username Token Profile require that the server store the user’s password in clear text. Avoiding this requires an out-of-band negotiation that cannot be specified using WS-SecurityPolicy.

Gunnar next takes SSL to task for providing a great big hole through the firewall. I have to confess that I don’t understand this point. Sure, SSL encrypts the content of the communication channel, but that’s what it’s supposed to do, and, as already noted, it seems to work pretty well. Furthermore, when using HTTPS, communication travels through a completely different port than HTTP. A port that can be monitored more carefully. With WSS, not only do we have SOAP tunneling over HTTP, we have encrypted messages in the SOAP envelope. I don’t see how that could be better. In fact, the SOAP tunneling alone is a fairly significant security hole. Requiring that companies invest in new infrastructure to secure against threats that bypasses all the stuff they’ve already deployed. Furthermore, it’s not like the SSL connection is being terminated on the application server, more likely than not there is some intermediary up front that terminates the session, e.g. an SSL accelerator or proxy server. And before you can say, “ah hah,” know that the same is true for WSS. A properly architected SOA security solution addresses security concerns in “the cloud” using either an XML gateway or a web services management solution. (An article I wrote on this subject goes into more detail.) In other words, done right, the message received by a WSS-secured endpoint is also in the clear.

Gunnar goes on to note that the point-to-point security offered by SSL is an interoperability and simplicity nightmare. I don’t get this point either. Not only is SSL well understood, broadly implemented, and properly constrained, it is quite evidently not an impediment to interoperability as I can securely connect to anyone on the planet right now, so long as they let me. In contrast, not only is SOAP well known for its continuing interoperability headaches, they show up in spades with WSS. Just for starters, how do I know that a SOAP endpoint is secured in any way? You can’t tell from the URL and there’s no handshake with which client and server can negotiate security. If a service is secure, in what way? Do I need to provide credentials? What kind? A Username token? (hashed?) A SAML token? (how much of SAML do you support?) Or an encoded certificate? Is encryption used? The whole message or just some elements? Do I need to sign it? You can say that all of this is addressed by WS-SecurityPolicy, but WS-SecurityPolicy isn’t baked yet nor is anyone using it. And, as we’ve just shown, it’s not comprehensive. Furthermore, we have the implementation details. Is, say, BEA’s WSS implementation really 100% compatible with Microsoft’s, and is that 100% compatible with IBM’s, and is that 100% compatible with AmberPoint’s, and so on down the line. I don’t know the answer to this (since there doesn’t seem to be any instances of WSS encryption/DSig use in the wild [read the comments too]), but I strongly suspect not. To demonstrate, please retrieve this encrypted representation of a common greeting. Now, I don’t know what browser you’re using (if any) or what OS or anything. And you don’t know what encryption algorithm I’m using or what version of SSL etc. However, even though we have no a priori information about each other, I am absolutely positive that you were able to retrieve that resource. What can be simpler than that?

With all of that behind us, I can get on to what seems to be Gunnar’s main point and the only significant difference (outside of the whole simplicity and interoperability thing) between SSL and WS-Security. And that is that SSL provides transport level, point-to-point security while WSS provides message level, end-to-end security. That’s true, but that doesn’t provide WSS with magical security powers, it just solves a different problem. Nor does it relegate SSL to the scrap heap of history. SSL is not a security panacea–nothing is, but it does what it is does very well. Regardless, there is nothing in REST that prohibits the use of message-level encryption, though the mechanism–should it be needed–would need to be spec’d out.

I’m not dismissing WSS, it’s a perfectly adequate specification for what it does (though it requires the WS-I Security Profile to introduce enough constraints to provide a reasonable chance of interoperability). But the value of message level security should still be questioned. For one thing, what’s the business case? If message-level encryption is so important, why isn’t anyone using it? When Burton Group queried its clients as to their use of WSS, it was found that the only use was to pass identity tokens over HTTPS. When I was working at Systinet (now HP) I vividly recall the WASP (not Systinet Server for Java) product manager spitting nails because his team had just spent six months implementing WSS at our customer’s request and no one–not even those who requested the feature–was using it. Also, this is not the first time message level security has been proposed. When I was working at Netscape back in 1997 I spent a fair amount of my time advocating for S/MIME. Now, nearly ten years later, how many people are using S/MIME to secure their email? And how many are using SSL? Exactly.

In summary, RESTful security, that is SSL and HTTP Basic/Digest, provides a stable and mature solution that addresses transport level credential passing, encryption, and integrity. It is ubiquitous, simple, and interoperable. It requires no out-of-band contract negotiation or a priori knowledge of how the resource (okay, service) is secured. It leverages your existing security infrastructure and expertise. And it addresses 99% of the use cases you are likely to encounter. SSL does not support message level security, and if that’s a requirement, then leveraging SOAP and WSS makes sense. Recall, though, that this is not a limitation of REST per se, though it is left as an exercise to the reader. SSL is not perfect and it does not address all security requirements, but nothing does or can.

Hope that helps.

Number of page views on Nov. 14th: 999

Number of page views on Nov. 17th: 13,989