Pete Lacey’s Weblog

This is my third attempt at posting over the past few days. The previous two aborted posts were of the SOAP vs. REST, simplicity vs. complexity variety. But I click around and I read a few things and I click around and read some more. And, you know what? It’s all been said before, again and again and again, and better than I ever could. So I’m just going to reference Bill de HÓra’s victory post one more time, and echo it here. The forces of simplicity have won. SOA is dead, SOAP is dead, the WS-Hairball is dead, and it seems even XML Schema is dead. Now begins the age of REST, and Rails, and Microformats, and getting shit done.

Or is it? Maybe you don’t work for or with a Global 2000 company, so I’ll let you in on a little secret: They Can’t Hear You! That’s right, the CIOs, and Enterprise Architechts, and, yes, even the journeyman programmer employed by these firms have no idea that there’s even a discussion going on. I’ve asked them. I really have. It’s what I do for a living. And the typical corporate technologist (broad strokes here, of course I don’t mean you) hasn’t considered REST and decided against it, they haven’t even heard the term. Ditto RelaxNG, Django, Atom, and everything else that makes the Web work and makes working with the Web easy. They don’t read technology blogs, they don’t know who DHH is, and they’re excited to get their new Vista gear.

Why? Two reasons. One is obvious: by and large, enterprise technology professionals know only what IBM, Microsoft, Oracle, and SAP tell them, tempered a bit with what Gartner and Forrester and, yes, Burton Group have to say. And what these companies are saying is SOA, SOA, SOA. Now, I don’t believe for a minute, as some people have suggested, that they are selling complexity because you can’t make money on simplicity (you can, by the way). No, they honestly belive SOAP, SOA, and the WS-Framework is the best way to build and architect distributed systems. And they’re smart people too; my SOA-preaching colleagues at Burton Group are scary smart.

The second is because business-oriented technologist refuse to beleive that simple solutions apply to their problem set. It’s always been complex before, and gosh darnit, it’s gonna stay that way. They want transactions, and reliability, and asynchronous messaging, and orchestration, and everything else. If it doesn’t look like Rendezvous or Tuxedo or BizTalk, then it can’t be a business grade solution, therefore it must be some toy. Closely related to this is the fact that the EJB/CORBA style of development, while not easy, is natural to the corporate developer; that is SOAP looks like what distributed systems are supposed to look like. Certainly no technology that’s been sitting under their noses for the last ten years can ever handle the needs that enterprises have!

So we’ve won and we haven’t. Outside the corporate firewall the message is out. Sure there’s work to be done yet, but people are doing it. Inside the firewall, life goes on as always. It’s this situation that prompted Steve Jones to say Want to be cool? Learn REST. Want a career? Learn WS.. Never has something been so right and so wrong.

Steve is absolutely right that, for the most part, businesses only know what the big vendors tell them. He’s also right in saying that every company will use or encounter web services. It’s sad that this is the “commercial reality.” However, this has always been the case, and while the best technology doesn’t always win, bad technology doesn’t win either. Hailstorm, anyone? He notes that if you want to be employed in three years time, you had better bone up on SOAP web services, with the implication that by that time this technology can be made to work.

And this is where Steve gets it all wrong. Web services won’t be pervasive in three years, for the simple reason that the technology can’t do what its proponents claim it can do. It’s too complex, it’s too poorly specified, it’s caught up in the ever-continuing vendor wars, and it’s simply not the right design for a pervasive and scalable distributed architecture. Yes, web services will be here three years from now, and, yes, there will be a handful of localized successes. But it will not be the all encompassing, enterprise spanning, solution to all your IT needs. It will, in fact, be yet another legacy system to integrate with. In the meantime, if you’re on one “the SOA team” at work, at least you have a few years of guaranteed employment ahead of you.

Wasn’t planning on posting again today, but this is beyond funny (to geeks).

Update:  And this too.  Move over Google, it’s time to ask Ms. Dewey.

Wow! There is so much going on right now. Here’s a rundown of some of the things you might have missed:

• Leveraging existing content types: Read the comments to Duncan’s “Setting Data” post, where he takes the time to educate me about the finer points of why not to invent new data formats if at all possible.
• eBay engineer responds: Did you see the comment from Dan Pritchett? Dan’s an actual eBay engineer who is providing his own voice in Duncan’s dialog in place of Duncan’s imaginary interlocutor. Part 1, Part 2. Must reads.
• Ad hominem attacks: There’s a long thread over on the Yahoo REST discussion list where some reasonable comments are unreasonably attacked.  There’s some good technical take-aways in there, but my take away is that certain members of the REST discussion list should read Steve Yegge’s Bambi Meets Godzilla post.
• Sex Sells:  Over at Wife’s blog, she takes the so-called “geekosphere” to task for its comparatively tepid response to my earlier “S stands for Simple” post.  Compared, that is, to how often someone named Julia and her husband have sex.  So, gentle readers, go confuse the heck out of the mommysphere, and post a comment on Wife’s blog.  Here’s a template:  When I was {using some awful, non-trendy technology} my sexual activity {was in some way diminished}, but since switching to {useful, trendy technology} my sex life is {frequent, robust}.  For example: When I was using statically type languages my sexual activity was optimized and predictable, but since switching to Ruby, my sex life is dynamic and, strangely, duck-like.

In the near future I will address some comments appended to my last two posts, I promise.  Others have asked me to take on UDDI and SOA in general, but I make no promises here.  Right now I’m 1,600 miles from home and getting ready for the Thanksgiving holiday.  To the American’s out there, have a happy Thanksgiving.

Well, that was interesting! Given that I’ve only reinvigorated this blog a few days ago, I didn’t expect my little SOAP-dialog post to get much play. However, not a few members of the digirati noticed, and the next thing you know I’ve been memed. There’s all sorts of interesting things to say about the traffic and where it’s coming from, but what stands out is the conversation. Aggregating the hundreds of comments, referencing posts, blurbs found on reddit and del.icio.us, etc., there was not a single dissenting voice! Bill de hÓra’s gone so far as to declare victory in The War on Error.

But I wonder if there might be a bit of an echo chamber effect going on. After all, some WS-Supporter must be actively blogging. So, here’s my question to you: where are the WS-Proponents? I’ve got Don Box and Anne Thomas Manes covered. Any other pointers?

Oh, and thanks for all the wonderful feedback.

There has been a long running debate in the Application Platform Services Group here at Burton Group between the REST people on one side and the SOAP people on the other. For the most part it mirrors the external debate. In one recent exchange, while discussing the complexity of SOAP and the web services framework, the SOAP side said, “Before all of the WS-* stuff, SOAP was actually simple. That’s what the ‘S’ stood for.”

And now a history lesson. It’s the year 2000, a harried developer has a problem

Developer: So, my boss was playing golf this weekend, and now I have to—quote, unquote—SOAP-enable the enterprise, but I don’t know what SOAP is. Can you help, SOAP Guy?

SOAP Guy: Sure thing. First, SOAP stands for Simple Object Access Protocol.

Dev: So it’s simple?

SG: Simple as Sunday, my friend.

Dev: Okay, lay it on me.

SG: Well, just like it says in the name, SOAP is used for accessing remote objects.

Dev: Like CORBA?

SG: Exactly like CORBA, only simpler. Instead of some complex transport protocol that no one will let traverse a firewall, we use HTTP. And instead of some binary message format we use XML.

Dev: I’m intrigued. Show me how it works.

SG: Sure thing. First there’s the SOAP envelope. It’s pretty simple. It’s just an XML document consisting of a header and a body. And in the body you make your RPC call.

Dev: So this is all about RPCs?

SG: Absolutely. As I was saying, you make your RPC call by putting the method name and its arguments in the body. The method name is the outermost element and each sub-element is a parameter. And all the parameters can be typed as specified right here in Section 5 of the specification.

Dev: (reads Section 5) Okay, that’s not too bad.

SG: Now, when your service is deployed, you specify the endpoint.

Dev: Endpoint?

SG: Endpoint, the address of the service. You POST your SOAP envelope to the endpoint’s URL.

Dev: What happens if I GET the endpoint’s URL?

SG: Don’t know. Using GET is undefined.

Dev: Hrrm. And what happens if I move the service to a different endpoint? Do I get a 301 back?

SG: No. SOAP doesn’t really use HTTP response codes.

Dev: So, when you said SOAP uses HTTP, what you meant to say is SOAP tunnels over HTTP.

SG: Well, tunnel is such an ugly word. We prefer to say SOAP is transport agnostic.

Dev: But HTTP isn’t a transport, it’s an application protocol. Anyway, what other “transports” does SOAP support?

SG: Well, officially none. But you can potentially support any of ‘em. And there’s lots of platforms that support JMS, and FTP, and SMTP.

Dev: Does anyone actually use these other transports?

SG: Uhm, no. But the point is you can.

Dev: Fine. How ‘bout this SOAPAction HTTP header, what’s that for?

SG: To be honest, no one’s really sure.

Dev: And these “actor” and “mustUnderstand” attributes, does anyone use those?

SG: No. Not really. Just ignore those.

Dev: All right, let me give it a shot.

(time passes)

Dev: Well, I could mostly make things work, but only if I stick with one SOAP stack. Also, I can’t say I like the idea of remote procedure calls and serializing objects.

SG: Remote procedure calls! Serialized objects! Where did you get the impression that SOAP was about RPCs? SOAP is all about document-based message passing, my friend.

Dev: But you just said…

SG: Forget what I said. From here on in we pass around coarse-grained messages—you like that term, coarse-grained?. Messages that conform to an XML Schema. We call the new style Document/Literal and the old style RPC/Encoded.

Dev: XML Schema?

SG: Oh, it’s all the rage. Next big thing. Take a look.

Dev: (Reads XML Schema spec). Saints preserve us! Alexander the Great couldn’t unravel that.

SG: Don’t worry about it. Your tools will create the schema for you. Really, its all about the tooling.

Dev: How are the tools gonna do that?

SG: Well, they will reflect on your code (if possible) and autogenerate a compliant schema.

Dev: Reflect on my code? I thought it was all about documents, not serialized objects.

SG: Didn’t you hear me? It’s all about the tools. Anyway, we can’t expect you to write XML Schema and WSDL by hand. Besides, its just plumbing. You don’t need to see it.

Dev: Whoa, back up. What was that word? Wizzdle?

SG: Oh, haven’t I mentioned WSDL? W-S-D-L. Web Services Description Language. It’s how you specify the data types, parameter lists, operation names, transport bindings, and the endpoint URI, so that client developers can access your service. Check it out.

Dev: (Reads WSDL spec). I trust that the guys who wrote this have been shot. It’s not even internally consistent. And what’s with all this HTTP GET bindings. I thought GET was undefined.

SG: Don’t worry about that. Nobody uses that. Anyway, your tools will generate a WSDL, and in the WSDL will be the schema.

Dev: But shouldn’t it be the other way ‘round? Shouldn’t I design the contract first and then generate the code?

SG: Well, yeah, I guess that sounds right in principle. But that’s not so easy to do, and very few SOAP stacks support WSDL-first development. Just let the tools worry about it.

Dev: One more question. If we’re now passing around XML Schema compliant messages, where do you specify the operation name?

SG: Well, remember that SOAPAction HTTP header? Most people are putting it there.

Dev: Most people?

SG: Well, this new style isn’t actually written down anywhere.

Dev: I’ll also note that your entire industry is built around ambiguous, sometimes erroneous, and definitely not standardized specifications. In fact, the SOAP and WSDL specs are just W3C Notes, not ever working drafts.

SG: We’re working on that.

Dev: Will this give me the interoperability I’ve been promised?

SG: Absolutely.

Dev: I’ll try it out.

(Time passes)

Dev: This is getting ugly. The WSDL my tools generated can’t be consumed by the tools my partners use. Not only that, the schemas it generates are impenetrable and can’t be reused. And no tool seems to have agreed on how best to handle the SOAPAction header.

SG: Sorry to hear that, buddy. On the bright side, nobody uses the Doc/Lit style anymore. In order to get transport independence back we’re all using wrapped-doc/lit now. Doesn’t that sound cool: wrapped-doc/lit?

Dev: What’s that?

SG: Well, it’s just like Doc/Lit, but you take the whole message and wrap it in an element that has the same name as the operation. Now the operation name is back in the message where it belongs.

Dev: Okay, where’s the spec on this?

SG: Oh, there is no spec. This is just what Microsoft seems to be doing. Looked like a good idea, so now all the cool kids are doing it. However, there is this new thing. I think you’re gonna like it. It’s called the Web Services Interoperability Group, or the WS-I. What they’re doing is trying to remove a lot of the ambiguity in the SOAP and WSDL specs. I know how you like specs.

Dev: So, in other words, the specs were so bad you need a standards body to standardize the standards. Lord. Well, will this solve my interoperability problems?

SG: Oh, yeah. So long as you use a WS-I compliant SOAP stack, avoid using 8/10ths of XML Schema, don’t use any unusual data types, and don’t count on working with WebSphere and Apache Axis.

Dev: And is wrapped-doc/lit explained in there?

SG: Ermm, no. But that’s okay, you’re tools understand it. Most of them, anyway.

Dev: Let me sum up. The definition of SOAP is in constant flux, SOAP is anything but simple, and it is no longer meant for accessing objects-even though that’s what all the tools still do.

SG: That’s about right, but we’re way ahead of you on this. We’ve deprecated the meaning of the SOAP acronym.

Dev: Really! What does it stand for now?

SG: Nothing.

Dev: (blink)

SG: Let me tell you about UDDI.

I see that Duncan Cragg has beat me to the punch by also using the dialog format for his most recent REST/SOAP related post. I take solace in the fact that this conceit has been used since the days of Socrates.

Leonard is looking for a formal definition of SOA. My colleague, Anne Thomas Manes, is a thought leader on this subject, and Burton Group clients and the industry as a whole tend to line up behind her definitions. I’ll share them here:

Service oriented architecture (SOA) is a software design discipline in which application and infrastructure functionality are implemented as shared, reusable services.

SOA is more about design and behavior than technology.

SOA is something an organization does—not something it buys or something it builds.

Admittedly, this is all kinda loosey-goosey and can describe most anything. In fact, a core tenet of Burton Group’s stance is that SOA is technology agnostic: you can create a SOA with CORBA or RMI or DCOM or SOAP or, some say, REST. However, the first definition of SOA given above and the first word in “service-oriented architecture” demand a definition of what a service actually is. People might argue this, but the only reasonable definition of a service is a network available collection of related operations. And an “operation” is an action (function/method) offered by the service that takes in a message or parameter list and generally returns some response. To be clear, however, while you can conceivably create a SOA using, say, RMI, in reality SOAP web services is both the recommended and assumed underlying technology.

Plugging this definition of a service into the SOA definition gives us:

Service oriented architecture (SOA) is a software design discipline in which application and infrastructure functionality are implemented as shared, reusable network available, collections of operations.

And this is where I part company with those that think that REST can be used as the underlying technology of a SOA. Sure, in the sense that SOA implies changes to how you think about designing applications or modifications to your SDLC, then fine. But a resource is not a service. A resource does not have arbitrary operations (that, by the way, have to be specified somewhere in the incoming message—and not in the HTTP headers, because SOAP is supposed to be transport agnostic). Stefan Tilkov put this into brilliant perspective when he demonstrated the differences between how people think about resources and how they think about services.

From this, we can also put to rest the notion that the choice between SOAP and REST is an implementation detail. How REST systems are architected at the resource level is fundamentally different from SOAP. And how REST leverages HTTP (URIs, response codes, caching) is fundamentally different from how SOAP leverages HTTP (as a tunnel).

Back to Leonard then. You can drop the name “Message-Oriented Architecture.” It is, as you noted, too similar to message-oriented middleware and it doesn’t properly reflect that other architecture. The name “SOA” works fine, so long as the “S” is defined. Further, I was being generous when I said that a service accepts either a message or a parameter list. While sending messages is considered best practice, today’s tools and today’s programmers still lean towards sending serialized objects (though serialized as document/literal messages).

ROA, while it doesn’t quite roll off the tongue, is also fine.

Leonard asks “Is it useful to distinguish between the typical robotic SOAP/WSDL interfaces autogenerated from Java or .NET code at the touch of a button, and the in potentia SOAP architecture that can be anything, even RESTful?”

Yes. Yes. Yes.

Stefan and Sam have both noted that you can craft a SOAP message with vi, and that SOAP can be considered orthogonal to WSDL, and that one can leverage the SOAP envelope while ignoring all the other baggage. This is technically all true. But you cannot ignore how SOAP and SOA are being sold and practiced. I’ve been briefed by SOA vendors of all stripes. I’ve sat in vendor sales calls with my customers. Heck, I have myself advocated a web services-based SOA. And this version of SOA assumes SOAP and WSDL and UDDI, and XML Schema, and WS-Whatever, and web service management tools, and enterprise service busses, and web service platforms, and so on ad nauseum. I don’t want to harp on this, but it’s wishful thinking at best and disingenuous at worse to assume that SOAP and SOA can be divorced from this reality.

And to answer the final question about how to categorize those not quite RESTful, HTTP+POX services so commonly found out there. They should be classified as aberrations, as devoid of architecture, and as an example of how not to do things (while acknowledging they work). I direct you once again to Duncan Cragg’s coinage of the term STREST.

It’s time to reawaken this blog, and the current POX/REST/SOAP thing is as good a reason as any. Everyone has something good to say, and I have comments for everybody.

Leonard Richardson kicked it off while ruminating about the forthcoming book on REST that he’s writing in collaboration with Sam Ruby. In it he notes that the HTTP+POX services made evident by Web APIs such as those from Flickr and del.icio.us are not completely without architecture, but lean towards the RPC style of SOAP. I contend that this is not true. That these services have an accidental RPC-like architecture that comes from trying to avoid SOAP while still being muddled in traditional RMI/CORBA style networked application development. While this may be the simplest thing that possibly works, they do themselves no favors by avoiding both REST and SOAP. This is what Duncan Cragg calls Service-Trampled REST (STREST). Leonard makes the great point that most browser-oriented web apps are not RESTful either.

Leonard goes on to note that the PUT and DELETE HTTP verbs are absolutely required by REST, because without them one is tempted to overload POST and put actions in the URL. In practice, this is somewhat true. However, you can’t make the argument that REST is all about the HTTP verbs without conflating the two. REST is informed by HTTP and vice-versa, but they are not synonyms. Tim Bray is correct, REST requires a means of safely retrieving a resource’s representation (Get) and a means of changing the state of that resource (Post), PUT and Delete are nice to haves. Just PUT really, as no one really deletes anything. I would argue that if all of WEBDAV’s methods (COPY, MOVE, LOCK, etc.) were part of the original HTTP specification, Leonard’s line of reasoning would mandate them too. That said, it is extremely convenient to be able to distinguish POST from PUT.

The ensuing discussion addressed most of the rest of Leonard’s post, and I’ll get to them in a second. I’ll answer one more question that Leonard raised, however. For all intents and purposes, the term “web services” has been lost to anything but SOAP and the rest of the WS stack. SOA is an architecture that (tends to) leverage web services. Contrary to what some claim, a SOA cannot be using REST, simply because not all resources are services.

Sam picks up the ball by implying that the line between SOAP and POX (not REST, POX) is finer than one would think. I’m fine with that, but that’s likely because both POX and SOAP are a mistake. SOAP for all the reasons that Steve Loughran pointed out (and them some; links anyone) and HTTP+POX because it is just a simpler RPC mechanism.

Things really get ugly when Sanjiva Weerawarana (of WS02 and Axis2) weighs in. Sanjiva gets it all wrong, but not necessarily entirely from a technical point of view. From my vantage point as a Burton Group consultant, I know is being peddled to enterprise customers, and it’s not some idealized notion of “SOAP is whatever you want it to be.” It’s the whole butt-ugly, can-never-be-restful, WSDL 1.1, WS-HairBall stack.

But on to the details. Mark Baker correctly brings Sanjiva to task for the bulk of his argument. Summary: SOAP is not whatever you want it to be; there are expectations that both communicating parties interpret a message identically. To think otherwise is ludicrous. To believe that a developer coding to Microsoft’s WSE expects BEA’s SOAP stack to freely interpret the intent of his message is just wrong. In Stefan Tilkov’s otherwise excellent response, this is where he gets things just a bit mixed up. Yes, the SOAP envelope all by itself can be a good thing, but SOAP is more than an envelope. You can’t ignore how SOAP is packaged and sold, how it is specified, and how it is implemented. SOAP, despite everyone’s best efforts, is still a remote object protocol, and somewhere in the message, whether it be in the SOAP Body, in a WS-Addressing header, the outer element of a wrapped doc/lit message, or in the SOAPAction HTTP Header, is the method to invoke on the remote object. And, remember, to the business developer, SOAP is also WSDL, and a mish-mash of encoding styles and serializations, and impenetrable XML Schema, and tModels, and vendor wars, and ESBs, and shoddy specifications, and on, and on, and on.

Sanjiva then goes on to make the increasingly common claim that SOAP is only useful when you want to solve enterprisey problems like reliable messaging and end-to-end security. This is interesting in a number of respects.

1. The number of people using WS-ReliableMessaging in production today totals zero.
2. The number of people using all of WS-Security (and not just username tokens) approaches zero
3. These numbers hold true for WS-this-that-and-the-other-thing too.

Zeroing in on WS-Security we can note a few other things. To argue that securing the message (admittedly handy, but handled by XML Digital Signatures and XML Encryption, and only complicated by WSS) and not the protocol is of absolute importance, ignores the fact that we’ve been building secure applications without this ability since time immemorial. It further ignores that a sound architecture does not ask the endpoint to do the decryption, but that the decryption is done upstream in a web service intermediary of some sort, possibly one that precedes other intermediaries, thus negating this argument entirely.

In short, if SOAP is only useful when used in conjunction with the advanced WS framework, then, for the time being at least, SOAP isn’t useful at all. With the exception of WSS, no other specification has been fully ratified. And to repeat, this is not what’s being sold to the Global 2000 customer. They are being sold SOAP and WS-* from top to bottom. Not, REST when you can and SOAP when you need to.

And, so it doesn’t look like I’m copping out, I’ll answer the next question, “So how then do we meet these enterprise goals of reliability, security, pub/sub, etc.?” The answer varies depending on the need. I suspect that we don’t need anything new to cover what WSS covers, though having it may open up some new opportunities. For the relatively small number of apps that need or desire reliable messaging, pub/sub, etc., then I would say use the technologies that exist to fit that need: MQ-Series, TIBCO Rendezvous, etc. Not everything has to be an HTTP/REST app. (Note, pub/sub is already RESTful in its own way: write a message, read message - nice. Also note, that three of the four main HTTP verbs are reliable enough; GET, PUT, & DELETE.)

And for the record, I do not believe that REST services are currently defined well enough. I think having a standard envelope would be kinda useful, HTML has one. I think it would useful to put a stake in the ground and say XML is the standard for representation. I think that REST needs a better name.

It’s good to be back.