Pete Lacey’s Weblog

Is REST over-hyped?  Is WS-* viable?  Is SOA evil?  Stefan steps up to the plate with RESTafarian SOA Killers.

This is a long-delayed post.

Next week is QCon San Francisco. Stefan generously offered me a slot in the “Connecting SOA and the Web: How much REST do we need” track, alongside Steve Vinoski, Sanjiva Weerawarana, Dan Diephouse, and other luminaries. Woot.

For my part, I’ll be demonstrating the “ilities” of REST. Yes, rather than just explain things and have the audience take my word for it, I’ll be showing running code of a real (albeit simple) RESTful application. Thus, for the last two weeks, I’ve been busily learning Rails and enjoying its support for REST. It’s a simple app, and if I knew when I started what I know now, it could have been done in a day or two. No loss, I’ll be able to leverage this time for the REST Workshops I deliver as part of my Burton Group gig.

Of course the intent of the demo is not to show off Rails (I don’t plan too) or to prove how leet I am (I’m not), but to put an emphasis on the fact that something as mundane as a Web application, when properly designed, leads to accessible information and evolvable systems. I’d love to demonstrate scalability and performance too, but, hey, I’ll only have one laptop with me. (Oh, and VMWare rocks.)

Hope to see you there.

So the battle still rages over on SOA Discuss. However, the list server seems to have crapped out. So, while I haven’t been an active participant by any means, I’m pulling this from my sent mail folder and posting it here.

Anne Thomas Manes wrote:

Previously I defined and redefined some terms to be used when discussing networked systems. With feedback from Sam Ruby, Josh Haberman (in the comments), and Steve Jones, plus smoe more thought on my part, I would like to present a more refined version of the same. With this post I’d like to introduce some new terms and revisit some old terms, all arranged in a taxonomical structure from most to least general. Note, the definitions given here will differ in places from those given earlier and from some defintions in general use.

Why bother? Principally because the terms in current use are heavily overloaded, especially SOA, which has at least three distinct definitions depending on who you ask. It’s my hope that some agreed upon terminology will help keep people from talking past each other and to better crystalize our thinking on the subject.

This taxonomy only addresses network programming models that are to be used for general-purpose systems. It does not care about anything below layer 7 of the OSI network model and it does not care abouut dedicated network protocols such as LDAP or SQL*Net.

At the top of the stack there is network-centric computing (NCC). NCC is the generally held belief by all interested parties: developers, architects, business analysts, and the business itslef, that exposing functionality directly on the network in a standard and interoperable manner is superior to building application silos. NCC is completely technology independent; it is an approach to systems design, a mind set. Why not just use the existing term distributed computing? Mainly because that term includes extremely tightly bound systems, including distributed systems that are in fact a single system that happens to reside on multiple machines; such as Oracle RAC, or systems that parralellize processing, such as SETI@home.

Next we have NCC designs that take into account the vagaries of the network. These are termed netwok-oriented computing (NOC) designs. NOC systems have incorporated into their design the fact that networks are not reliable, do not have zero latency, etc. In contrast, there are NCC designs that attempt to abstract the network away, such as system exhibits a network-independent computing (NIC) design. Such a design attempts to ease the burden on the developer by modelling remote systems as if they were local.

Either model can be further refined by applying an architectural style. “An architectural style is a coordinated set of architectural constraints that has been given a name for ease of reference.¹” In other words, by restricting the universe of possible design approaches an architectural style becomes a way of talking about architectural designs. An architecural design, then, is the blueprint of a running system. And an architecture is the system itself. Typically, as I do here, the latter two terms are conflated so that an architecure is an architectural design. By loose analogy, Art Deco is an architectural style, the plans for the Chrysler Building are an architecural design, and the Chrysler Building itself is an architecture.

The most commonly applied architecural style used in NOC systems is Representational State Transfer (REST). REST is a style that has the resource as it’s key abstraction, where a resource is anything important enough to name. A resource has state that changes over time, and clients interact with the resource via a uniform interface. Critically, resources also point to other related or interesting resources. REST as an architecural style is one step removed from an architecture. One possible implementation of the REST style is the resource-oriented architecture (ROA). ROA is the set of best practices encouraging the principled use of HTTP to create RESTful systems.

As regards NIC designs (e.g. CORBA, DCOM, and WS-*) there is no identified architecural style. Such architectures have evolved over time to address the needs of their creators and users, and later systems are typically modeled after earlier systems. Even so, it is possible to infer certain fine-grained styles inherent in these architecures, such as distributed objects or client server. One thing such systems have in common is the promotion of the interface as the key abstraction, where the interface describes a collection of non-unifrom operations/methods exposed by a service. Collectively, we can refer to these systems as having a service-oriented architecture (SOA).

In my earlier post I claimed that while governance and business process analysis are selfevidently important, that SOA (and ROA) did not exhibit sufficient uniqueness from traitional application design and development to warrant specialized versions of these. I’m backing off that stance. After all, if devolving a network-accessible application into its constituent components can justify not one but six distinct concepts, then certainly the same is true for governance and business analysis. In addition, NCC is indeed different enough that it requires a level and style of governance that needs to be clearly spelled out and executed. Ditto business analysis. I firmly reject, however, the idea that governance and/or analysis equal architecture and that the technology is not important.

As pertains to governance, ROA and SOA are different enough from each other that the policies and processes that govern them should also be distinct. Hence we’ll give them the simple terms ROA governance and SOA governance. But while the governance processes will be different, the governance needs will often be the same. For instance, either architecture could benefit from a directory. In today’s SOA environments that typically means UDDI. While in a ROA environment it is likely a Web page or simply the result of a search. (Or vice-versa.) Therefore, the common governance abstractions (directories, repositories, documentation, etc.) are given the term network-centric governance.

Regarding business process analysis, I don’t think that ROA and SOA warrant specialized techniques (but I’m willing to be convinced otherwise), thus we have simply networked business process analysis.

Summarizing the above taxonomically gives us:

– Network-centric computing (NCC)
—– Network-oriented computing (NOC)
——– REST
———– ROA
—–Network-independent computing (NIC)
——– SOA
– Network-centric governance
—– SOA governance
—– ROA governance
– Networked business process analysis

(I would have used nested unordered lists, but WordPress keeps horking that up.)

To use these terms in discussion we might have the following:An enterprise that recognizes that application silos are to be avoided and exposes functionality directly on the network for general consumption in a standardized and interoperable manner is undertaking a network-centric computing initiative (an NCC initiative). They may choose to acknowledge that the network imposes certain architectural constraints on this initiative and thus choose a network-oriented computing approach. This would be especially likely whenever crossing network boundaries, but is considered by some to be a good approach to most NCC designs. Alternately, they may choose to abstract the network away by using a network-independent computing approach. This is especially likely when the clients and services are on a single, well managed, local area network.

Once again the participants in the SOA discussion group have got themselves all riled up about what exactly SOA is and why it may or may not be working. Here’s my two cents. We’ll start with another history lesson. (Skip it and get to the meaty bits.)

Once upon a time, the employees of Example Corp. complained to the Finance department that they were tired of using spreadsheets to file expense reports. So Finance went to IT and said, “Build me an expense report management system — and I want it by the end of the year.” And IT went among the people in HR and Finance and the company at large and gathered requirements and analyzed business processes. After which they returned to Finance and said, “The end of the year’s out. How ’bout June?”

“Just get it done,” said Finance.

In December of the following year IT delivered a system that more or less met most of the requirements they had gathered. “Fine. Sure. Whatever,” everyone said.

Then one day, Finance came back to IT and said, “Hey, I need to pull all expense data into our new ERP system.”

“No problem,” said IT, “We’ll code up a batch process that produces a monthly report and then we’ll insert that information right into the ERP system’s database.”

“Erm, Okay.” said Finance, and went away bewildered. But some months later Finance returned to IT and said, “We’ve upgraded the ERP system, and now that expense thingy isn’t working anymore.” As it happens IT had just been visited by Consulting who had said, “My consultants are tired of entering expense data twice. Fix it.”

IT was sullen. How were they ever going to integrate all these systems? And shouldn’t they prepare ahead of time for the pending acquisition of Sample Systems?

“We could publish our database schema,” said one IT member.

“Won’t work. People will enter things wrong,” said the lead developer.

“Stored procedures?”

“Maybe, but that means deploying database client libraries everywhere.”

“We could create an API that we distribute to everyone who asks,” suggested another IT member.

“Too many languages, frameworks, and operating systems,” replied the lead developer.

“Didn’t someone just buy an EAI engine? Couldn’t we use that?” asked a third developer. Everyone laughed.

“What about CORBA?” queried the new guy.

“What about what?”

“CORBA. Look, see. Interoperable remote procedure calls. Keep the business logic over here, publish the interface, let the client worry about the rest. I’ve already built a prototype.”

The IT people were impressed. They showed the prototype to Finance and Consulting. The prototype was quickly moved into production. Flush with success, the lead developer said, “You know what, we should make everyone do this. From now on no more application silos. Everyone must make functionality available over the network.” But few listened. The Microsoft guys said that it sounded right in principle, but we should use DCOM. The mainframe guys went on building CICS apps. And, frankly, it went right over the heads of the PowerBuilder and ColdFusion guys. Worse, CORBA wasn’t quite as easy to use as it seemed. It was complex, nothing interoperated, it didn’t work through firewalls, and a bewildering number of specs were coming out of the OMG.

“Well, how about this new SOAP thing,” said the new new guy. “It’s simple, it works through firewalls, all the vendors are on board, and, look, there’s only three specs, and we don’t need this UDDI one.” There was much rejoicing. “This just has to work,” said the lead developer (now promoted to enterprise architect). “I can’t prove it, but I can feel it in my bones. We just need to give this idea a name.”

“I’ve heard it called service-oriented architecture,” said the new new guy.

“Service-oriented architecture, ay? I like it. Nice acronym too. SOA. Lets pronounce it soh-uh.”

Soon, the word spread throughout the land: No more silos. Make functionality available on the network using standardized, interoperable protocols. We’re service-oriented now. And, believe it or not, people got it. Now that they could see that it just might work, it seemed like a painfully obvious idea. Building silos is bad. Exposing functionality on the network is good.

And… Well, you know the rest. They picked the wrong technology again, despite the fact that the right choice was staring them in the face. Like CORBA before it, SOAP wasn’t quite as easy to use as it seemed, interoperability was problematic, and a bewildering number of specs were coming out of the W3C, OASIS, and the vendors themselves. Not only that, there was a big pile-on of suspect products, pie-in-the-sky promises, and ever changing architectures. But, really, all that’s besides the point. The point is that more people than not understood the value of deploying standards-based, network accessible business logic.

So, then, what is SOA? For one thing, SOA is misnamed. It’s not an architecture in any sense of the word. It is, to use a Burton Group phrase, a mind set. It is the generally held belief that when implementing systems one should expose system functionality for general consumption directly from the network, as well as or instead of burying it behind a user interface. It is, as well, the belief that there is a great deal of value to be generated by retrofitting network accessibility into most existing systems. And it is the belief that this can only work if the means of doing so aren’t locked to a particular language, framework, operating system, vendor, or network architecture.

Another problem with the SOA name is the “service” bit. At least for me, the term “service” connotes a collection of non-uniform operations. I don’t even like the phrase “REST Web services.” Certainly, SOAP/WS-*, CORBA, DCOM, etc. fit this definition. But REST? Not so much. In REST the key abstraction is the resource, not the service interface. Therefore SOA (and I know this is not anyone’s strict definition) encompasses the above mind set, but includes SOAP and similar technologies and excludes REST.

A better name for SOA, then, might be network-oriented computing (NOC). This encompasses both WS-* and REST (and most everything else from the socket level up). We can, if we want, make SOA and resource-oriented architecture (ROA) a subset of NOC. In which case the “architecture” bit makes sense again.

“But wait,” I hear my SOA-loving readers say. “SOA is not about exposing business logic on the network. That’s just a technology thing. SOA is about the business! CxOs and business units don’t care about technology, they will only pay for business solutions.” Which always makes me scratch my head. What exactly does IT ever do that’s not about the business? Do they not work for the same company as the other BUs? Is a firewall about the business? Of course it is; there’s a business requirement to maintain information security. Is a router about the business? Obviously, the business is demanding networked communications. Is an application server about the business? Yep, having someone else write all the plumbing gets systems out faster, and that’s a business requirement. How about Agile? That too is a business requirement, faster, better software. Testing? Yes. VoIP? Yes. SOA? Yes.

“No, no,” the SOA-is-business advocates reply. “It’s not just ‘business’ it’s better alignment with business. You see, if we can identify a business process such as ‘open new account,’ then we can create a service called OpenNewAccount. You see how those things line up there.” All well and good, say I, but as Stu and Steve say, “Things change. And besides, what if you’re in perfect alignment with the business, but the system doesn’t work? What if it doesn’t scale? What if clients can’t use it? What if the programmers can’t code it?”

“Well, it’s not just about business alignment,” another group of SOA advocates claim. “It’s really about governance.” Again, I’m scratching my head here. Everything is about governance! Software development (network-oriented or not) is about governance: ‘You must use a version control system; you must write unit tests.’ Moving systems into production is about governance. Updating a routing table is about governance. Hiring a new employee is about governance. Buying a plane ticket is about governance.

“No, no.” They go on. “With SOA there’s new things to govern.” That’s true, there are. But really, is it that much different than any other governance process? Not really.

(By the way, I’m also on record as saying that REST requires less governance than WS-*. While others might say the governance needs are the same. I stand by what I say by noting that if there’s any chance in hell of getting all this to work, it’s going to require a truck-load of governance.)

So, that’s it. As Stefan also points out, SOA today has a number of fluid definitions: It’s the notion of tearing down silos and making functionality available on the network (frequently with the WS-* stack implied), it’s the use of governance to ensure that people do this right, and its the alignment of business with IT. If any of these can be considered more right than the other (by usage or historical precedent), then I would have to say it’s the first.

No matter which definition works for you, though, SOA is misnamed. So I’ll leave you with some updates to your lexicon:

• Network Oriented Computing (NOC): An approach to computing that makes business logic available over the network in a standardized and interoperable manner.
  • Service Oriented Architecture (SOA): A technical approach to NOC that has a non-uniform service interface as its principle abstraction. Today, SOAP/WS-* is the chief implementation approach.
  • Resource Oriented Architecture (ROA): A technical approach to NOC that has an addressable, stateful resource as its principle abstraction. Today, REST/HTTP is the chief implementation approach.
• Business Service Architecture (BSA): An unnecessary term (also not an architecture) that tries to make the obvious something special. Aka, business analysis. Aka, requirements gathering.

Over on REST-Discuss, Roy Fielding dropped an innocuous message containing a link to the slides he’s delivering (has delivered) today at RailsConf Europe.  I’m pleased to see that much of the material lines up nicely with my own REST Easy workshop.

You may or may not know about Burton Group’s Catalyst Conference that’s held twice a year; once in North America and once in Europe. But at the North American conference we offer in-depth, 4-8 hour workshops on a variety of subjects. This year, I created and delivered a four hour workshop on REST, though it went for 4.5 hours and could easily have been six. The workshop gave a concise overview of REST, a lengthy tutorial on how to build RESTful systems, and a brief comparison of REST with WS-*. It even include working code to illustrate most of the concepts.

Well, after just a little bit of cajoling on my part, Burton Group is making the REST Workshop slides available—for free—to everyone. Yes, they’re behind a registration wall. And, yes, you’re likely to be contacted by us if you download them. But, if I do say so myself, it’s a really good deck. It was even rated the best workshop at Catalyst by attendees. Now, paging through some slides is not going to make you all knowledgeable, but if you haven’t got your mind fully around REST, maybe they’ll help. More important to the readers of this blog is that you now have a resource that you can repurpose to help sell and explain REST to your colleagues and managers.

As mentioned, the workshop as delivered also showcased some running code. I’ve left the code out of this offering for several reasons. Mainly code quality. I wrote the services in Java using Restlet 1.0 (and the clients in Ruby). But Restlet’s a moving target, and is now at 1.4. Furthermore, I wrote it fast, while learning Restlet at the same time. And I ignored everything except what was being demonstrated, like error handling and comments. In other words, while it all works, it’s ugly and out of date. BTW, this explains why a number of slides are given over to Restlet architecture.

Aside: Restlet rocks. The Restlet mailing list rocks. Jerome Louvel rocks. The documentation, well, it doesn’t rock. Note to Tim O’Reilly: Please give Jermoe enough money to live on for 6-12 months and have him write a Restlet book.

Burton Group also asked me to create a “Take 5″ on the subject. A Take 5 is an audio enhanced five minute PowerPoint show on a pertinent topic. The REST Take 5 is really 15 minutes, but gives a good 40,000 foot, managerial overview of REST. (That link goes straight to the 10MB PPT deck, unless you need to register or login. You’ll need to view the deck in slide show mode to hear the voice over.) Give it a listen over your coffee this morning, email the link to your boss and the EA team, speak truth to power and all that.

Anyway, I hope you go through the trouble of downloading this material. Largely, it was you that created it (you might even find yourself quoted in there); I just compiled it. Of course, any errors are mine, so let me know if you find any and I’ll fix them. And let me know if you’d like to have me present this to your organization (it’s much better with a narrator to provide the details and answer questions).

Minor update:  A couple of errors in the installation instructions have been fixed and atom-tools has revved to 0.9.4, fixing all known issues.  If you’re using appfs, you should upgrade your atom-tools gem.
$ mkdir myblog
$ appfs.rb myblog
$ cd myblog
$ ls -l
dr-xr-xr-x 1 placey placey 4096 2007-08-02 11:25 WordPress_Workspace
$ cd WordPress_Workspace
$ ls -l
dr-xr-xr-x 1 placey placey 4096 2007-08-02 11:25 WordPress_Media
dr-xr-xr-x 1 placey placey 4096 2007-08-02 11:25 WordPress_Posts
$ cd WordPress_Posts
$ ls -l
dr-xr-xr-x 1 placey placey 4096 2007-08-02 11:25 drafts
dr-xr-xr-x 1 placey placey 4096 2007-08-02 11:25 published
$ cd published
$ ls -l
-rw-rw-rw- 1 placey placey 0 2007-08-02 11:25 First_Post
$ cat First_Post
This is a test. This is <em>nothing</em> but a test. If this had been an actual post, then I wouldn’t have many readers.
$ vim First_Post
This is a test. This is <em>nothing</em> but a test. If this had been an actual post, then I wouldn’t have many readers. Look *RedCloth* support.
:wq
$ cat First_Post
This is a test. This is <em>nothing</em> but a test. If this had been an actual post, then I wouldn’t have many readers. Look <strong>RedCloth</strong> support.
$ vim New_Post
I can create new posts. too. The title will be the filename with underscores replaced by spaces.
:wq
$ ls -l
-rw-rw-rw- 1 placey placey 0 2007-08-02 11:25 First_Post
-rw-rw-rw- 1 placey placey 0 2007-08-02 11:25 New_Post
$ cd ../drafts
$ vim Another_Post
When I create files in the “drafts” directory, they are published as drafts in the Atompub server (if it supports it).
:wq
$ rm Another_Post
$ vim Yet_Another_Post
This post is worthy of publishing.
$ mv Yet_Another_Post ../published

---

Installation instructions. Usage Instructions.appfs leverages Fuse (Filesystem in Userspace) and the Fuse Ruby binding, FuseFS, as well as the Atom-Tools Ruby library by Brendan Taylor. appfs simply glues the two together.If you want to play with this and don’t have an Atompub enabled server, I have created an anonymous account on a WordPress install I care nothing about. You can access this server at:url: http://72.249.21.88/nonintersecting/wp-app.php/service
user: anon
pass: sekrit

(Incidentally, that’s a valid appfs config file.)

appfs is tested only against a (heavily patched) WordPress 2.2. I would love people to try it against other Atompub servers.

In this first release there are a number of limitations, all of which will be addressed in the near future.

Limitations

• No support for media entries
• No support for categories
• XHTML content only
• Cannot create/delete workspaces or collections

I decided to use WordPress 2.2 in order to test appfs. That may have not been the best decision as the Atompub code in WordPress is chock full of bugs. The good news is that I fixed all the bugs that I found. I’ve reported them to the WordPress developers, but most won’t be fixed until 2.4. So I’ll make the fixes available here.

WordPress 2.2 Atompub bugs fixed here

• Service document uses Atompub 09 formatting
• Draft entries not included in a feed
• If the above worked, drafts would have been shown to all feed consumers, not just the draft entry authors (enhancement, not a bug)
• Entries updated from draft to published are not actually updated
• Atom parser doesn’t parse properly

To fix that last one, I grabbed Elias Torres‘ PHP Atom Parser, phpatomlib, and updated the wp-app.php code accordingly. Elias is the original author of the Atompub support in WordPress, but his code has been assumed into the trunk, so Elias broke out the parser for continual refinement (fortunately).

To use these fixes, backup your existing wp-app.php file and replace it with this one. Then drop the latest version of the atomlib.php parser (Revision 7, copied here for convenience) into the same (WordPress top level) directory.

Update: Tim Bray found some more lingering issues in WordPress’s support for Atompub. I fixed these and updated the local version of wp-app.php. This does not guarantee that WordPress can now be made fully Atompub compliant. Unfortunately, the Ape is having some issues of its own right now. But I’ll squash ‘em as I find ‘em.

Also unfortunately, fixing the WordPress issues exposed similar issues with atom-tools. I have fixes if needed, but Brendan is pretty responsive and hopefully will put out an 0.9.4 soon.

I’m heads down with work related issues, and so have not been able to participate or even keep up with the latest REST vs WS-* permathreads over on the SOA discussion list (the Web interface for which sucks so much I can’t give direct links), or provide any insight or opinion here.  For the time being then, “What Stu said.”

In the past I noted that while REST doesn’t say anything about transport- or message-level security, there does not currently exist any standard means of using message-level security in a RESTful system. I was wrong.

I don’t know how I missed it, but section 5 of the Atom Syndication Format (Atom) describes how to encrypt and sign an Atom document. Like WS-Security, Atom leverages the existing W3C specifications for digital signatures and encryption. What does Atom have to do with REST? Technically, nothing. It’s just a well known, high-level, media type that you are strongly encouraged to use as your representation format—as an envelope around your content.

Traditionally, Atom feeds are polled by subscribers and thus forms a pub/sub model. Securing the feeds or the internal entries will allow me, for instance, to subscribe to my cell phone usage data and know that the message is confidential, unmodified in transit, and really created by my cellular provider. But what about the more common request/response needs, where, say, a buyer wants to purchase 1,000 widgets? Well, that’s where the Atom Publishing Protocol comes in. I, the supplier, can expose my order processor as a resource (http://example.com/orders) and my customers can POST a signed and encrypted purchase order wrapped up in an Atom entry to that URL.

What about authentication credentials? Well, if the entry is signed, then the signature serves as the originator’s credentials. Coolness! If for some reason digital signatures don’t work for you, then Atom supplies an extensible author element. It’s lacking a password field, but to address that just drag in the WSS Username Token Profile: