If transactions over REST are unachievable, how can REST ever be really useful? [closed]

Question

When looking into REST one of the first things probably anybody will notice is there isn't any transaction semantics defined, some say this is implicitly against what REST is, whilst others say any attempt to do so would result in 'tainting' the REST systems.

But lets say for arguments sake that REST did become a popular 'api' choice, and every site in the universe started to expose restful entry points.

How exactly are these usable without transaction behavior (and I'm talking non compensating)? because it seems to me one of the benefits of REST is that it breaks down the components of data, this you would think opens them up to having smart clients compose data (and add and adjust such composed data) from multiple services. But if I cannot do my changes to this composition of data atomically and in isolation then using REST becomes useless.

As time goes on and the need for serious data exposition arrives, we're gonna want something that is: Simple (REST wins there), and supports transactional behavior so we can manipulate this data reliably.

Now, I've read a specific argument a couple of times in my research, and its related to how we're supposed to think about transactions in REST, and the example given is the shopping cart, where you implicitly have isolation because the cart is YOURS.

However I disagree with this argument, firstly, the isolation a shopping cart has is merely convenient, this isn't a transaction isolation.. what happens if I'm simultaneously doing an operation against my cart whilst some part of my application is reading data from it? I wouldn't expect the reading part of my application to see data that is 'still in transaction'.

Not to mention the fact that not all data changes have an implicit transaction model, transactions over multiple services definitely don't.

It seems to me that transactions need to happen, and need to happen in a way that enables the actual REST calls to be ignorant of the fact (adding to the rest payload being a big no no, but adding headers being OK).

I've read a few suggestions about how a transaction model can be created over REST, and some of the specifications being written seem to be very recent.

Are there any real thoughts about this? shouldn't there be something 'more' than REST so that REST's simplistic nature can be harnessed against solid data manipulation ('acid' transactions).

If not are we expected to really UP the ante, and tell service developers that if they want to interact in a pure data world they need to support something arguably monolithic like soap? or even worse try and build their own custom transaction support into something like REST, making each service non-standard and breaking the entire power of REST?

Thanks in advance for any thoughts.

---

Edit, added brief scenario:

Imagine a client form that handle creation of an Album, for convenience on that Album, rather than asking the user to give the uri for the artist resource, they can pick from a list of artists (most likely GET'd from the artists catalog).

For sake of usability the client can write the artist name in manually so they can create an artist 'inline'.. in the posting scenario, the client code understands this, and before sending the request to create the album, it firstly attempts to determine if the artist already exists, if so, gets the uri for that artist, otherwise creates the artist and gets the artists uri.

The client code then continues on to create the album, this is the smarter than usual client, it isn't sat right on top of REST and 'dumbly' posting, but instead has some interaction that handles the purer REST logic.

However in this scenario it would be nice to guarantee that the artist isn't created unless the album is, given the artist is created first.

This isn't as 'critical' as a transaction would imply, but it defines a set of work the client code would prefer to be happening as one operation (after all, it IS making this look like a single operation to the user).

The only guidance I've seen for this scenario is to have the client do compensating action in the event the album creation fails, and specifically call to delete the artist. But this seems problematic, because the client is assuming the artist was isolated, as unlikely as it may be, what happens if another client already 'saw' that artist, and assigns to it?

These are my concerns regarding doing data changes, and whilst there are certainly other gaps (who says the artist couldn't be just deleted at a later date anyway), those actions are NOT transparent (ie, the actions aren't something automated by the client, but something a user has specifically requested).

Suggestive

Firstly, you reference some things but don't provide links. I'd suggest you do that, when formalising such a lengthy argument. Secondly, I voted to close, as this is obviously argumentative and probably better suited for a blog. Thirdly, I don't see the issue. If you want transactioning, for some purpose, then can't you just implement it? Or are you dishonouring "someone" if you do so? I don't think so. I think all so-called "REST" implementations are different anyway, so it's fairly meaningless, IMHO. – Kochi

@meandmycode: I think this is a good question, but it's liable to get closed if you don't make it Community Wiki (edit the question, check the CW box), because it's subjective and has no correct answer. – Quixotic

@skaffman: CW is not meant to excuse a subjective & argumentative question. – Iaverne

@John: Agreed. However, compared to the dozens of "what's your favourite programming joke"-style questions, this one is useful and well thought out. – Quixotic

@skaffman: two wrongs... – Iaverne

OK, but if you didn't like the question, why did you answer it? – Quixotic

Thanks for the feedback, I changed this to CW, as for closing the question, it seems a bit backwards, if this is such a troublesome topic then people should argue their case.. I'd rather get some form of information exchange than never create the topic and sit 'wondering to myself'. – Suggestive

@silky You accuse the original poster of being argumentative and then finish with "REST [is] fairly meaningless". – Scheffler

Whether REST should/can support transactions is a regularly debated topic and is a very valid question for StackOverflow. I see no reason why it should have been made a community wiki. – Scheffler

@Darrel: I agree that it's a valid question, but with the way SO works, CW is the way to go. – Quixotic

@Darrel: Hi. Whether to permit "discussion" questions on SO is a regularly debated topic, and is a very valid question for meta.stackoverflow.com. :-) – Iaverne

@Darrel: While a specific question about using transactions with a REST service may very well be appropriate for SO, a general discussion on whether REST can be a valid tool without distributed transaction support is appropriate for some other site - not for a Q&A site like SO. – Iaverne

@skaffman: I answered because it wasn't getting closed - it might actually survive - and because I've been a developer of distributed transaction systems in addition to my experience with web services. Most people haven't had the pleasure. The nature of the "transactions" that the OP was talking about was getting lost. – Iaverne

@Darrel: That's not at all how I finished my statement. – Kochi

Answer 1

I am going to assume that when you talk about transactions you are talking about a distributed Two Phase Commit protocol.

If I understand correctly you are trying to understand how we could ever use REST to perform operations that span multiple systems if REST cannot support transactions across distinct REST requests. The problem is you are making a potentially flawed assumption that we should be using transactions to achieve consistency. What price are we paying for using them, and what alternatives exist?

Pat Helland who used to work for Amazon and is now at Microsoft, wrote a paper Life beyond Distributed Transactions. In the paper the Author makes the following statement:

Unfortunately, programmers striving to solve business goals like eCommerce, supply-chain-management, financial, and health-care applications increasingly need to think about scaling without distributed transactions. They do this because attempts to use distributed transactions are too fragile and perform poorly.

His paper explores alternative solutions to distributed transactions that do scale and perform well.

Maybe REST will be successful because it does not support transactions. Here is a quote from Roy Fielding, the guy who invented the term REST

If you find yourself in need of a distributed transaction protocol, then how can you possibly say that your architecture is based on REST? I simply cannot see how you can get from one situation (of using RESTful application state on the client and hypermedia to determine all state transitions) to the next situation of needing distributed agreement of transaction semantics wherein the client has to tell the server how to manage its own resources.

...for now I consider "rest transaction" to be an oxymoron.

This is from a message on the REST-discuss list from June 9th, 2009. I can't provide a link because Yahoo groups is useless.

Scheffler

Hi Darrel, thanks for posting a reply, I'm certainly not out to try and infer my own theory on how REST should work, this merely started with 'OK, so I use REST to do these scenarios.. what happens when I want to do this?' and these questions arise, transactions might not be the exact semantics I'm trying to describe, but I hope my scenario concerns are better illustrated in the edits I made to the original question. Thanks for the feedback. – Suggestive

@Suggestive I think what you are trying to describe is a workflow. Have a look at How to Get a Cup of Coffee: infoq.com/articles/webber-rest-workflow – Charlinecharlock

Could you please answer #9483786 ? – Someday

Answer 2

If you want transactions in a ReST application, at the ReST API function, it's usually because you still have your techie webservice guy googles on.

Let's look at it another way.

Soap googles on: Open a transaction, create a customer record, create an order record, commit transaction.

ReST googles on: Ask server what to do. Server says "create customer resource", so POST /customers. Server says, "now create an order if you want", client creates the order by following the form.

In ReST, the application protocol is expressed in terms of resources being created and manipulated, not in terms of data that has transaction boundaries.

If you want to have a long-running transaction that spans all those operations, it's the server that decides to initiate it, not the client.

You can still implement long running transactions on the server side. If you attempt to want transactions from the client side, then you assume the client already knows all the operations it's going ot execute and the transaction boundaries that exist between those operations. If that's what your expectations are of the client, you've already given up the late binding, hypermedia-driven nature of a rest architecture.

So indeed, if you're not doing ReST and are trying to fit in RPC over http, you'll have a problem with not having transactions.

Kailyard

Giggles at googles cause it's goggles. – Tabethatabib

The problem with your example is it entirely server orientated, the client is what is important, in a world where the client wants to orientate changes to a domain and wants them to happen all or nothing the argument of 'you are thinking about it wrong' falls over. I definitely get the example, but I think it is perhaps too purist about what REST clients are about, REST as a simple way to communicate data changes is predominately what I think is important. – Suggestive

If you redefine the term REST to mean what is convenient to you, then you are not going to get very helpful answers. Believe it or not, REST has a very well defined meaning and within that context REST clients can only do what the server says they can do via hypermedia. In RESTful systems the server has way more control over the conversation than in RPC based systems. The client simply follows one of the options presented to it. – Scheffler

I'm not trying to redefine REST, but in terms of how real physics ui clients interact with them, I'm getting a better feeling for the reason why transactions in such a controlled fashion are not ideal, but really a unit of work is somewhat important. – Suggestive

Again, in ReST the server is responsible for declaring unit of work boundaries, not the client. If you want the client to know about such things, as opposed to cross teh boundaries by following where the server lead it, then you're better off putting more knowledge in the client than ReST allows. You'll have stronger coupling, you wont be ReST, but whatever it is you're trying to do you'll be able to do. – Kailyard

Really, when you stop and think, serialseb is absolutely right. We get our terms mixed up when talking about web apps b/c we call client what is really just the service boundary of the server. Web pages are messages to a client (typically the browser) from the server. – Charlinecharlock

Answer 3

I think most actions that normally require transactions can be reworked to occur without them.

For example, the classic bank transfer. Suppose I want to move $100 from account A to B:

Begin Transaction
  /Debit  A, $100  
  /Credit B, $100
Commit Transaction

This could be reworked as:

 /Transfer  A, B, $100  

In this way, the server might do this in two steps, but the action from the client is a single, atomic operation that makes logical sense.

I'm sure there are lots of examples where it is more convenient to do an all or nothing set of operations (and I'm curious what people can come up with to address them), but I usually rework things in this way.

Jovanjove

What happens when the transfer is complete at server but just before the response could be written on the wire, internet connection breaks? Client will assume that the transfer errored out. In this event, the customer is going to try the transfer again. – Longs

I would suggest using optimistic concurrency in that case. E.g. when you send your transaction, include some value that indicates the state of the data as you know it. The server can check this "last updated" value when it does its atomic operation. If they match: continue. If not: error. When the client get this concurrency error it can refresh everything and see that the transaction already happened. Note that this problem is not exclusive to REST-- regular RDBMSs have it, too. Related: en.wikipedia.org/wiki/Two_Generals'_Problem – Jovanjove

@Longs This would be clearly a REST client issue, and has nothing to do with the service... – Perfectible

Isn't this RPC instead of REST? – Imbalance

@Imbalance Everytime I've seen this debate, the solution proposed is to encapsulate the unit of work in a single rest resource, which is an RPC method with a really strange name. They take a verb phrase and turn it into a noun phrase. The resource semantics become strange, and what you have is an RPC style method with a resource style name. Same suggestion in other answers: "You can implement ... transactions on the server side." My opinion: strive for REST, leverage optimistic concurrency to minimize need for transactions, but occasionally you need a couple RPC style operations on an API. – Wormeaten

Yes, this kinda makes sense, so the idea is to try to do everything as much CRUD/REST as possible, so basically part of what REST actually means is to keep try the API as simple as possible, i.e. CRUD + transactions limited to single entities/rows (using optimistic "locking"). OTH, Transfer in this example can be seen as an event. This resonates nicely with the idea of event sourcing. – Imbalance

Answer 4

A REST operation can start a transaction, perform multiple database or other transactional operations, then commit or rollback - all within the scope of a transaction.

What REST cannot do is to be other than the root of a transaction. You can't start a transaction, then perform two REST operations and a database operation, then commit them all together. That's just like the situation of ASMX web services in .NET. They can be the root of a transaction, but that's all. They were successful for years, until WCF was introduced, supporting WS-Transactions. Even today and using WCF, most web service operations do not need to be transactional in the sense you're asking about.

Iaverne

I think the majority of people understand that within the 'operation' that rest is acting it would be transactional like, in the sense the changes to that entity should appear atomic and shouldn't leave the data in a broken state. This seems somewhat obvious and perhaps even outside of the scope of REST, I would expect a single operation to be successful or not. My question is specifically about getting transactions between operations and what peoples feedback about A) thoughts about doing this, B) attempts to make this happen they've seen. – Suggestive

Answer 5

This is an interesting topic. As you mentioned, SOAP has this sort of functionality already, but it took many years before SOAP matured to the point where people would consider doing real security and transactions with it. Before that, it was CORBA.

The various high-grade extensions to SOAP, including security and transactions, took a lot of work by a lot of people to get them right. This isn't going to happen to REST overnight, and it may never happen at all.

I feel that much of REST's currently popularity is something of a backlash against poorly-designed and over-complicated SOAP implementations, which is a shame, because SOAP doesn't have to be like that. As with anything else, it needs good design to make it work nicely.

Quixotic

Personally I don't mind soap, I think the interchange is completely for the machines so to what degree those messages are 'verbose' or not doesn't really bother me, and I think as an architecture it works fine. However, it does raise the bar on how sophisticated a client has to be, which is where REST comes in, of course a fully thought out transaction model for REST increases the bar, but I don't think anywhere near that of SOAP. – Suggestive

Answer 6

I think that transactions using REST are actually achievable. Think of a transaction as a resource which you can create (start), edit (post changes through), and delete (commit, roll back). Changes posted to the transaction wouldn't need to modify the global state until the transaction was committed, and during the commit, you could enforce any global state consistency rules you'd need to. The transaction resource would, of course, be protected through authorization rules.

You already mentioned the common illustration for this concept:

Now, I've read a specific argument a couple of times in my research, and its related to how we're supposed to think about transactions in REST, and the example given is the shopping cart, where you implicitly have isolation because the cart is YOURS.

However I disagree with this argument, firstly, the isolation a shopping cart has is merely convenient, this isn't a transaction isolation.. what happens if I'm simultaneously doing an operation against my cart whilst some part of my application is reading data from it? I wouldn't expect the reading part of my application to see data that is 'still in transaction'.

Why wouldn't you allow the viewing of the transaction? As long as you present it as what it is, a list of pending changes, it's actually helpful to provide that feature. But if you didn't want it to be viewable, you could disabling GET on the resource.

Obe

@Jacob: this doesn't work unless all parts of the transaction are being performed by the same service. What if part 1 is service 1, part 2 is service 2, part 3 is service 1 again, and part 4 is in the local database? Your technique can't solve that problem. – Iaverne

Sure it can. The cooperating services just need to communicate the URL, and the coordinating server needs to set up the rules for authorizing access to the transaction. – Obe

that works fine if the services cooperate. But what if they're from different vendors who have never heard of each other? – Iaverne

@JohnSaunders Just curiosity, how do you implement transactions by services "from different vendors who have never heard of each other" with other techniques? – Perfectible

I can tell you, you will never achieve that, because you send your commands over the network, and you can always lost network connection... Therefore you will never implement any transaction solution over a network in which you cannot perfectly control the connections... Btw I don't think REST needs such a feature... – Perfectible

@inf3rno: sorry, but you're mistaken. Take a look at WS-ReliableMessaging, which takes network failures into account. – Iaverne

@JohnSaunders Yes, it is possible to check whether a message is received, but that is irrelevant. You cannot do anything to solve issues caused by persistent connection losses. So it is possible to have a situation when one of the services got a commit request and another a rollback request, and you cannot contact with them. If the client handles the transaction, it can even forget the error by a restart... So it's far from a robust system, and I don't think you can implement any ACID property on your transactions, but good luck! – Perfectible

@inf3rno: I guess you didn't read the Wikipedia article that I linked to. This has been sorted for well over a decade with non-SOAP solutions like HP's Reliable Transaction Router. Which product I was one of the developers of, so kindly stop telling me it can't be done. This is software that has been running banks, stock exchanges and railroads, so you seem to be disagreeing with reality. – Iaverne

It's hard to accept :-P Maybe I just don't understand the algorithms behind these solutions, can you tell me more about them? (links are enough) – Perfectible

Answer 7

Because not every system requires transactions.

Jeep

And of the systems which require transactions, most will not require distributed transactions. – Iaverne

Says who? if you are writing an open api, how do you know who will consume your service? the topic probably isn't best described, I'm sure REST will always have its place, but when it comes to composition of services that weren't specifically intended to 'transact' together, REST is seemingly out of the question.. so is that it? is that ah well, that's the limitation, now go write a soap service like a good boy, or is there actually some work to try and layer some some of transact behaviour over REST? lots of headers can change the results of a REST request, so why not an (ie) transact header? – Suggestive

@meandmycode: just counting heads. It's not my fault if there aren't as many heads in the room as you'd like. How many SOAP web services are required to be part of a distributed transaction? I've yet to see my first one. – Iaverne

@John, I'm not sure what you mean at the start, but in terms of transactions, I wrote a scenario (which is common) in the initial post, I noted points about how without a transaction, the set of operations has to deal with rollbacks manually, and how these actions are still flawed. Having the option to say, well those things don't bother me is one thing, but some systems having such 'untidy' ends IS problematic, and performance overhead of a transaction is absolutely acceptable vs. 'corrupting' data. – Suggestive

@meandmycode: what we seem to be arguing about is my suggestion that the set of systems that require distributed transactions is a subset of the set of systems that require transactions. – Iaverne

Answer 8

Actually, transactions in REST work in much the same way that transactions are supposed to work in traditional SOAP services.

RPC-style transactions, where the client issues a "begin transaction" command (or some operation that implicitly begins but does not commit a transaction), followed by some additional operations, followed by another implicit/explicit "end transaction" command, are antiquated. The Web Service world moved away from the RPC model a long time ago; RPC/encoded in SOAP is not even WS-I compliant!

Instead of procedures, SOAP services today are built around the concept of messages. A single message contains all of the information necessary to perform a complete transaction. If you are submitting an order, the OrderRequest message will contain the customer information, order information, order details, payment details... everything that the server could possibly need to know about a single order.

We do have WS-Transactions that define semantics for distributed transactions, but these aren't really intended to support RPC-like semantics from the client, they're meant for orchestrations when you need to have several services participate in the same transaction. For example your order service needs to enlist a payment processing service to validate the credit card info and post a payment, which needs to be rolled back if the fulfillment service finds that you're out of stock, or something like that... you get the idea. This all takes place in the server environment so there's really nothing stopping you from doing the same thing in REST.

In REST, you have resources instead of messages, but the concept is really quite similar. Instead of an OrderRequest message the client submits (PUT) an Order resource, and the resource should contain all the same information necessary to complete the transaction. Admittedly, if you're trying to perform complex orchestrations, you might find REST a bit unwieldy compared to SOAP, but that doesn't mean you can't keep using REST for your front-end, just that SOAP will serve you better for your back-end services.

The reality, though, is that 99% of the time, people don't need the complexity of WS-Transactions. I know this because I use SOAP (WCF) almost exclusively and rarely use WS-Transactions.

In REST, the "state" resides on the client. The server tells the client, "here is all the information you need to give me to create this resource (complete this transaction)." But this blank form doesn't actually begin a transaction. It's up to the client to actually provide the information - fill in the form, so to speak. What the client does while it is filling in the form is of no concern to the server; when the client finally sends the finished form to the server, it is validated in its entirety. It's similar to a Unit of Work, except that the client is the one keeping track of the "work".

Hope this gives a better idea of how transactions can be and are achieved over REST. They just rely on a richer message/resource concept and a form of optimistic concurrency.

Lenticular

@Aaronaught: he's talking about distributed transactions, where several database operations, web service calls, or other transactional operations, are all combined into a single transaction using three-phase commit. – Iaverne

@John Saunders: That's not what it looks like to me. He seems to be talking about allowing clients to set transactional boundaries in order to wrap sequential web service calls in an atomic transaction; my answer was explaining why this makes no sense in a REST architecture or even a modern SOAP architecture. I even mentioned distributed transactions and why I don't think they apply to this question. – Lenticular

@Aaronaught: look at his third paragraph where he talks about clients being able to compose data from multiple sources. Also, consider that the multiple web service calls, in general, need not be on the same server or using the same service, and may, in general, be composed with other transactional operations. But not with REST. – Iaverne

@John Saunders: But then read his example, where he talks about adding an artist and then an album. That doesn't need to be a distributed transaction. Anyway, I've never heard of trying to compose a single resource across multiple services with REST; that seems to go against the very concept of REST. – Lenticular

@Aaronaught: the general question would be - if you had a set of SOAP services, database transactions, etc., could you replace one of the SOAP services with a REST service? The answer is "no". The question then becomes, "does this make REST useless?" – Iaverne

@Aaronaught: BTW, I don't think it makes REST useless. I think it makes it useless as part of a system of distributed transactions. That gives us one reason why REST shouldn't take over the world, at least not until there's no longer any use for distributed transactions across service boundaries. – Iaverne

@John Saunders: I agree completely, SOAP is not interchangeable with REST. REST imposes certain architectural constraints, one of which is that an atomic operation must be expressed as a single resource and managed by one service at a time. I also think that most (obviously not all) public-facing APIs can be expressed this way. REST won't take over the world; it's just a different (and useful) method of designing web services. – Lenticular

Answer 9

Take a look at the OpenStreetMap API. I think it is kind of "REST" and it provides kind of "transactions" – called 'changesets' there. Is that what you need?

Eugeniusz

It's not "Kind of Rest". It's not restful. URIs are documented, no links are used between documents. It's better than say Twitter or Flickr, but it's far from being a hypermedia application, hence it cannot be restful. – Kailyard

It's interesting, and its one of the examples of REST I've previously seen, but this imo taints the model of rest, the links become transaction aware, and this causes the linkage and therefore resources to become transaction aware, and this is definitely not the route I think transactional REST should take. – Suggestive

We've already seen that transactions are not somehting useful for the client to know about, and as ReST is about the client-server interaction, there is no such thing as transactional ReST. I'm not sure I understand your comment on links becoming transaction aware? Links are used by a client to operate a state change to the whole application. If a transaction boundary or UoW boundary is crossed by the clietn following the link, it is done without the client knowledge. – Kailyard

@serialseb: why do you restrict yourself to clients which cannot understand transactions? What's so special about transactions that you cannot imagine a client platform understanding them? – Iaverne

Because HTTP is the application protocol in ReST, and HTTP does not formulate transactions. Implementing explicit transaction support for the client to use (as opposed to trasaction boundaries being inherent to the state transitions) means you're reintroducing coupling where it didn't exist before. There's lots of litterature on ReST and transactions already that I won't repeat here. – Kailyard

Answer 10

I know of very successful online accounting systems (SaaS) which have a REST-based non-transactional API for creating, retrieving and manipulating invoices etc. They are widely acclaimed for their API and the ease of integration their system with other parties. The API is easy to maintain, usable from a variety of platforms and ensuring backwards compatability is reasonably straightforward.

The lack of transactions can be a real pain in the neck, but most of the time, when their servers aren't too loaded, the API works very well.

Sometimes, less-than-perfect is just good enough.

Alnico

I like the argument about simplicity but I think in terms of a bigger picture for how REST could be used this would require clients to be very disciplined and almost pre-guarantee their operation should work (bar catastrophic failures, such as servers going down). – Suggestive

Any reason why you did not name the service? We are always looking for good REST apis to use as examples. – Scheffler

@Darrel Miller: e-conomic.com – Alnico

I am unable to find any REST api information. The only api documentation that I can relates to using a client library that talks to the server using SOAP. Did I miss something? – Scheffler