Archive
Building RESTful Services – Part 1
Over the last few years REST adoption has seen tremendous growth. It is fast becoming the defacto standard for exposing a service to client applications over the web. Often however several services that attempt to be RESTful miss out on the benefits of using REST due to not looking past the basic concepts. There are significant benefits to both the service provider and the service consumer when an API is built and consumed RESTfully but in order to realize those benefits one has to understand and embrace all its principles. This is a three-part article structured as follows
Part 1 – Basics and the importance of leveraging the HTTP protocol
Part 2 – Importance of hypermedia in a RESTful service (HATEOAS)
Part 3 – Practical Issues in building a RESTful service
Shift in Thinking
The Web as a Platform
The web as it exists today consists of several building blocks such as user-agents, intermediaries such as various types of proxy servers, the web server , the http protocol and so on. A key REST concept involves treating this web infrastructure as a platform on which applications and APIs can be built on. Previous approaches such as WS* and SOAP have largely ignored the web and the protocol followed by the web (http). They simply use http as a transport protocol. A RESTful web service on the other hand understands the role played by each component of the web and uses them as building blocks. Another interesting way to look at this concept of embracing the web fully is by looking at the Richardson Maturity Model . This model gives us a structured way of looking at the degree to which a web services leverages the web infrastructure as building blocks. The higher your service is on the RMM scale, the more it leverages the web. That said, Level1 and Level2 of this scale are really prerequisites to get to level 3, which is where the real power, flexibility and scalability of RESTful services are realized. Furthermore one could easily argue that according to Fielding, without using hypermedia (Level3) a service is really not RESTful.
Thinking in Resources
The other key concept in REST is to think in terms of resources that can be manipulated using a fixed set of methods (HTTP Verbs) instead of thinking in terms of custom RPC methods exposed over http. We are used to modeling our business API in terms of several custom methods that can be invoked remotely. The RESTful way of modeling is to look at our business API in terms of Uniquely identifiable resources which can be manipulated using the http verbs. Let’s say we are dealing with a catalog system. Traditionally we would have several methods such as
http://server/estore/createCatalogItem,
http://server/estore/estore/editCatalogItem,
http://server/estore/estore/getCatalogItemDetails,
http://server/estore/estore/addReviewToCatalogItem,
http://server/estore/estore/addReviewToCatalogItem,
etc.
In a RESTful model, we would identify the key business “resources” as
http://server/estore/catalog/item/12345
http://server/estore/catalog/item/12345/reviews
and clients would manipulate these resources using the well-known set of http verbs such as PUT, POST, GET and DELETE (maybe a couple of others if required). Similarly, an email system can look at a user’s inbox messages as resources that can be manipulated
http://server/mail/folder/inbox/message/id (GET, POST, PUT and DELETE can be invoked on this message)
This conforms to how the web as a platform works, and makes the interface a lot more structured.
Doing REST Right
While there are several parts to doing REST right, this article attempts to stick in general with the Richardson Maturity Model to stress the importance of leveraging the web. While Level zero and Level one services don’t really get us anywhere, we’ll take a quick look at them to mainly highlight what services at these levels are NOT doing right!! .
RMM Level Zero
Services in this category don’t really make use the web as a platform since they don’t use any aspect of the web. Consider for example SOAP based web services (WS*). These typically expose a single URI and use a single http verb (POST). Services at this level simply use http to send a payload which dictates all aspects of what the client or server would do. Consider a SOAP based shopping web service which provides a generic endpoint, and all operations are made by the client by doing an HTTP POST against that end point.
POST /estore/ShoppingService HTTP/ 1.1 Content-Length : zzz
<request>
<method name="getItemsOnSale">
<param name="dept" value="electronics"/>
<param name="count" value="15"/>
</method>
</request>
This would then return a response with a list of items on sale in the electronics department. Similarly another method to add one of those sale items to the cart would look like
POST /estore/ShoppingService HTTP/ 1.1 Content-Length : zzz
<request>
<method name="addItemToCart">
<param name="itemId" value="123456789"/>
</method>
</request>
Notice that nothing in the http protocol is used by the client or the server at an application level. All the information required to fulfill the operations is self-contained in the payload. You could be passing the exact same messages over TCP sockets and the client and server would work exactly the same way. Http is that dispensable here !!.
RMM Level 1 – Using Unique Identifiers
Services at this level use URIs to uniquely identify resources. Instead of making every request to the same endpoint, requests are made to specific endpoints. These services leverage one important aspect of the web – each resource should be uniquely identifiable. So the request to get the Items on Sale becomes
POST /estore/getSaleItems ( typical POX implementations use POST everywhere and pass parameters in the request body)
<request>
<param name= "dept" value= "electronics" >
<param name= "count" value= "15" >
</request>
Note that this could be POST or GET. In level 1 no conscious decision is made as to whether GET/POST should be used since the http protocol itself is not necessarily respected here. The call made to add an item to the cart would now look like
GET (or POST)
/estore/addItemToCart?itemId=123456789
<order>
<cart>
<items>
<item code="abc" qty="1">
</items>
</cart>
</order>
Notice that making separate identifiable resources (yes they don’t look like resources, but that’s just the way we named them) eliminates the need to pass some information in the xml payload since this information is now taken care of in the identity of the URL called. So we don’t have to pass methodName=getSaleItems or methodName=addItemToCart anymore. Even though Level 1 services use a unique URI for services, they typically don’t make use of the http protocol in any way, and aren’t something to strive for as an end goal.
RMM Level 2 – Respecting the protocol (HTTP)
This section goes over various aspects of the HTTP protocol that play an important role in REST. There is an existing ecosystem consisting of millions of servers, and programs and user-agents all of which already understand HTTP. A RESTful approach to building services makes the case that by respecting the http protocol all this existing infrastructure can help our application scale. In the past we have been trained to think in terms of keeping the protocol and the data separate – we have always strived to keep our message payload protocol agnostic. We would in the past try very hard to build our system such that the same message payload could be used with http, tcp/ip asynchronous messaging etc. However in a RESTful system, http is NOT a transport protocol, but an application protocol. The http verb used to make requests and the http headers are a critical part of this application protocol. Several important concepts are conveyed via the headers (e.g. status code, content type, caching etc). Failure to adhere to this application protocol means that our application does not speak the language that all the user-agents and intermediaries understand and hence won’t be able to effectively leverage them as building blocks. This is a very broad area, but in general can be categorized into two general buckets
- Using the right http verb
- The importance of Http Headers (including status codes)
Getting the verb right
There are 4 commonly used http verbs (GET, POST, PUT, DELETE) and others that are not used as often such as HEAD , OPTIONS etc. Each verb has certain characteristics and it is critical to choose the right verb .
GET
GET should be used only for safe (no side effects for which the client is responsible) and idempotent (can be repeated multiple times with the same result) requests. So GET should not be used to update or delete a resource or to make a new resource. Every system that understands the http protocol assumes that a GET will NOT change the state of the system and hence will make assumptions based on that. For example a client which is not able to make a successful GET request is well within rights to make another attempt for the exact same request and see if the request can be fulfilled. So if GET is used for requests which are not safe there could be serious consequences. Consider for example a scenario where GET is used to add an item to an online shopping cart. Let’s say the item was added successfully on the back-end but due to a flaky connection, the client doesn’t get a success response. An abiding http user-agent (such as a browser or many of the http client libraries) will retry the request since GET is supposed to be safe and idempotent. Now we have a situation where the item exists in the cart twice even though the user wanted to add it only once.
Not using GET for safe idempotent requests also has undesired consequences. These mainly involve caching which are less serious functionally but could be serious with respect to scalability. In general, responses to GET requests can be cached by the client, proxies etc which is extremely critical from a performance and scalability point of view. There are lots of use cases where having a proxy server or the user’s browser cache the result of a request (could be data or things like images) would save a trip to the server resulting in saved computing power as well as saved bandwidth. Not using GET for such requests would result in a serious loss in scalability.
POST
POST can be used to either create a resource or update a resource. All http aware components assume that POST will make a change to the state of the system (not safe) and will not repeat a POST request in case of an error. For example any time you are submitting an order online and hit the submit button for the second time, the browser will prompt you to confirm that you really did want to make the request a second time, since POST is not idempotent. POST can also be used to create a resource but more about that a little later.
PUT
PUT is an important part of HTTP but due to not being part of HTML forms, a lot of people shy away from using PUT. PUT creates a resource for the URI specified by the client with the information in the request. If the resource already exists, PUT simply replaces what existed with the new information. Because of this, PUT is idempotent as long as the client uses it the right way ( with complete information as part of the request and not just a partial update) . If the same PUT call is made twice, the previous resource will simply be replaced by the same resource again.
PUT vs POST
There are two factors that determine whether PUT or POST should be used for creating or modifying a resource.
1. Who owns the URI identity ?
This is relevant while creating a new resource. Anytime the client is responsible and aware of the complete URI where the resource needs to be created, PUT can be used. However, often the client will not know the URI of the resource (since it may be generated by the server). Here using PUT is not possible and a POST needs to be used. For example a new online shopping trip for a customer may be tracked by an id generated by the server. When the first item is added to the cart, a new Cart resource is created by the server with a generated unique identifier as the cart id, which is part of the unique URI for the cart. The client will make a POST request with the required information and the server on creating the resource will use the Location header to send back the URI of the newly created resource.
POST /estore/order/ results in a new order created with id 178567 , and the server sends back the complete URI for the newly created resource in the location header 200 OK Location : /estore/order/178567
From this point onwards, the client can manipulate the URI returned in the Location header. So in general while creating a new resource, if the client owns the URI identify, then PUT should be used, otherwise POST is the only option.
2. Partial updates vs full replacement.
This is relevant for updating an existing resource. Once the client is aware of the URI of a resource (from the location header after resource creation) and wants to update the resource, it has two options. One is to use PUT and send the complete representation of the new resource. The other is to just send the delta. Sending a delta is often the more practical option (e.g. sending the item added to the cart instead of sending the entire cart + the extra item) and is a lot simpler on the client. The other reason why deltas are more practical is that a full resource representation involves hypermedia links which we really don’t want to be shipping back and forth between the client and server (more about hypermedia later). Now recall that a PUT request simply replaces the current resource with the new value. Hence PUT is not really suitable for sending deltas. So anytime we operate under the assumption that only deltas to the resource are being sent, the right method to use is the POST method (at least till the “PATCH” method garners more support).
Let’s look at the same example as earlier where a new cart was created with one item
GET /estore/order/178567
<order>
<cart>
<items>
<item code="abc" qty="1">
</items>
</cart>
</order>
Now lets say the client wants to add another item to the cart. The client could do a PUT with the representation of the entire cart,
PUT /estore/order/178567
<order>
<cart>
<items>
<item code="abc" qty="1"/>
<item code="def" qty="1"/>
</items>
</cart>
</order>
or the client could make a POST request with just the additional item such as
POST /estore/order/178567
<order>
<cart>
<items>
<item code="def" qty="1"/>
</items>
</cart>
</order>
The server would interpret this request as a delta request and add the new item to the existing cart.
So in either scenario, the current cart state is
GET /estore/order/178567
<order>
<cart>
<items>
<item code="abc" qty="1">
<item code="def" qty="1">
</items>
</cart>
</order>
Obviously, as the number of items in the cart starts increasing, the PUT to replace the entire cart isn’t that efficient or appealing.
DELETE
DELETE /estore/order/178567
This asks the server to do a logical delete of the resource. It is important to realize that this is a logical delete. The server can certainly move the existing order to archive, so that the order can now be accessed as
GET /estore/archive/order/178567
and the original request that we were using,
GET /estore/order/178567
would now result in a 404 Not Found.
Content Negotiation
A really key concept in REST is the idea that what is being exchanged between the client and server is a representation of the resource. It is certainly possible for different clients to ask for different representation of the same resource. For example, a mobile client may prefer to get JSON as the response format, while a web browser may prefer a pdf and some other client may only want xml. If the server decides to support multiple representations, then the same data can be presented in different formats. One way of doing this is to simply pass the type of response desired using a parameter such as
GET /eshop/sales/topItems/monday?format=json GET /eshop/sales/topItems/monday?format=pdf GET /eshop/sales/topItems/monday?format=xml
However this ignores the robust existing mechanism for “content negotiation” which is part of http. Embedding the content type in the URI in a non standard manner means that the url links are now not portable. If a person emails a link to the report from the web browser (which displays pdf) to the mobile phone (which expects json), the client needs to do ugly manipulations to the URI to convert the pdf request to a json request. Instead a much better way of dealing with multiple representations is to use the Accept and Content-Type headers which is well understood by all http clients. For example
The mobile request wanting JSON back would look like Accept: text/json GET /eshop/sales/topItems/monday and the web browser request which wants pdf would send Accept: application/pdf GET /eshop/sales/topItems/monday
Now the link can be passed around from one client to another and since the URI is unique, each client can use the http headers to tell the server what it can accept. Imagine another scenario where a client prefers a certain format, but is willing to degrade to a different format if the first one is not available. This is especially true in case of mobile clients. Handling this via the Accept header makes use of a well defined existing mechanism which all user agents and servers already understand and execute. No need for custom logic that needs to be reinvented every time.
Error Handling and Status
Anytime we write an API we are trained to send status codes and error codes. How often have we seen xml like this
200 OK <response> <statusCode>AE1987</statusCode> <errorCode>Authorization Failure </errorCode> </response>
Think about the thousands of APIs out there each having a custom way of doing this, reinventing the wheel every time. Further more, none of the intermediaries involved (proxies, etc) know that this is a response not worth caching, since they are all trained to look at the Http Status code (which in this case is 200, OK). The http protocol has a variety of status codes. For example
2xx status code = success 3xx status code = additional action is required by the client to finish the request 4xx status code = client has made an error and may need to modify the request before retrying 5xx status code = server is having problems handling the request
REST API developers should look here at the status codes to understand each error code and incorporate them in the API. Here are a few examples of how a a RESTful API should use them
200 OK - This should be used only for success and nothing else. Period. 201 Created - This should be used as a response to a POST request when a resource is created. The location header should contain the URI to the newly created resource 202 Accepted - This can be used for async processing, with a link to the status/next step being sent by the server. 400 Bad Request - This can be used for input validation errors. 403 Forbidden - Return this in case of an authentication error instead of returning a 200 405 Method Not Allowed - If there is a read only resource and a client makes a non GET request, this should be the status code returned by the server with the valid methods included as part of the Allow header 412 precondition - This can be used to indicate that a logical precondition check has failed
To summarize, the only status codes that are understood by the http clients as well as various intermediaries, are the http status codes. So misusing these status codes results in not being able to make use of the web infrastructure effectively.
Caching / Scalability
Caching and making use of intermediate servers for caching is critical to the scalability of a RESTful system. There are a wide range of http headers, both on the request as well as response side which heavily influence caching and performance and should be used appropriately. For example if the weather report for a day can be safely cached till the next update arrives, the weather service API in it’s response can indicate that using the appropriate http header . Some commonly used headers that can be used for smart caching include
Expires
This header in the response gives the time when the response should be considered stale. For example “Expires: Thu, 02 Jun 2011 16:00:00 GMT” tells not just the client but all other intermediaries as to how long to cache the response.
ETag
More about the ETag can be found here but to summarize, it serves as a way for caching, as well as a way to deal with optimistic concurrency. With each response, the server may send an ETag (Entity tag). Consider for example some large amounts of data that doesn’t change too often and hence is cached by the client.
GET /server/someLargeData
The response contains the header ETag : “123456”
Now if the client wants to refresh the data, it can make a request as follows
GET /server/someLargeData If-None-Match:
This will make the server return the complete large response, only if something caused the resource to change on the server (and hence have a new ETag different from the old one) . There are multiple ways to generate an ETag , such as using some hash of the data, or even a timestamp – something which changes every time the resource changes.
Another use of the ETag is to use it with the If-Match header in the request. Consider for example the possibility of a resource being edited by multiple users at the same time. In this scenario, the client simply checks for the right ETag, and if the ETag has changed, can tell the user that the resource has changed ,refresh with the latest data and ask the user to retry the operation again.
If-Modified-Since / If-Unmodified-Since
This request header can be used to ask the server for content based on modification time. This can help save network resources, similar to the ETag.
Summary
In this part of the article, we looked at some basic REST concepts, then focussed mainly at the importance of leveraging the HTTP protocol and how various aspects of it can be used in a RESTful web service. The important thing is to not stop here, but go one more step and incorporate hypermedia as an integral part of building web service to truly be RESTful. This will be covered in part 2 of this series.
Satish's Blog 