REST API Discoverability and HATEOAS

Table of Contents

1. Overview

This article will focus on Discoverability of the REST API, HATEOAS and practical scenarios driven by tests.

2. Why make the API Discoverable

Discoverability of an API is a topic that doesn’t get enough well deserved attention, and as a consequence very few APIs get it right. It is also something that, if done correctly, can make the API not only RESTful and usable but also elegant.

To understand discoverability you need to understand that constraint that is Hypermedia As The Engine Of Application State (HATEOAS); this constraint of a REST API is about full discoverability of actions/transitions on a Resource from Hypermedia (Hypertext really), as the only driver of application state.

If interaction is to be driven by the API through the conversation itself, concretely via Hypertext, then there can be no documentation, as that would coerce the client to make assumptions that are in fact outside of the context of the API.

In conclusion, the server should be descriptive enough to instruct the client how to use the API via Hypertext only, which, in the case of a HTTP conversation, may be the Link header.

3. Discoverability Scenarios (Driven by tests)

So what does it mean for a REST service to be discoverable? Throughout this section, we will test individual traits of discoverability using Junit, rest-assured and Hamcrest. Since the REST Service has been previously secured, each test first need to authenticate before consuming the API.

3.1. Discover the valid HTTP methods

When a REST Service is consumed with an invalid HTTP method, the response should be a 405 METHOD NOT ALLOWED; in addition, it should also help the client discover the valid HTTP methods that are allowed for that particular Resource, using the Allow HTTP Header in the response:

@Test
public void
 whenInvalidPOSTIsSentToValidURIOfResource_thenAllowHeaderListsTheAllowedActions(){
   // Given
   String uriOfExistingResource = restTemplate.createResource();

   // When
   Response res = givenAuth().post( uriOfExistingResource );

   // Then
   String allowHeader = res.getHeader( HttpHeaders.ALLOW );
   assertThat( allowHeader, AnyOf.<String> anyOf(
    containsString("GET"), containsString("PUT"), containsString("DELETE") ) );
}

3.2. Discover the URI of newly created Resource

The operation of creating a new Resource should always include the URI of the newly created resource in the response, using the Location HTTP Header. If the client does a GET on that URI, the resource should be available:

@Test
public void whenResourceIsCreated_thenUriOfTheNewlyCreatedResourceIsDiscoverable() {
    // When
    Foo newResource = new Foo(randomAlphabetic(6));
    Response createResp = givenAuth().contentType("application/json")
      .body(unpersistedResource).post(getFooURL());
    String uriOfNewResource= createResp.getHeader(HttpHeaders.LOCATION);

    // Then
    Response response = givenAuth().header(HttpHeaders.ACCEPT, "application/json")
      .get(uriOfNewResource);

    Foo resourceFromServer = response.body().as(Foo.class);
    assertThat(newResource, equalTo(resourceFromServer));
}

The test follows a simple scenario: a new Foo resource is created and the HTTP response is used to discover the URI where the Resource is now accessible. The tests then goes one step further and does a GET on that URI to retrieve the resource and compares it to the original, to make sure that it has been correctly persisted.

3.3. Discover the URI to GET All Resources of that type

When we GET any particular Foo resource, we should be able to discover what we can do next: we can list all the available Foo resources. Thus, the operation of retrieving an resource should always include in its response the URI where to get all the resources of that type, again making use of the Link header:

@Test
public void whenResourceIsRetrieved_thenUriToGetAllResourcesIsDiscoverable() {
    // Given
    String uriOfExistingResource = createAsUri();

    // When
    Response getResponse = givenAuth().get(uriOfExistingResource);

    // Then
    String uriToAllResources = HTTPLinkHeaderUtil
      .extractURIByRel(getResponse.getHeader("Link"), "collection");

    Response getAllResponse = givenAuth().get(uriToAllResources);
    assertThat(getAllResponse.getStatusCode(), is(200));
}

Note that the full low level code for extractURIByRel – responsible for extracting the URIs by rel relation is shown here.

This test covers the thorny subject of Link Relations in REST: the URI to retrieve all resources uses the rel=”collection” semantics.

This type of link relation has not yet been standardized, but is already in use by several microformats and proposed for standardization. Usage of non-standard link relations opens up the discussion about microformats and richer semantics in RESTful web services.

4. Other potential discoverable URIs and microformats

Other URIs could potentially be discovered via the Link header, but there is only so much the existing types of link relations allow without moving to a richer semantic markup such as defining custom link relations, the Atom Publishing Protocol or microformats, which will be the topic of another article.

For example the client should be able to discover the URI to create new Resources when doing a GET on a specific Resource; unfortunately there is no link relation to model create semantics. Luckily it is standard practice that the URI for creation is the same as the URI to GET all resources of that type, with the only difference being the POST HTTP method. Forms can also be used to achieve this.

5. Conclusion

We have seen how a REST API is fully discoverable from the root and with no prior knowledge – meaning the client is able to navigate it by doing a GET on the root. Moving forward, all state changes are driven by the client using the available and discoverable transitions that the REST API provides in representations (hence Representational State Transfer).

This article covered the some of the traits of discoverability in the context of a REST web service, discussing HTTP method discovery, the relation between create and get, discovery of the URI to get all resources, etc.

The implementation of all these examples and code snippets can be found in my github project – this is an Eclipse based project, so it should be easy to import and run as it is.

I usually post about REST APIs and HTTP on Google+ - you can follow me there:

Free eBook - REST Services with Spring
Join more than 2,200 engineers!

, ,

  • Kevin

    Hey, loving the idea behind this post, but I have been a hard time with this concept. I happen to be working with two companies (both start ups).. one is integrating with the other. I wrote the REST api (using java/jersey) for the “service” company, and the “client” company I am contracting for. I am trying to consume the service API from the client. I have been trying to keep the service API HATEOAS as much as possible.. it’s still a work in progress. I am glad you pointed out the use of “collection” as a rel type for a resource that returns multiple items of the given resource. I was just using GETALL.

    Anyway, the problem I have is discovery. I understand what you mean by there should be no documentation.. but for any developer using any modern day REST api, such as Facebook, Twitter, etc.. without information on the resources and what they do, there is no way we could consume it. This is where things start to fall apart for me. In my client, I am using the “entry” URL to get the list of resource links available to the client authenticated user. This allows me to tailor the response links based on the authenticated user.. if they are an admin, I return more links, if not, I return only what a developer at a given level should be able to use.

    So in my case, the response is a set of links. Now.. at this point, in our existing client app, if the user creates a new user, I have to look for the rel=”users” link and then submit a POST /username to create a new user on the service. Without knowing that rel=”users” allows me to create a new user.. I am not sure how I, a developer consuming the service API would just “guess” at what the rel=”” value should be to create a new user. I could easily get it wrong and try to create a new user on a resource other than the users without me, the developer, knowing to parse for the rel=”users” value to get the right resource to send a POST to.

    So please, enlighten me (us.. there has to be others that have this issue) on how without any service API documentation, a consuming developer could possibly know all the resources an what they do and then to integrate them within their existing (or not) application correctly?

    Thank you.

    • baeldung

      This is an interesting point you raise – and I don’t think there’s any perfect solution to this problem. I have however implemented a few approaches and have seen others work as well.
      In your case, first – a few notes:
      - when you create a user, is should be a POST on /users or similar – not sure that /username is the best URI for this
      - supporting OPTIONS requests should let you know that you can do POST on the /users URI – to figure out that it supports creation of a new user
      One option is to go beyond the standard rel types and do something like: rel=”create” and rel=”edit” to signal to the user that this is a create URI.
      So a developer, consuming the service, coudl do an OPTIONS on /users, see that it supports POST, and then send a POST with a new user and get back a 201 Created with the Location header set to the new URI of the user.

      Hope this helps.

      Thanks.

  • Pedro

    Thanks for this very interesting post. I think i’m having the same problem as Kevin: I can’t find a proper, manageable, elegant way to let developers discover the links and learn about link relations and available methods without having them wander through the application in a trial and error basis… that’s calling for disaster.
    Any help or further info in this subject would be greatly appreciated.
    Keep up with the great work!

  • http://cdivilly.wordpress.com cdivilly

    I’d suggest using the Location header to communicate the URI of the newly created resource instead of the Link header, see RFC 2616 #14.30: “For 201 (Created) responses, the Location is that of the new resource which was created by the request.”

    • Eugen

      Only using Link (and not Location) to address the HATEOAS constraint was my not so inspired experiment in straying from the norm – I have updated the article and github project to make use of the standard Location header for creation.

  • Kevin

    @cdivililly.. what you say I do. I am not referring to returning Link elements in response to a POST..

    What I am referring to and what I believe Pedro is saying is as a developer consuming say Facebook or Twitter API.. I couldn’t blindly make a REST call to their entry and write code that can simply parse all the response links and know that the link with some string in it is to get a list of contacts. If I, as a human, looked at the json or xml returned and saw it say I could make a guess that this is the resource I would make a GET request to to get the list of contacts for the authenticated user. However, in order to write a program that does this on it’s own.. it would need natural language parsing capabilities and be human-like to know that “contact” the string means to get a list of contacts, or add new ones.

    Mine and Pedro’s issue is.. how do I write a client that consumes a rest service without any documentation and understand how to use it? I have an existing site.. it creates users locally. I want to integrate a rest API that say, centralizes user management. So without reading some sort of documentation as the developer writing the code to use the service, how do I know which link resource to use to create the user? If the service returns with 10 different links, one of which is “users”, and others that would be “contacts”, “banking”, “global thermo nuclear war” and more.. how do I know short of me, as a human reading the doc and seeing “users” as an option and getting info on what the “users” resource allows me to do.. how do I possibly know to look for that link, and use it? It’s impossible to do. Hence why APIs like twitter, facebook and the likes ALL provide developer documentation on the resources they provide.

    To me, as both a developer writing code to consume a REST API AND the developer who is also writing a REST api that is trying to stay true to HATEOAS, both sides are difficult. My current API has an entry URL, which returns a response full of links that the authenticated user can then use. From there, the consumer can scan the links returned and make use of whatever. Now, on the client side, I have an existing site that already works.. but I want to now integrate with this service too. As I gave an example above, creating users on the service to tie my client side users with the service users.. so that when a user on my web app logs in, their user is associated with a service user too. Now, my web app, which has an authenticated API token.. makes calls to the service on behalf of a logged in web app user, passing in the service user id that ties them together. As the developer of the client web app, I know that I make a REST GET call to the entry URL, it returns a list of links, and I know to then look for the rel=”users” to make REST calls to the users resource. BUT, without documentation on HOW to use that resource, I would not know how to use it. What info does that resource need to tie a user on my web app system with the user on the service side? An id? A name?

    Part of HATEOAS is that the links themselves may change.. for example, perhaps the link initially is:

    but later the service api changes.. they decide to put it a level down..

    Well, as the consumer I do NOT care about the href.. I just get rel=”users” and whatever href link comes back, I use. But, I STILL need to know to search for the rel=”users” in order to get the correct resource link to use.

    So all this comes back to what this document talks about.. discovery of rest resources. Without documentation that explains to a developer what each resource my service provides does..and how to use them (what headers, query params, body, etc to provide and what comes back in response), I don’t see how any developer writing rest clients could programatically know that the rel=”users” is the right link to use to create users. The service could have a rel=”usethistodouserstuff” and without documentation telling me the string rel to look for to do user stuff with is called that.. I would not know. I might make educated guesses.. I know the service provides some sort of centralized user managements, so I am guessing there would be a rel=”users” or rel=”user” to do stuff.. but there is no guarantee.

    So to finish off this response.. to me, self discovery of rest resources is only workable for web bots. It makes no sense for a developer looking to consume a weather api resource to guess at what the resources are and figure out what they do. How do I know what query params to pass a rest API to get the weather for New York.. is it ?city=New York, ?c=newyork.. what is it? There is just know way to do this without documentation to explain each resource, each method supported, any media-type needed, and what responses come back including list of relevant links.

    Ok.. so all that said, I think it’s completely possible to remain HATEOAS bound to a degree. If I call the entry URL and get back a list of links.. and the one I want to use is “users”. I then look up the docs on “users” resource. It says, send a GET to /users/ and you’ll get back this json/xml info about the user at that id if it exists. You’ll also get back several elements.. one to PUT to to update the user (if your authenticated api token allows for you to change the user), one to delete it, and one to get a list of contacts associated with that user.. and so forth. As long as I, the developer use the documentation of the REST API and use all the rel=”” to move through the API using all the href=”” values given to me, I should be HATEOAS safe as a consumer. The only thing I “add” is query params.. I don’t store the HREF values and build up URLs to resources..I use what is returned in responses. I can read the http headers in responses to see if they are cacheable/bookmarkable and so forth.

    That is a bit long..but I am not sure how to say what I did above in a couple sentences. Hope this clears things up with what I and I think Pedro are trying to understand and do correctly.

    Thank you.

    • Eugen

      OK, this discussion warrants a new article, but in the meantime I will try to provide as much clarity to the subject as I can, based on my own previous experiences.
      First, let’s clear up a few premises: I am not discussing any particular API, such as Twitter or Facebook – how much these conform to the RESTful constraints is clearly not in the scope of the article. I am however referring to the principles and practices to use when building (and consuming) your own RESTful API.
      Moving forward, the first goal of discoverability is to make minimal or no use of documentation and have the client learn and understand how to use the API via the responses it gets. In fact, this shouldn’t be regarded as such a far fetched ideal – it is how you consume every new web page – without any documentation. So, if the concept is more problematic in the context of REST, then it must be a matter of technical implementation, not of a question of whether or not it’s possible. I do agree that, technically, we are still far from the perfect solution – that the specs and framework support is still evolving, and that, because of that, we may need to compromise, but that is nevertheless the goal.
      Now, in relation to the specifics of some of the questions – you mention using a rel=”users” and the client having to guess that a POST on the URI would create a user; there are several ways around that. First, you can create new Link relation (see http://tools.ietf.org/html/rfc5988) with very clear semantic meaning, such as, in your case, “create-user” or even “create-user-POST”. You could also use a default Link relation – the “help” relation – to give a description of the URI, to clear up any confusion the client may have.
      Next, you could encourage the client to do GETs on the URI (which should be idempotent and completely safe), and then use the “Allow” header to list out the allowed operations (POST in your case). What is more, you should never trust user input, so the client shouldn’t be able to “get it wrong” as you say – and try to create a user with POST on another URI, or create another entity with POST on the users URI, because the JSON payload wouldn’t match, in which case no user will be created (and 400 or 412 back to the client).
      There are several other techniques (also more semantically rich microformats that go beyond just XML or JSON), which I will try to cover in an upcoming article.
      Now, related to the consumption of the API by a droid – that’s another problem altogether. First, I would not expect that is an actual goal – you wouldn’t expect an unsophisticated droid to be able to navigate intelligently through a web page the same as you wouldn’t expect it to navigate a RESTful API. If you need that, you would hardcode the path (more or less) meaning that you would first do it yourself, and then the droid would just follow predetermined operations. If you need more, that means more standardized semantics – a host of proposals for richer semantics are done in HTML 5 – perhaps you should get involved in that effort.
      Also if your API returns to many links without clear meanings, like your example says: “users” and “contacts”, then besides my suggestion to use richer semantics, there may be a problem of clarity in the design of the API itself.
      I’m sure that there are points that I may have not addressed fully, but that is the gist of it until a proper article on the subject.
      In conclusion, yes, documentation may need to be a necessary compromise, but that doesn’t mean that you should embrace it, but rather that you should recognize that it is indeed a compromise and always look for the better technical solution to get rid of it.
      Thanks for the good feedback and the discussion.

  • Pingback: This Week in #REST – Volume 40 (Sept 12 2011 – Nov 16 2011) « This week in REST

  • pablo iturralde

    hi,
    I followed your tutorial which is graeat by the way!! and I connect an iphone app to it but now I want to use google analitycs to record the the visited urls.. is this posible

  • Stephane

    In 3.3 what is the difference between a Foo instance and a Foo resource ?

    • baeldung

      No difference – the language is just somewhat ambiguous – I’ll clarify the wording there – thanks.

  • vvarma

    nice article. you should also consider adding a bit about Interceptors.. found them to be a great tool for exception handling