Best practices for REST API design Edit

Murugan Andezuthu Dharmaratnam | 03 May 2021 | 416

Representational State Transfer (REST) is an architectural style of building distributed systems. It was proposed by Roy Fielding in the year 2000. REST APIs can be written using any language and they can be consumed by any tool or application written in any language. In this article, we will look at some of the best practices to design a REST API so that It is fast, secure, easy to understand, scalable, simple, portable, reliable & future proof.

Constraints

For any application to be considered RESTful the following 6 constraints must be present. If the application will not be considered RESTful if it violates any of the following constains. The idea of having these constains is to restict the ways server can process and respond to request so that system gains properties such as performance, scalability, simplicity, modifiability, visibility, portability, and reliability.

    1. Client–server architecture: Principle behind client server architecture is the separation of concerns. In client server architecture the server takes care of application logic and displaying of data and the client takes care of getting and displaying the data. The main davantage of this approach is both client and server can be developed independently
    2. Statelessness: No session information is retaned by the server. The client will always send the relavant session data in such a way that every client request has received by the server can be understood in isolation. This improves scalability as the request can be processed by any server
    3. Cacheability: Caching can greatly improve scalabity & performance. Any response from the server must, implicitly or explicitly, define themselves as either cacheable or non-cacheable to prevent clients from processing old or inappropriate data in response to further requests
    4. Layered system: There should not be needed to update the client or server code even if the client is connected to the server using a load balancer. Intermediary servers can improve system scalability by enabling load balancing and by providing shared caches. Also, security can be added as a separate layer on top of the web API.
    5. Uniform interface: REST defines a set of well-defined operations that can be executed on a resource. It simplifies and decouples the architecture, which enables each part to evolve independently/
    6. Code on demand (optional)

Resource & Collections

The fundamental concept in any RESTful API is the resource. REST clients can access and modify the resources provided by the server. Resources are similar to objects in object-oriented programming, with the main difference that only a few standard methods HTTP GET, POST, PUT and DELETE are defined for the resource. Resources can be grouped into collection. Collections are themselves resources as well. Resources can also exist outside any collection. Collections can exist globally, at the top level of an API, but can also be contained inside a single resource. In the latter case, we refer to these collections as sub-collections.

Once the resources are identified, we have to decide on a standard format like JSON for its representation. The server and client would use this format to send and receive data from and to the server. While designing the representation format of a resource, understandability, completeness, and linkability are the important points to be considered.

What makes good API's

Key things are

    • Easy to learn
    • Easy to use
    • Hard to misuse
    • Easy to read and maintain code that uses it
    • Sufficiently powerful to satisfy requirements
    • Easy to Extend
    • Appropriate to the audience

REST API Best Practices

Here are some of the practices to implement in REST API development.

Data

Use JSON: REST allows using different output formats, like plain text, JSON, CSV, XML etc, but the best option is to use JSON, 99% of the REST applications use json for passing data to and from the server.

URI Formatting

Use Nouns: The best way to format your URI/endpoint is to use nouns instead of verbs. This is because our HTTP request method already has the verb. Treat the resource like a noun, and the HTTP method as a verb.

Insted of using 
/getallcountries
/getcountry/91
/deletecountry/91
/updatecoountry

use

GET /countries		- Get all countries
GET /countries/91	- Get country with Id 91
POST /countries		- Add a new country and return the details
DELETE /countries/91	- Delete country with Id 91
GET /countries/91/states	- Get all the states of the country with country Id 91

Always name using plural nouns eg countries instead of the country even for the method where you get one country by Id. Use resource nesting to show relation or hierarchy Eg /countries/91/states to Get all the states of the country with country Id 91. But keep in mind best practice would be to Avoid requiring resource URIs more complex than collection/item/collection.

How to handle actions which do not fit into CRUD operations?

Define operations in terms of HTTP methods

Common HTTP methods used by most RESTful web APIs ar

    • GET: retrieves a representation of the resource at the specified URI.
    • POST: creates a new resource at the specified URI.
    • PUT: either creates or replaces the resource at the specified URI.
    • PATCH: performs a partial update of a resource
    • DELETE: deletes the resource at the specified URI

    The difference between POST PUT AND PATCH is that POST is used to create a resource, PUT is used to create or update a resource. If a resource with this URI already exists, it is replaced. Otherwise, a new resource is created. PATCH is to perform a partial update to an existing resource. PUT requests must be idempotent. If a client submits the same PUT request multiple times, the results should always be the same (the same resource will be modified with the same values). POST and PATCH requests are not guaranteed to be idempotent.

Conform to HTTP semantics

This section describes some typical considerations for designing an API that conforms to the HTTP specification.

Support content negotiation where ever applicable. In the HTTP protocol, formats are specified through the use of media types, also called MIME types. Most web APIs support JSON & XML. You can specify the media type in the header. If the server doesn't support the media type, it should return HTTP status code 415 (Unsupported Media Type).


media type = application/json for Json
media type = application/xml

Example of post request that include JSON data. 

Content-Type: application/json; charset=utf-8

A client request can include an Accept header that contains a list of media types the client will accept from the server in the response message. For example, if you are requesting an image from the server, you can specify the media type the client can handle such as image/jpeg, image/gif, image/png. The server should format the data by using one of these media types and specify the format in the Content-Type header of the response.

Eg.

Accept: application/json

If the server cannot match any of the media type(s) listed, it should return HTTP status code 406 (Not Acceptable).

GET, PUT, DELETE, HEAD, and PATCH actions should be idempotent which means even if you make repeated requests to the same resource the state of the resource should be the same. If you send multiple delete request to the same URI should have the same effect. The HTTP Status code may change for example for the first delete request where the delete had happened you can return status code 204 while the subsequent request might return status code 404

A successful GET method typically returns HTTP status code 200 (OK). If the resource cannot be found, the method should return 404 (Not Found).

If a POST method creates a new resource, it returns HTTP status code 201 (Created). The URI of the new resource is included in the Location header of the response. The response body contains a representation of the resource.If the method does some processing but does not create a new resource, the method can return HTTP status code 200 and include the result of the operation in the response body. Alternatively, if there is no result to return, the method can return HTTP status code 204 (No Content) with no response body.If the client puts invalid data into the request, the server should return HTTP status code 400 (Bad Request). The response body can contain additional information about the error or a link to a URI that provides more details. You should Avoid implementing chatty POST, PUT, and DELETE operations.

If a PUT method creates a new resource, it returns HTTP status code 201 (Created), as with a POST method. If the method updates an existing resource, it returns either 200 (OK) or 204 (No Content). In some cases, it might not be possible to update an existing resource. In that case, consider returning HTTP status code 409 (Conflict).

With a PATCH request, the client sends a set of updates to an existing resource, in the form of a patch document. The server processes the patch document to perform the update. The patch document doesn't describe the whole resource, only a set of changes to apply. JSON merge patch is somewhat simpler. The patch document has the same structure as the original JSON resource, but includes just the subset of fields that should be changed or added. In addition, a field can be deleted by specifying null for the field value in the patch document. (That means merge patch is not suitable if the original resource can have explicit null values.)

If the DELETE operation is successful, the web server should respond with HTTP status code 204, indicating that the process has been successfully handled, but that the response body contains no further information. If the resource doesn't exist, the web server can return HTTP 404 (Not Found).

Asynchronous operations Sometimes a POST, PUT, PATCH, or DELETE operation might require processing that takes a while to complete. If you wait for completion before sending a response to the client, it may cause unacceptable latency. If so, consider making the operation asynchronous. Return HTTP status code 202 (Accepted) to indicate the request was accepted for processing but is not completed. You should expose an endpoint that returns the status of an asynchronous request, so the client can monitor the status by polling the status endpoint. Include the URI of the status endpoint in the Location header of the 202 response. If the client sends a GET request to this endpoint, the response should contain the current status of the request. Optionally, it could also include an estimated time to completion or a link to cancel the operation. If the asynchronous operation creates a new resource, the status endpoint should return status code 303 (See Other) after the operation completes. In the 303 response, include a Location header that gives the URI of the new resource:

You can implement a simple polling mechanism by providing a polling URI that acts as a virtual resource using the following approach:

  1. The client application sends the initial request to the web API.
  2. The web API stores information about the request in a table held in table storage or Microsoft Azure Cache, and generates a unique key for this entry, possibly in the form of a GUID.
  3. The web API initiates the processing as a separate task. The web API records the state of the task in the table as Running.
  4. The web API returns a response message with HTTP status code 202 (Accepted), and the GUID of the table entry in the body of the message.
  5. When the task has completed, the web API stores the results in the table, and sets the state of the task to Complete. Note that if the task fails, the web API could also store information about the failure and set the status to Failed.
  6. While the task is running, the client can continue performing its own processing. It can periodically send a request to the URI /polling/{guid} where {guid} is the GUID returned in the 202 response message by the web API.
  7. The web API at the /polling/{guid} URI queries the state of the corresponding task in the table and returns a response message with HTTP status code 200 (OK) containing this state (Running, Complete, or Failed). If the task has completed or failed, the response message can also include the results of the processing or any information available about the reason for the failure.

Options for implementing notifications include:

  1. Using a notification hub to push asynchronous responses to client applications.
  2. Using the Comet model to retain a persistent network connection between the client and the server hosting the web API, and using this connection to push messages from the server back to the client.
  3. Using SignalR to push data in real time from the web server to the client over a persistent network connection.

Provide links to support HATEOAS-style navigation and discovery of resources. The HATEOAS approach enables a client to navigate and discover resources from an initial starting point. This is achieved by using links containing URIs; when a client issues an HTTP GET request to obtain a resource, the response should contain URIs that enable a client application to quickly locate any directly related resources. For example, in a web API that supports an e-commerce solution, a customer may have placed many orders. When a client application retrieves the details for a customer, the response should include links that enable the client application to send HTTP GET requests that can retrieve these orders. Additionally, HATEOAS-style links should describe the other operations (POST, PUT, DELETE, and so on) that each linked resource supports together with the corresponding URI to perform each request.

Example Get request

GET https://yourdomain.com/customers/2 

{"CustomerID":2,"CustomerName":"Bert","Links":[
    {"rel":"self",
    "href":"https://yourdomain.com/customers/2",
    "action":"GET",
    "types":["text/xml","application/json"]},
    {"rel":"self",
    "href":"https://yourdomain.com/customers/2",
    "action":"PUT",
    "types":["application/x-www-form-urlencoded"]},
    {"rel":"self",
    "href":"https://yourdomain.com/customers/2",
    "action":"DELETE",
    "types":[]},
    {"rel":"orders",
    "href":"https://yourdomain.com/customers/2/orders",
    "action":"GET",
    "types":["text/xml","application/json"]},
    {"rel":"orders",
    "href":"https://yourdomain.com/customers/2/orders",
    "action":"POST",
    "types":["application/x-www-form-urlencoded"]}
]}


The return link will have the following structure

public class Link
{
    public string Rel { get; set; }
    public string Href { get; set; }
    public string Action { get; set; }
    public string [] Types { get; set; }
}

The HTTP GET operation retrieves the customer data from storage and constructs a Customer object, and then populates the Links collection. The result is formatted as a JSON response message. Each link comprises the following fields:

The relationship between the object being returned and the object described by the link. In this case self indicates that the link is a reference back to the object itself (similar to a this pointer in many object-oriented languages), and orders is the name of a collection containing the related order information. The hyperlink (Href) for the object being described by the link in the form of a URI. The type of HTTP request (Action) that can be sent to this URI. The format of any data (Types) that should be provided in the HTTP request or that can be returned in the response, depending on the type of the request.

When you are dealing with large objects or include large fields, such as graphics images or other types of binary data A web API should support streaming to enable optimized uploading and downloading of these resources. The HTTP protocol provides the chunked transfer encoding mechanism to stream large data objects back to a client. A single request could conceivably result in a massive object that consumes considerable resources. If during the streaming process the web API determines that the amount of data in a request has exceeded some acceptable bounds, it can abort the operation and return a response message with status code 413 (Request Entity Too Large).

A client application can explicitly request data for large objects in chunks, known as partial responses. The client application sends an HTTP HEAD request to obtain information about the object. If the web API supports partial responses if should respond to the HEAD request with a response message that contains an Accept-Ranges header and a Content-Length header that indicates the total size of the object, but the body of the message should be empty. The client application can use this information to construct a series of GET requests that specify a range of bytes to receive. The web API should return a response message with HTTP status 206 (Partial Content), a Content-Length header that specifies the actual amount of data included in the body of the response message, and a Content-Range header that indicates which part (such as bytes 4000 to 8000) of the object this data represents.

A client application that is about to send a large amount of data to a server may determine first whether the server is actually willing to accept the request. Prior to sending the data, the client application can submit an HTTP request with an Expect: 100-Continue header, a Content-Length header that indicates the size of the data, but an empty message body. If the server is willing to handle the request, it should respond with a message that specifies the HTTP status 100 (Continue). The client application can then proceed and send the complete request including the data in the message body.

Error Handling

When you make a request to web API, sometimes an error occurs and you have to handle errors gracefully and return standard error codes. There are over 71 distinct HTTP Status codes, they are well defined and easily recognizable the best practice would be to use them and not to reinvent the wheel. A perfect error message would consist of an HTTP Status code, Code Id (which may be an internal reference, you may also provide a link to the API documentation containing all the code id's), message & might be more info.

The HTTP protocol distinguishes between errors that occur due to the client application (the HTTP 4xx status codes), and errors that are caused by a mishap on the server (the HTTP 5xx status codes). Make sure that you respect this convention in any error response messages.

To handle exceptions in a consistent manner, consider implementing a global error handling strategy across the entire web API. You should also incorporate error logging which captures the full details of each exception; this error log can contain detailed information as long as it is not made accessible over the web to clients.

Versioning

It's important to version the API right from the start this would allow you to manage version upgrade gracefully. At some point, you might have to manage more then one version of the application, But this will allow you to modify and improve the application. With REST API there are two approaches to implement the version. using the request header or in the URI. Using the URI is the most commonly used option. You can also combine both methods.

Example of URI endpoint versioning

https://yourdomain.com/3.0/ (major   minor version indication)
https://yourdomain.comv1/ (major version indication only)
https://yourdomain.com/v3/  (major version indication only)
https://yourdomain.com/2010-04-01/ (date based indication)

Status Code

You should always return the status code consistently, for example, if you use post request to create a resource you should return 201 Created status code. Be consistent, and if you stray away from conventions, document it somewhere with big signs.

You should use 202 Accepted where every applicable. 202 accepted means the server has understood your request, but not acted upon the modification requested to the request. Two cases where 202 would specifically usable would be. If the resource will be created as a result of future processing or If the resource already existed in some way, but this should not be interpreted as an error.

Filtering, Sorting, Paging

Filtering Sorting and pagination are used when there is a large number of rows of data from a get request. Filtering lets you narrow down the query result by specifying a parameter Eg. for a City we can provide filters like Country, State. Sorting allows you to sort a column ascending or descending by a chosen Eg sort by Country. Paging when you have a large number of rows in the result you can use paging to break down the full list of data into pages.

Make sure to use query string for filtering, sorting, and pagination.

GET /city?country=USA&creation_date=2019-11-11&sort=Name:desc&limit=100&offset=2

Example

GET: /countries/?page=1&page_size=10

Security

RESTful API's are stateless and each request should come with some sort of authentication credentials.

One thing to remember is the difference between 401 unauthorized and 403 Forbidden. When dealing with security you have to return the correct error code with respect to authentication or authorization. return 401 unauthorized if the authentication failed. 403 forbidden when you are authenticated but you don't have the required permission to access the resource.

Caching

The good thing about caching is that the user can get data faster. Application-level caching is a must to improve performance. We can add caching to get data from the memory instead of querying the database. There are several options for caching like in-memory caching or Redis.

The HTTP 1.1 protocol supports caching in clients and intermediate servers through which a request is routed by the use of the Cache-Control header. When a client application sends an HTTP GET request to the web API, the response can include a Cache-Control header that indicates whether the data in the body of the response can be safely cached by the client or an intermediate server through which the request has been routed, and for how long before it should expire and be considered out-of-date.Cache management is the responsibility of the client application or intermediate server, but if properly implemented it can save bandwidth and improve performance by removing the need to fetch data that has already been recently retrieved

Example of HTTP response 

Cache-Control: max-age=600, private
Content-Type: text/json; charset=utf-8
Content-Length: ...
{"orderID":2,"productID":4,"quantity":2,"orderValue":10.00}

In this example, the Cache-Control header specifies that the data returned should be expired after 600 seconds, and is only suitable for a single client and must not be stored in a shared cache used by other clients (it is private). The Cache-Control header could specify public rather than private in which case the data can be stored in a shared cache, or it could specify no-store in which case the data must not be cached by the client. The max-age value in the Cache-Control header is only a guide and not a guarantee that the corresponding data won't change during the specified time.

HTTP provides a built-in caching framework! All you have to do is include some additional outbound response headers and do a little validation when you receive some inbound request headers.

There are 2 approaches: ETag and Last-Modified

Provide ETags to optimize query processing. When a client application retrieves an object, the response message can also include an ETag (Entity Tag). An ETag is an opaque string that indicates the version of a resource; each time a resource changes the ETag is also modified. When generating a response, include a HTTP header ETag containing a hash or checksum of the representation. This value should change whenever the output representation changes. Now, if an inbound HTTP requests contains a If-None-Match header with a matching ETag value, the API should return a 304 Not Modified status code instead of the output representation of the resource.

The client makes the first request & in the subsequent, The client constructs a GET request containing the ETag for the currently cached version of the resource referenced in an If-None-Match HTTP header:

GET https://yourdomain.com/orders/2 HTTP/1.1
If-None-Match: "2147483648"

The GET operation in the web API obtains the current ETag for the requested data and compares it to the value in the If-None-Match header.If the current ETag for the requested data matches the ETag provided by the request, the resource has not changed and the web API should return an HTTP response with an empty message body and a status code of 304 (Not Modified). If the current ETag for the requested data does not match the ETag provided by the request, then the data has changed and the web API should return an HTTP response with the new data in the message body and a status code of 200 (OK). If the requested data no longer exists then the web API should return an HTTP response with the status code of 404 (Not Found). The client uses the status code to maintain the cache. If the data has not changed (status code 304) then the object can remain cached and the client application should continue to use this version of the object. If the data has changed (status code 200) then the cached object should be discarded and the new one inserted. If the data is no longer available (status code 404) then the object should be removed from the cache.

Etag can also be used by PUT & DELETE methods. If, after fetching and caching a resource, the client application subsequently sends a PUT or DELETE request to change or remove the resource, it should include in If-Match header that references the ETag. The web API can then use this information to determine whether the resource has already been changed by another user since it was retrieved and send an appropriate response back to the client application.The PUT operation in the web API obtains the current ETag for the requested data and compares it to the value in the If-Match header. If the current ETag for the requested data matches the ETag provided by the request, the resource has not changed and the web API should perform the update, returning a message with HTTP status code 204 (No Content) if it is successful. The response can include Cache-Control and ETag headers for the updated version of the resource. The response should always include the Location header that references the URI of the newly updated resource. If the current ETag for the requested data does not match the ETag provided by the request, then the data has been changed by another user since it was fetched and the web API should return an HTTP response with an empty message body and a status code of 412 (Precondition Failed). If the resource to be updated no longer exists then the web API should return an HTTP response with the status code of 404 (Not Found). The client uses the status code and response headers to maintain the cache. If the data has been updated (status code 204) then the object can remain cached (as long as the Cache-Control header does not specify no-store) but the ETag should be updated. If the data was changed by another user changed (status code 412) or not found (status code 404) then the cached object should be discarded.

Last-Modified: This basically works like to ETag, except that it uses timestamps. The response header Last-Modified contains a timestamp in RFC 1123 format which is validated against If-Modified-Since. Note that the HTTP spec has had 3 different acceptable date formats and the server should be prepared to accept any one of them.

For security reasons, do not allow sensitive data or data returned over an authenticated (HTTPS) connection to be cached.

Using SSL/TLS

Most communication between client and server should be private since we often send and receive private information. Therefore, using SSL/TLS for security is a must.

Documentation

It's important to document the rest API. It would be recommended to create good API documentation like how they have done for twillio.

https://www.twilio.com/docs/usage/api

API documentation should provide information about the available endpoints and methods, preferably with request/response examples, possible response codes, information about the authorization, available limits, or throttling.