I recently attended SXSW 2015 and heard a great panel discussion titled “Secrets To Powerful APIs” with Erik Michaels-Ober (Soundcloud), Jim Heising (IFTTT), Wynn Netherland (GitHub), and Leah Culver (DropBox). Among many other interesting things the panelists said was their uniform agreement that they don’t use HATEOAS (Hypertext As The Engine Of Application State) API designs, and that they don’t see much adoption of this approach generally across the industry. My own take:
- HATEOAS is fine for APIs that primarily manipulate content. So if you need to store and process a wide range of content types for video, audio, documents, and pictures, and the operations you will perform are primarily CRUD, then HATEOAS is a good approach to build dynamic behavior around the content, that adapts to its many different forms, for both providers and consumers of the API.
- Contract-first development still makes sense for APIs. Most enterprise use-cases for APIs, including the external (partner community) case, benefit from providers and consumers establishing prior agreement that governs the behavior and data for API interchanges. The panelists at SXSW agreed – they said things like “I miss certain things about WSDL, such as having a Type that limits the Maxoccurs.” They also missed other things from the XML world such as XSLT and XPath, but all agreed JSON is dominant today, and a better solution overall for most things except binary content. The panelists were also enthusiastic about JSONs (JSON Schema) and use it a lot.
- The trade-off between flexibility and complexity is universal. HATEOAS is more flexible, but the reason people avoid it unless they need it is its added complexity. It’s harder to write an app that can deal with the API behavior varying depending on the content and its object model.
But what about the more general question of how best to deal with situations where the application context demands that an API handle more dynamic behavior? This need can arise in a wide range of situations, including:
- Information-centric APIs that return many different views. Each view requires a different set of information to be returned, from the same source.
- Workflow-oriented APIs that manipulate “manifest” documents. More flexible adaptive case management approaches benefit from APIs like this.
- Edge-oriented APIs that must deal with a wide range of edge conditions. Netflix is the best-known example, with a multitude of “consumption” APIs that match the needs of individual media players.
Despite the trend in the API community toward consumption APIs, this is neither the only approach to this situation, nor necessarily the best, depending on requirements. It’s a great solution for Netflix, and if you have similar requirements, then you may also need to take that approach. But what if you need dynamic behavior, but the differences are more typical of information-centric APIs?
Enter The Query Type
Query types are not a new idea – I’ve seen firms (including digitalML customers) implement this approach across many different generations of technology, and there are many variations on the theme. But the basic idea is that the API:
- Includes a query-type element in the request and response messages.
- Defines message payloads with most elements optional.
- Defines an implicit contract linking each allowed value for query-type with a specific information view – governing which elements are returned.
This is a great pattern for MDM APIs, where all consumers want canonical gold-master data, but don’t want all of it, all the time. Each usage scenario that requires a different view simply defines a different query-type and associated set of expectations about which elements will be returned. You can even wrap these different scenarios with purpose-specific client-side objects (via an SDK), which can help limit the risk exposure of a developer writing code that does the wrong thing at runtime, obtaining incorrect results.
So in this case we’re managing the flexibility/complexity trade-off by having one more-complex API that has to deal with managing all the different query types, versus having more APIs, each one dedicated to handling one query type. Deciding which is best depends on how many query types you have, how often they change, whether you have a team willing and able to own the API and keep it working for all the use-cases, and how different the various use cases are from one another.
This last point is particularly important: if all the consumers want the same information, that is, they are ready and able to consume canonical formats, then this tends to argue in favor of a single API, especially if there’s no need for other forms of variation such as in the operations provided.
When Not To Use Query Types
If your API consumers each need different formats for the same data, then the complexity of a single API is probably too great, and likely outweighs the advantage of maintaining all the master-data behavior for a resource in one place. In this case you’ll probably be better off with two layers of APIs – a core set of canonical resource APIs that return big payloads of canonical data, consumed by a population of more edge-specific (“consumption”) APIs. Even though a larger population of APIs is more complex, in some ways, than a single API for a canonical resource, it’s more agile, and may be a better fit to organizations that value team autonomy and time-to-market more than consistency of results across the different APIs.
Do’s And Don’ts Of Using Query Types
- Do: minimize the number of query-types where possible. If two consumers need almost the same information, handle both with one query-type.
- Don’t: overload an API with multiple resource types. Unless the types are trivially different from one another, it’s best to give each its own API (like Customer, in our example above).
- Do: use query-type APIs to centralize business rules and behavior. There should be one way to calculate profit, revenue, etc., and the business should own the definition.
- Don’t: push too much complexity onto the client. Ensuring that this approach succeeds requires widespread adoption, so do what you reasonably can to centralize complexity in the API and make it easy and quick to consume.
- Do: use data-driven implementations to minimize internal complexity. One interesting example I saw recently was presented by a large insurance firm at Enterprise Data World: their Reference Data system is driven by an underlying data store that defines how reference data differs between client systems, and how instances can be mapped from one to another. Rules engines can provide a useful variation on this theme.
- Don’t: forget about needing to maintain this behavior over time. Some coding approaches to this problem that work fine at first tend to break down over time as a maintenance nightmare of inconsistent behavior.
Regarding #6, one version of that problem is using an overly code-centric means of managing mappings and transformations between layers. An API that delivers canonical master data typically front-ends a wide range of complex and disparate back-end systems. As much as possible, use metadata to define those sources and targets and the mappings between them, and either generate the code from that, or write code that operates off the metadata at runtime. XSLT offered a nice way to handle this for XML, but there’s no equivalent for JSON, so other tools have to fill that gap.
Attend The digitalML Service Design Forum, 11-12 May, Chicago
I’m happy to report that our Chief Architect Andy Medlicott has new capabilities to show at the Forum including powerful new code generation usable via the ignite Service Design Platform for speeding implementation of a wide range of design patterns, including this one. If you’re not already registered, I recommend you sign up to attend now!