In my opening session at digitalML’s fifth annual customer Forum, the Service Design Forum, I presented this slide to guide a discussion of the way Data Governance has evolved in this era of OmniChannel experiences and Big Data:
The blue layers represent all your data, whether in motion or at rest, and I’ve called out the flow of data between the layers as data in motion – but in reality the upper two layers, “Interaction data & events” and “Product, sales, and service data” are also largely composed of data in motion, conveyed by APIs, services, messaging, etc. It’s the bottom layer that manages data at rest in core systems (Systems of Record), and in stacks for BI and analytics (most of which also expose APIs).
This is a very simplified view, highly abstracted from real systems architecture, but it helps me make the point that all these layers must work together to deliver OmniChannel experiences for customers, and desired outcomes for your business. Yet historically many firms have tended to treat data governance as a discipline mainly focused on data at rest – or if, like many digitalML customers, you have also aimed to govern data in motion, that effort has been largely disconnected from your governance of data at rest.
That separation was more justifiable in the past than it will be in the future – and that evolution, to a more comprehensive approach to data governance – is the topic of this blog post.
Models Of Data In Motion And At Rest Are Very Different
The way you model data varies depending on the context: in-motion, at-rest, and in-program (language objects):
Each context brings its own concerns, with variations within each context such as mobile. In working with customers and prospects, digitalML sees the same issues coming up again and again from firms using the wrong approach or the wrong tools, such as using a data modeling tool optimized for data at rest to generate a schema for data in motion. Such schemas tend to be too deep and abstract, including concepts such as “party” that are obscure and not helpful from the point of view of a mobile developer who just wants a simple flat structure for their API.
One of the new features that Andy Medlicott previewed at our recent Service Design Forum was “flatteners” that enable modelers to more easily maintain different kinds of model structures in ignite, all tied back to the same canonical elements, but optimized for various distinct use cases. Today the most common variation is between XSD and JSON, but now that we’re working on adding support for Avro for Big Data, it’s clear to me that there’s a broader pattern – enterprises are facing a challenge and opportunity to better manage the way they govern data across these various environments. It’s an opportunity to extend the leverage of your canonical model by making it more adaptable to a wider range of usage contexts.
Avro Is “Just Another Format”
Once you’re wrapped your head around the idea that your canonical model can and should be made relevant to an increasing number of usage contexts, you encounter the opportunity to generate an increasing range of formats relevant in each context. Andy described the work he did to add support for Avro by saying “it’s just another format,” meaning that once you have the architecture we have established inside ignite, the effort for us to add generation support for more formats is not large.
Beyond Avro could lie support for other formats /approaches like Protocol Buffers, Thrift, or others that gain significant support and customer demand. But the point of my argument is not to suggest that any particular list of supported formats is what matters – rather, what matters is:
- Having a canonical model flexible enough to handle all needed contexts. This requires tooling built around a rich and flexible underlying metamodel that supports all the various “degrees of freedom” required to support the widely varying requirements across all those contexts.
- Being able to quickly generate schemas in many different formats as needed in each of these contexts. Today it takes someone on the digitalML engineering team to build these generators. As quickly as we can do that, we will ultimately offer ways that customers and third parties can build them, speeding up the rate at which support for new formats can be added.
Once you have both these things (as we do in ignite), it means that for the first time it begins to be possible to take a more comprehensive approach to governing the model of your data, helping to ensure that when a business person consumes information from a BI dashboard fed from a Big Data analytical model, they see information that’s consistent with what customers see through a mobile app consuming an API that’s mapped to the same information. That’s huge, and nobody has the ability today to provide that in one data governance solution.
It Takes A Village
digitalML is not a big company, so there’s a limit to how much of this problem we can solve. But in today’s world, that a feature, not a problem – we recently shipped our API Developer Toolkit, which is about making use of the APIs that we expose from ignite. As Andy says, “if you can see it in ignite, there’s an API for it” – and that’s not only for read (Get), it’s a fully capable API with update capabilities.
This means that it’s possible for you to construct a more comprehensive solution for data governance by combining ignite with your other tools. Not every tool is as far along as ignite in exposing APIs, but for those in which vendors are still investing, APIs should be and often are a key part of the landscape going forward. digitalML has already helped customers use our APIs to connect ignite into their broader service governance environment, so we know it’s quite feasible to do this.
So it’s time for you to begin to take control of your data governance approach across all kinds of enterprise data! You probably can’t do that today for every kind of data that needs governing up and down your stack, but the range is getting broader every day. Ultimately your aim should be to enable customer experiences and business outcomes to be as consistent as your business requires them to be, at a reasonable cost, with agility.