woensdag 16 juni 2021

Managing a Data Lake Project Part II: A Compelling Business Case for a Governed Lake


In Part I A Data Lake and its Capabilities we already hinted towards a business case but in this blog we make it a little bit more explicit.

A recap from Part I: the data lake capabilities

The business case for a data lake has many aspects, some of them present sufficient rationale on their own but that depends of course on the actual situation and context of your organisation. Therefore, I mention about eleven rationales, but feel free to add yours in the comments.

 We are mixing on-premise data with Cloud based systems which causes new silos

The Cloud providers deliver software for easy switching your on-premise applications and databases to Cloud versions. But there are cases where this isn’t possible to do this in one fell swoop:

  • Some applications require refactoring before moving them to the Cloud;
  • Some are under such strict info security constraints that even the best Cloud security they can’t be relied on. I know of  retailer who keeps his excellent logistic system in something close to a bunker!
  • Sometimes the budget or the available skills are insufficient to support a 100 % Cloud environment, etc…

This provides already a very compelling business case for a governed data lake: a catalog that manages lineage and meaning will make the transition smoother and safer.

Master data is a real pain in siloed data storage, as is governance...

A governed data lake can improve master data processes by involving the end users in evaluating intuitively what’s in the data store. Using both predefined data quality rules and machine learning to detect anomalies and implicit relationships in the data as well as defining the golden record for objects like CUSTOMER, PRODUCT, REGION,… the data lake can unlock data even in technical and physical silos. 

We now deal with new data processing and storage technologies other than ETL and relational databases: NoSQL, Hadoop, Spark or Kafka to name a few

NoSQL has many advantages for certain purposes but from a governance point of view it is a nightmare: any data format, any level of nesting and any undocumented business process can be captured by a NoSQL database.

Streaming (unstructured) data is unfit for a classical ETL process which supports structured data analysis so we need to combine the flexibility of a data lake ingestion process with the governance capabilities of a data catalogue or else we will end up with a data swamp.

We don't have the time, nor the resources to analyse up front what data are useful for analysis and what data are not

There is a shortage of experienced data scientists. Initiatives like applications to support data citizens may soften the pain here and there but let’s face it, most organisations lack the capabilities for continuous sandboxing to discover what data in what form can be made meaningful. It’s easier to accept indiscriminately all data to move into the data lake and let the catalogue do some of the heavy lifting.

We need to scale horizontally to cope with massive unpredictable bursts of of data

Larger online retailers, event organisations, government e-services and other public facing organisations can use the data lake as a buffer for ingesting massive amounts of data and sort out its value in a later stage.  

We need to make a rapid and intuitive connection between business concepts and data that contribute, alter, define or challenge these concepts

This has been my mission for about three decades: to bridge the gap between business and IT and as far as “classical” architectures go, this craft was humanly possible. But in the world of NoSQL, Hadoop and Graph databases this would be an immense task if not supported by a data catalogue.  

Consequently, we need to facilitate self-service data wrangling, data integration and data analysis for the business users

A governed data lake ensures trust in the data, trust in what business can and can't do. This can speed up data literacy in the organisation by an order of magnitude.

We need to get better insight in the value and the impact of data we create, collect and store.

Reuse of well-catalogued data will enable this: end users will contribute to the evaluation of data and automated meta-analysis of data in analytics will reinforce the use of the best data available in the lake. Data lifecycle management becomes possible in a diverse data environment.

We need to avoid fines like those stipulated in the GDPR from the EU which can amount up to 4% of annual turnover!

Data privacy regulations need functionality to support “security by design” which is delivered in a governed data lake. Data pseudonimisation, data obfuscation or anonimisation come in handy when these functions are linked to security roles and user rights. 

We need a clear lineage of the crucial data to comply with stringent laws for publicly listed companies

Sarbanes Oxley and Basel III are examples of legislation that require accountability at all levels and in all business processes. Data lineage is compulsory in these legal contexts. 

But more than all of the above IT based arguments, there is one compelling business case for C-level management: speeding up the decision cycle time and the organisation’s agility in the market.

Whether this market is a profit generating market or a non-profit market where the outcomes are beneficial to society, speeding up decisions by tightening the integration between concepts and data is the main benefit of a governed data lake.

Anyone who has followed the many COVID-19 kerfuffles, the poor response times and the quality of the responses to the pandemic sees the compelling business case:

  • Rapid meta-analysis of peer reviewed research papers;
  • Social media reporting on local outbreaks and incidents;
  • Second use opportunities from drug repurposing studies;
  • Screening and analysing data from testing, vaccinations, diagnoses, death reports,…

I am sure medical professionals can come up with more rationales for a data lake, but you get the gist of it.

So, why is there a need for a special project management approach to a data lake introduction? That is the theme of Part III.  But first, let me have your comments on this blogpost.

woensdag 9 juni 2021

Managing a Data Lake Project Part I: A Data Lake and its Capabilities

 A data lake can provide us with the technology to cope with the challenges of various data formats arriving in massive amounts, too fast and diverse for a classic data pipeline resulting in a data warehouse. As a the data warehouse is optimised for analysis of structured data, the inflow of unstructured data strings, entire documents, JSONs with n levels of nesting, binaries, etc… is simply too much for a data warehouse.

A data lake is an environment that manages any type of data from any type of source or process in a transparent way for the business. In tandem with a data catalogue, a lake provides data governance and facilitates data wrangling,  trusted analytical capabilities as well as self-service analytics to name a few.

If we zoom in on these capabilities, we can list these as the basic requirements for a minimum viable product:

  • Automated discovery, cataloguing and classification of ingested data;
  • Collaborative options for evaluating the ingested data;
  • Governance of quality, reliability, security and privacy aspects as well as lifecycle management;
  • Facilitates data preparation for analytical purposes in projects as well as for unsupervised and spontaneous self-service analytics;
  • Provides the business end users with an intuitive search and discovery platform;
  • Archives data where and when necessary.


Generic data processing map
Data comes from events that lead to business processes as well as from outside events that may become part of the business processes

Some vendors launch the term “data marketplace” to stress the self-service aspects of a data lake. But this position depends on the analytical maturity of the organisation. If introduced too early it may provide further substantiation for the claim that:

“Analytics is a process of ingesting, transforming and preparing data for publication and analysis to end up in Excel sheets, used a “proof” for a management hypothesis”.

What makes a data lake ready for use?

Meta data: data describing the data in the lake: its providence, the data format(s), the business and technical definitions,…;

Governance: business and IT control over meaning, application and quality of data as well as information security and data privacy regulation;

Cataloguing: either by machine learning or precooked categories and rule engines, data is sorted and ordered according to meaningful categories for the business.

Structuring: data increases in meaning if relationships with other concepts are modelled in hierarchies, taxonomies and ontologies;

Tagging: both governed and ungoverned tags (i.e. user tags) dramatically improve the usability of the ingested data. If these tags are evaluated on practical use by the user community they become part of a continuous quality improvement process;

Hierarchies: identical to tagging, there may be governed and personal hierarchies in use;

Taxonomies: systematic hierarchies, based on scientific methods;

Ontologies: a set of concepts and categories in a subject area or data domain that shows their properties and the relations between them to model the way the organisation sees the world.

zaterdag 29 mei 2021

Managing a Data Lake Project

With the massive growth of online generated data and IoT data, the proportion of unstructured and semi-structured data constitutes the bulk of the data that needs to be analysed. Whereas a 50 Gigabyte data warehouse to facilitate analysis of structured data was quite an achievement up to now, this number dwindles compared to the unstructured and semi-structured data avalanche.

Data Avalanche?

Yes, because compared to the steady stream of data from transaction processing systems, we now have to deal with irregular flows and massive bursts of incoming data that needs to be adequately processed to provide meaning to the data.
New data sources emerge, other than social media and IoT data, like smart machines and machine learning systems generating new data, based on existing sources. Managing various data types and metadata in impressive volumes are just a few technical aspects which can be solved by technology. The HR- , legal- and organisational aspects are level more complex, but aspects these are not in scope of this series of blog posts. 
We are adding extra process and event based decision support to our management capabilities and that alone is worth the cost, the trouble and the change management efforts to introduce a data lake.

See you at the Webinar!

Wednesday 9th June you can tune in on a short webinar hosted by the Great IT Professional. You can still register via this link. The webinar will be followed by a series of articles on how to manage the Data Lake project. Stay tuned!

Bert Brijs Webinar on Managing a Data Lake Project

woensdag 30 december 2020

New Inroads for Analytics in the Post Corona Era


OK, 2021 will not get rid of the virus immediately but the new consumer behaviour, induced by the pandemic will have lasting effects that need to be taken care of by brand owners, distribution channels and -consequently- by the analytics approach and infrastructure.

So, what is exactly this new consumer behaviour?

You already guessed: more online shopping and more pervasive switching to web shops from the local shops to compete with the incumbents. The local shop owners finally have understood the value of proximity combined with the convenience of online browsing and online ordering or preordering and collecting the order at the local shop.

But there’s more. Not only have the predominant shopping logistics changed; the product range has also undergone the influence of the various lockdown periods. Consumers have a tendency to shop for more luxury products in the food section as a means of self-indulgence and the dichotomy between convenience and fun shopping is getting clearer and larger. Some retail chains are already experimenting with automatic replenishment of convenience products using automated algorithms. But some supermarkets in the Benelux are combining convenience, fun and self-indulgence offering prepared meals that can be consumed in the shop. Plus, Albert Heyn and Jumbo are experimenting with the concept. This can have an impact on local restaurants who have survived on their take away service during the pandemic.

Due to Covid-19 this section where you can have a meal at a Plus supermarket is closed...

And how does this emerging consumer behaviour affect the analytics profession?

The larger distribution chains will continue to develop their centralised analytical systems. The data flows from the outlets’ cash registers to the central data warehouse and delivers customer and product insights as this has been the case since AC Nielsen built the first embryo of a retail data warehouse somewhere in the seventies.

New opportunities for innovation in analytics for large retailers lie in edge computing. Think of directed dialogues with the customer, analysing conversion rates from looking at products, holding them, inspecting them and finally putting the product in the shopping trolley and feeding it back to the pricing and communication in the isles.

Now, as local shops discover the value of customer data, syndicators will emerge to provide economies of scale and of scope to aggregate data of the local shops and provide benchmarks and high level customer insights as a first deliverable. It will take some serious investments in persuading the local shops to share their data but it will happen in the next three to five years. My experience with a data warehouse project for an association of independent retailers tells me it’s doable if you mimic the architecture of epidemiological analytics. These systems have the highest levels of information security combined with state of the art analytical capabilities. And so another product of this pandemic may contribute to new analytical solutions.

But the major shift in the analytics landscape is happening with the brand owners. Up to now, most brand owners were OK with the idea that customer behaviour data resided in the systems of large retailers. Some of the clever ones developed a data sharing approach with the retailers accepting the possibility of a biased view on their final customers.

Now the need for massive customer data for brand owners is unavoidable. New ways of collecting unfiltered customer data will emerge. Smartphones, fit bits and other devices will have new roles to play in this strategic movement.





woensdag 30 oktober 2019

Enterprise Architectures for Artificial Intelligence (III)

Taxonomies of Artificial Intelligence

There are at least five ways to position AI in the enterprise landscape:
  1. By processing method: batch, micro batch and real time
  2. By algorithm type: pattern recognition, clustering, associations, scoring, predictive, classification, text, speech and image mining, …
  3. By data type: high vs low dimensionality, graph data, self-describing data vs structured schema data, machine vs human sourced data, mediated data registration vs direct data registration,…
  4. By data behaviour: volatile vs stable data values, long vs data persistency,
  5. By analytics goals and or business process: churn prevention, prospect qualification, complex evaluations of loan applications, CVs, customer feedback, basket analysis, next best action proposals, fraud detection …

The enterprise architect will choose the relevant combinations between these taxonomies to produce a coherent end-to-end vision on the architecture. A possible selection criterion is the governance model used in the organisation. In a business monopoly analytics goals will be leading and combined with algorithm type. In an  IT monopoly processing methods combined with data behaviour is the most probable direction and in a duopoly, well,… that depends.

Let’s do an exercise and suppose this is the outcome of a duopoly governance model: combining the processing method with the algorithm type to indicate which processing method is most suited for the chosen algorithm type. Using this schema may help to manage expectations between the business and the IT people better.

Micro batch
Real time
pattern recognition
Ideal method for large data sets
Suited for simple patterns
Only as a binary in/out of pattern decision which implies a large (batch) training set
Ideal method for large data sets
Suited for simple clustering criteria
Only as Y or N adherence to an existing cluster which implies a large (batch) training set
Ideal method

Hardly possible
Develop a base line
Adjust the base line
Score against the base line
Develop a base line

Adjust the base line
Match with the trend
Train the dataset

Classify new data
Simple classification
text mining
Train the dataset
Reveal polarity, topics, etc…
Deliver alerts
speech mining
Train the dataset
Reveal polarity, topics, etc…
Deliver alerts
Image & video mining
Train the dataset
Classify images
Deliver alerts

From this crosstab, it becomes possible to position the concrete algorithms, the data sets and their life cycle management, the ingestion volumes, timing and the technology to deliver on the various promises made.
Other methods will give you paths to the same end result: a coherent and methodical inventory of the landscape, linking business processes to AI and data mining initiatives and routines as well as the data and the applications to deliver the goods. Based on a gap analysis, the enterprise architect can develop a roadmap that communicates with all parties concerned.

maandag 30 september 2019

Enterprise Architectures for Artificial Intelligence (II)

A generic model for primary processes

Every organisation is unique but most organisations share some basic principles in the way they operate. Business processes have some form (between 5 and 100%) of support by online transaction systems (OLTP). Business drivers like consumer demand, government regulations, special interest groups, technological evolutions, availability of raw materials and labour and many others influence the business processes intended to deliver a product or service that meets market demand within a set of constraints. These constraints can range from enforcing regulatory bodies to voluntary self-regulation and measures inspired by public relations objectives.
This is a high level approach of how AI can support business processes

AI and enterprise architecture
High level generic architecture

Business drivers are at the basis of business processes to realise certain business goals and delivering products for an internal or external customer.  These processes are supported by applications, the so-called online transaction processing (OLTP) systems.
Business process owners formulate an a priori scoring model that is constantly adapted by both microscopic transaction data as well as historic trend data from the data warehouse (DWH). Both data sources can blend into decision support data, suited for sharply defined data requirements as well as vague assumptions about their value for decision making.  The decisions at hand can be either microscopic or macroscopic. 

Introducing AI in the business processes

As an architect one of the first decisions to make is whether and when AI becomes relevant enough to become part of routine business processes. There are many AI initiatives in organisations but the majority is still in R & D mode or –at best- in project mode.  It takes special skills to determine when the transition to routine process management can provide some form of sustainable added value.
I am not sure if these skills are all determined and present in the body of knowledge of architects but here are some proposals for the ideal set of competences.
  • A special form of requirements management which you can only master if the added value as well as the pitfalls of AI in business processes are thoroughly understood,
  • As a consequence, the ability to produce use cases for the technology,
  • Master the various taxonomies to position AI in a correct way to make sure you obtain maximum value from the technology (more on this in a next post),
  • Have clear insights in the lifecycle management of the various analytical solutions in terms of data persistency, tuning of the algorithm and translation into appropriate action(s).

In the next post, I will elaborate a bit more on the various taxonomies to position AI in the organisation.