Shadow BI: shady or open for business?

Shadow BI is a common phenomenon in any organisation where the business has an Open or Microsoft Office on the PC; i.e. 99.9%  of the users can mash up data in spreadsheets, perform rudimentary descriptive and test statistics and some predictions using linear regression. Some of them download open source data science tools like Weka and KNIME and take it a step further using fancier regression techniques as well as machine learning and deep learning to come up with new insights.

On October 18, BA4All’s Analytic Insight 2016 had a peer exchange with about 60 analytics professionals to reflect on three questions:

• ·        What are the top three reasons for Shadow BI?
• ·        What are the top three opportunities Shadow BI may bring to the organisation?
• ·        What are the top three solutions for the issues it brings about?

The most quoted reasons for Shadow BI

Eww! IT is taking some heavy flak from the business: “ICT lacks innovation culture”, “IT wants to control too much!” and especially the time IT takes to deliver the analytics was high on the list.

Other, frequently mentioned reasons were the lack of business knowledge, changing requirements from the business  and the inadequate funding clearly indicate a troubled relationship between ICT and the business as the root cause for Shadow BI.

Yet, opportunities galore!

Shadow BI can improve efficiency in decision making provided the data quality is fit for purpose.  In case of bad data quality it may provoke some lessons learned for the business as they are the custodians of data quality.

This under-the-radar form of BI can also foster innovation as users are unrestrained in discovering new patterns, relationships and generate challenging insights. and provide faster response to business questions.

A mix of tech and HR came up in the discussions

The top 3 solutions for issues with Shadow BI

The group came up with both technical and predominantly organizational and HRM solutions. Here are the human factors:

• ·        market BI to the business and IT people,
• ·        governance (also a technical remedy if the tools are in place)
• ·        empowerment of the business
• ·        adopt a fail fast culture
• ·        knowledge sharing and documentation
• ·        strategy alignment
• ·        integrate analytical culture and competencies in the business
• ·        engage early in the development process

And these are the technical factors:

• ·        governance tools
• ·        Self-service BI and data wrangling tools
• ·        Sandboxes
• ·        Optimise applications for analytics

For a discussion on some of the arguments we refer to our next post in a few days

Why Master Data Management is Not Just a Nice-to-have…

Sometimes the ideas for a blog just land on your desk without any effort. This time, all the effort was made by one of the world’s largest fast moving consumer goods companies with 355.000 employees worldwide.

But this is not a guarantee for smart process and data management as the next experience from yours truly will illustrate.

The Anamnesis

One rainy day, the tenth of May, I receive a mail piece with a nice promotional offer: buy a coffee machine for one euro while you order your exquisite cups online. On rainy days you take more time to read junk mail and sometimes you even respond to them. So I surfed to their website and filled out the order form. After introducing the invoice data (VAT number,invoice address,…) an interesting question popped up:

Is your delivery address different from your invoice address?

As a matter of fact it was, it was the holiday season and the office was closed for a week but I was at a customer’s site and thought it would be a good idea to have it delivered there.

So I ticked the box and filled in the delivery address.  That’s when the horror started.

Because, when I hit the order button, there was no feedback after saving, no chance to check the order and wham, there came the order confirmation by e-mail.

Oops: the delivery address and the invoice address were switched. Was this my fault or a glitch in the web form? Who cares, best practice in e-commerce is to leave the option for changing the order on details and even cancelling the order, right? Wrong. There was no way of changing the order, all I could do was call the free customer service number to hopefully make the switch undone.

10th May, Call to Consumer Service Desk #1 

IVR: “Choose 2 if this is your first order”

Me: “2”

Client service agent: “What is your member number?

Me: “I don’t have member number since this is my first order. It’s about order nr NAW19092… “

Client service agent: “hmmm we can’t use the order number to find your data. What is your postcode and house number?”

Me: “This is tricky since I want to switch delivery address with the invoice address. You know what, I’ll give you both”.

Client service agent: Can’t find your order”

So, I am completely out of the picture: not via the company, the address, the order number, let alone a unique identifier like the VAT number

Client service agent: “Please send a mail to our service e-mail address “yyy@zzz.com”.

Me: “Send e-mail” Result: no receipt confirmation, no answer from this e-mail address. Great customer experience guys!

10th May Call to Consumer Service Desk #2

Client service agent: “Oh Sir, you are calling the consumer line, you should dial YYY/YYYYYY for the business customers”

Me:" But that’s the only phone number on your website and the order confirmation???!!!"

10^th May 2 PM Call to Business customer service #3

Client service agent: “Let me check if I can find your order”… (2’ wait time) “Yes, it’s here how can I help you?”

Me: “I want to switch the invoice with the delivery address”

Client service agent“OK Sir, done”

11^th May: The delivery service provider sends a message the delivery is due on the original address from the order.

No switch had been made…

Call to DPD? Too late.. these guys were too efficient...

The Diagnosis, What Else?

Marketing didn’t have a clue about the order flow and launched a promotion without an end-to-end view on the process which resulted in a half-baked online order process: no reviewing of the order possible, no feedback and the wrong customer service number on the order confirmation.

Data elements describing CUSTOMER, ORDER and PRODUCT may or may not be conformed (from the outside hard to validate) but they are certainly locked in functional silos: consumers and companies.

Customer service has no direct connection to the delivery process

The shipping company (DPD) provided the best possible service under the circumstances.

And this is only a major global player!  Can you imagine how lesser Gods screw up their online experience?

Yes, it can get worse!

One of my clients called me in on a project that was under way and was seriously going south.

What happened?  The organisation had developed a back office application to support  a public agenda of events. As a customer of this organisation you could contact the front desk who would then log some data in the back office application and wrap up the rest of the process via e-mail. Each co-worker would use his own “data standards” in Outlook so every event had to be handled by the initial co-worker if the organisation wanted to avoid mistakes. No wonder some event logging processes sometimes took quite a while when the initiator was on a holiday or on sick leave…

A few months later -keep that in mind- the organisation decided to push the front desk work to the web and guess what? Half the process flow and half the data couldn’t be supported by the back office application because the business logic applied by the front desk worker wasn’t analysed when developing the back office app.

Siloed application development can lead you to funny (but unworkable) products

So, please all you folks out there, invest some money in an end-to-end analysis of the process and the master data. It’s a fraction of the building cost and it will save you tons of money and bad will with customers, coworkers and suppliers.

Afterthoughts on Data Governance for BI

Why Business Intelligence needs a specific approach to data governance

During my talk at the Data Governance Conference, at least one of my audience was paying attention and asked me a pertinent question. “Why should you need a separate approach for data governance in Business Intelligence?”

My first reaction was “’Oops, I’ve skipped a few stadia in my introduction…” So here’s an opportunity to set things right.

Some theory, from the presentation

At  the conference, I took some time to explain the matrix below.

Data portfolio management as presented at the 2016 data governance conference in London

If you analyse the nature of the data present in any organisation, you can discern four major types.

Let’s take a walk through the matrix in the form of an ice cream producer.

Strategic Data: this is critical to future strategy development; both forming and executing strategy are supported by the data. By definition almost, strategic data are not in your process data or at best are integrated data objects from process data and/or external data. A simple example: (internal) ice cream consumption per vending machine matched with (external) weather data and an (external) count of competing vending machines and other competing outlets create a market penetration index which in its turn has a predictive value for future trends.

Turnaround Data: critical to future business success as today’s operations are not supported, new operations will be needed to execute. E.g.: new isolation methods and materials make ice cream fit for e-commerce. The company needs to assess the potential of this new channel as well as the potential cannibalizing effect of the substitute product. In case the company decides not to compete in this segment, what are the countermeasures to ward off the competition? Market research will produce the qualitative and quantitative data that need to be mapped on the existing customer base and the present outlets.

Factory Data: this is critical to existing business operations. Think of the classical reports, dashboards and scorecards. For example: sales per outlet type in value and volume, inventory turnover… all sorts of KPIs marketing, operations and finance want every week on their desk.

Support Data: these data are valuable but not critical to success. For instance reference data for vending locations, ice cream types and packaging types for logistics and any other attribute that may cause a nuisance if it’s not well managed.

If you look at the process data as the object of study in data governance, they fall entirely in the last two quadrants.

They contribute to decision making in operational, tactical and strategic areas but they do not deliver the complete picture as the examples clearly illustrate. There are a few other reasons why data governance in BI needs special attention, If you need to discuss this further, drop me a line via the Lingua Franca contact form.

Data Governance in Business Intelligence, a Sense of Urgency is Needed

The Wikipedia article on data governance gives a good definition and an overview of the related topics. But although you may find a few hints on how data governance impacts the business intelligence and analytics practice, the article is living proof that the link data governance with BI and Analytics is not really on the agenda of many organisations.

Sure, DAMA and the likes are reserving space in their Body of Knowledge for governance but it remains on the operational level and data  governance for analytics is considered a derived result from data governance for on line transaction processing (OLTP). I submit to you that it should be the other way around. Data governance should start from a clear vision on what data with which degree of consistency, accuracy and general quality measures to support the quality of the decision making process is needed. In a second iteration this vision should be translated into a governance process on the source data in the OLTP systems. Once this vision is in place, the lineage from source to target becomes transparent, trustworthy and managed for changes. Now the derived result is compliance with data protection, data security and auditability to comply with legislation like Sarbanes Oxley or the imminent EU directives on data privacy.

Two observations to make my point

Depending on the source, between 30 and 80 percent of all Business Intelligence projects fail. The reasons for this failure are manifold: setting expectations too high may be a cause but the root cause that emerges after thorough research is a distrust in the data itself or in the way data are presented in context and defined in their usability for the decision maker. Take the simple example of the object “Customer”. If marketing and finance do not use the same perspective on this object, conflicts are not far away. If finance considers anyone who has received an invoice in the past ten years as a customer, marketing may have an issue with that if 90 % of all customers renew their subscription or reorder books within 18 months.  Only clear data governance rules supported by a data architecture that facilitates both views on the object “Customer” will avoid conflicts.

Another approach: only 15 – 25 % of decision making is based on BI deliverables. On the plus side it may mean that 75 % of decision making is focused on managing uncertainty or nonsystematic risk which can be fine. But often it is rather the opposite: the organisation lacks scenario based decision making to deal with uncertainty and uses “gut feeling” and “experience” to take decisions that could have been fact based, if the facts were made available in a trusted setting.

Let’s spread the awareness for data governance in BI

More on this in my speech at the Data Governance Conference in London, the 17thMay

In the meantime,I’d like your opinion on this matter. It takes just three minutes to answer four questions. 

Many thanks in advance!

May I have three minutes of your time?

But I need your help...

To asses the present state of art in Data Governance and Analytics: how are data definitions, formats, locations, security, privacy and other aspects governed for analytical purpose? But most of all, why are you governing data and what is the level of data governance in your organisation?

Get Lingua Franca’s Presentation on the Data Governance Conference Europe 2016

“How Data Governance Works with BI”

But before we send you the proceeds of the conference, we ask you for a favour in return.

Fill in four answers on a questionnaire you can find here. We expect about 400 answers from all industries in the EU and the Americas. A high level report will be integrated in our presentation but you will get the full report if you tick the box on the form. And rest assured, you will not be spammed with offers or other unwanted solicitations!

For a full programme on the Data Governance Conference, follow this link.

Many thanks in advance!

Bert

A Little Side Step into Norway: BI 2016 Forecast

The Norwegian leading blog on Business Intelligence published my view on BI in 2016.

Check the prognosis here and get back to me at the end of the year to see if my crystal ball was clear on at least a few issues.

A Little Late: My Professional New Year's Resolution