Download our Webinar slides from the ITMPI

We have posted the slides of two webinars on our Lingua Franca site for you to download:

How Business Analysis for Business Intelligence Creates Strategic Value

How to Keep Business Intelligence in Sync with Your Strategic Priorities

Send us your feedback, we look forward to reading your comments.

Business Intelligence has become too big to allow failure.

Four speakers between Ralph Kimball’s sessions, four topics and one unifying thought: BI is getting to big to allow failure. The first Business Analytics for All Insight session which took place in Brussels the 12th November gathered over 250 attendees to hear Ralph Kimball’s insights on the data warehouse design principles and how the Big Data phenomenon fits in this architecture. I gladly refer to the Kimball Group’s website with articles like these for his vision on Big Data.  

But between Ralph’s talks in the morning and in the afternoon, four other topics were discussed which all lead to the same conclusion: BI has become too big, too much of a strategic commitment to allow for sloppy business analysis and project management.

Annelies Aquarius, European BI Project manager from the Coca-Cola Company illustrated the anytime- anything-anywhere aspects of mobile BI. Jelle Defraye from Laco made a case for self service BI.  Jos Van Dongen from SAS taught us the basics from data visualisation and Guy Van der Zande from USG ICT Professionals explained why a well organised BI Competence Center (BICC) is essential to manage technology trends and changing business requirements.

For a full description of their talks we refer to the website.

It is time for proper BI business analysis and project management

Let me explain my point. With the growth of users, user types, data a lot of side effects have come into play since the early days of DSS where you offloaded a few tables to make reports for the CFO.

Exabytes of data flowing in at incredibly high speeds from a myriad of data sources in structured, semi-structured and structured formats need to be exploited by more people in a faster decision making cycle which is not limited to the strategic apex anymore. Thus the feedback loops become more complicated as the one-to-many relationship of top management and the workforce now becomes a many-to-many relationship between more and more decision making actors in the organisation. Self service BI, mobile BI and visualisation are all part of the solution and the problem if your organisation has no duopolistic governance from IT and the business. because both business processes and data management processes need to be mutually adjusted to allow for maximum return on investment . The alternative is chaos. So there you have the true value of  a well working BICC.

But to get there and to stay on that level, only a thorough business analysis process and the proper BI project management method will increase the success rate of business analytics. This success rate worries me. Because after twenty years in this business I am still seeing failure rates of 80% in BI. If we’d had the same rate of improvement in medicine as in BI we would still be using leeches and bleeding our patients regardless of the disease.

Interview with a Business Intelligence User

Let’s call him Eric. Because after the interview Eric decided he’d better remain anonymous. Some of his answers could cause too much controversy in the organization, a major European logistics company.

Eric is BI manager in this company and when listening to his vision, his worries and his concerns, it is like taking stock of the most common disconnects between IT and the business.

Question: What struck you the most when reading the book “Business Analysis for Business Intelligence”?

Eric: I think you have documented your book well and chose a useful starting point. Most literature in Business Intelligence (BI) is divided in two categories.  On one side you have a myriad of theoretical works on strategy and management,  performance management  and the inevitable scorecards and dashboards. On the other side are plenty technical publications available discussing IT performance and optimum data structures. What many of these books lack is a vision of how business and IT should join hands to produce optimum BI results. From my 20 years’ experience with BI, this is a serious problem.

Question: What are the major impediments for your performance as a BI manager?

Eric: I see three roadblocks: IT is either unaware or unwilling to admit that BI cannot be standardized. But the business itself is not always capable of producing crisp and consistent definitions to produce a coherent analytical frameworks changes its mind”. And last but not least: the complexity of some analytics also causes a lot of problems and is –of course- compounded by the two previous roadblocks.

Question: Why would IT not be aware of the need for flexibility? Some IT guys we know say stuff like “The business guys always change their mind”

Eric:  No, it’s not about business changing its mind because that can be prevented through thorough analysis as described in your book. It is more about the prejudice that BI solutions are templates you can use anywhere. IT people underestimate the uniqueness of each business process and its context, culture and informal issues that make every business unique. Management can shift its attention and rearrange its priority list in days and weeks. If IT can’t follow, the users look for ad hoc (and often badly architected) solutions.

Thank you for sharing this with  us, Eric. 

To our readers: don’t hesitate to share your experience with the gap between business and IT in BI. We can all learn from this!

Book Review Retail Analytics, The Secret Weapon by Emmett Cox

by Emmett Cox Wiley and SAS Business Series 2012 

The book cover opens with promising references and from Tom Davenport and all this on just 142 pages of 10 point Times new Roman with double spaced interlines. Wow! This needs further reading I guess. Chapter One gives a very high level introduction to the main process: sales and replenishment and chapter two introduces the ins and outs of retail data management. A diagram of the transaction log files could improve the didactic impact of this chapter and the comparison between denormalised and normalised data is at best superficial. But I guess the readers that, according to the editor are helped with their critical management decisions don’t bother much about technical mumbo jumbo, so let’s move on. Although this inspires me to publish an in depth blog on the pros and cons of these datamodels that might interest senior-level management. Because it is about money, time and quality and there is no one size fits all choice to be made.

Inspiring introduction

Cox sums up a few interesting case studies ranging from trade area modelling, site selection modelling, competitive threat analysis, merchandise mix modelling, marketing effectiveness tracking, brand analysis, clicks and mortar analysis, cross sell analysis to market basket analysis. And he does so in plain English which at least will trigger the interest of senior management. The schema on page 25 gives a quick and clear view on how the analytical domains are part of a chain driving important retail and merchandising decisions. While reading these case studies you are confronted with a very mild and unobtrusive form of product placement with sentences like “The statistical software company SAS has a strong set of utilities within its SAS Enterprise Miner that we used heavily” (p. 33) Yet, the case studies are very credible and well presented, even if they don’t go in depth into the statistical details. I guess the target group of this book will tell guys like me: “Find out how we can apply this method in our stores”. Chapter three pays attention to the specifics of the apparel industry and does so in a didactic manner, illustrating clearly the constraints apparel buyers have to work with: long order lead times, optimum price elasticity management, optimum sales promotion management,… Having worked in sales promotion for a mail order company doing 80% of its sales in fashion I recognised the stressful situations even the best retail analytics won’t attenuate. The fourth chapter spends quite a few pages on explaining what a GIS tool can do and how it works which is a bit redundant in the days of Google and Wikipedia, leaving only a few pages for the application of GIS in retail. For example: the combination of category management and GIS analytics can produce powerful results I am sure the author could illustrate to his readers. It is as if Mr. Cox apologises for this in the fifth chapter opening with “Chapter 4 was an extremely technical illustration of the GIS tool.” Er.. well,… I wasn’t bothered with points, vectors and surfaces of the GIS world, I just wanted to know more about the applications…. 

 A 101 on in store promotion 

 In-store marketing and presentation get a lot more attention with 31 of the 142 pages. Pricing strategies are explained very well as they are the meat of retail analytics. But Cox’ emphasis on convenience shopping hides its counterpart: fun in shopping which also needs tailor made analytical insights to enhance the profitability potential. His reference to DB2’s datablades brought me back to the heydays of Informix, once the leading RDBMS before Oracle displayed a superior sales and marketing strategy. All in all this chapter is a little too narrowly focused on the US situation. Phrases like “Most retail chains are now open 24 hours a day…” (p. 98) will raise eyebrows in Europe and sometimes the author gets lost in details like the description on how floor graphics can be removed “without leaving any residue behind”. I wonder who in the analytical community will bother about this. And when I read stuff like “another form of subliminal messaging” (p. 101) related to clearly detectable things like baking smells I get confused. “Subliminal” means a stimulus the audience is not aware of. And literature and research are not in unison about the effects of subliminal messages in stores. Chapter 6 handles the store operations, an underestimated source of information opportunities to be used; Think of workforce management where you optimise the match between labour cost and service level based on POS data and longitudinal time series. Think of differentiated messages to consumers, HVAC control, intrastore communication, replenishment: Cox explains these operations and the link with retail analytics in a clear and concise way. 

Introduction to loyalty marketing

Last but not at all least, the chapter on loyalty marketing. Since Reichheld and Sasser’s publications on customer loyalty we are aware that the link between excellence, customer satisfaction and loyalty is not always obvious and in many cases it is not even existent! Measuring and managing customer loyalty is a real pain in retail. To me, attracting customers to loyalty programs is not equal to creating customer loyalty. It is just a form of prolonged sales promotion that –if stopped- would cause an immediate drop in sales. True loyalty is quite different. It took me one and a half page to read about it: “(…) as long as the consumers do not just cherry-pick the discounts and rewards offers.” Spot on Emmett! Weeding out the cherry pickers is the true challenge for retail analytics. The author offers a checklist before you begin a loyalty program.

To conclude

This book will certainly deliver value for novices in retail marketing and analytics. If you're a CEO or CMO of a retail chain and wonder whether you should invest heavily in an analytical environment, you are probably too late as the competition will have obliterated your obsolete business.That doesn't mean other C-level executives couldn't benefit from this book.  But to be really relevant for other profiles like a CIO or COO, it should have related data and process management to analytics with diagrams and models. 

A Short Memo for Big Data Sceptics

In an article in the NY Times from 17th August by James Glanz, a few Big Data sceptics are quoted. Here is a literal quote: Robert J. Gordon, a professor of economics at Northwestern University, said comparing Big Data to oil was promotional nonsense. “Gasoline made from oil made possible a transportation revolution as cars replaced horses and as commercial air transportation replaced railroads,” he said. “If anybody thinks that personal data are comparable to real oil and real vehicles, they don’t appreciate the realities of the last century.” I respectfully disagree with the learned scholar: the new oil is a metaphor for how our lives have changed through the use of oil in transportation. Cars and planes have influenced our social lives immensely but why shouldn't Big Data do so in equal or even superior order? Let me name just a few: 

• Big Data reducing traffic jams (to stick close to the real oil world) 
• Big Data improving the product-market match to the level of one to one, tailoring product specifications and promotions to individual preferences, 
• Big Data improving diagnostics and treatments in health care combining the wisdom of millions of health care workers and logged events in diagnostics epidemiologic data, death certificates etc... 
• Big Data and reduction of energy consumption via the smart grid and Internet of things to automate the match between production and consumption, 
• Big Data in text mining to catch qualitative information on a quantitative scale improving the positioning of qualitative discriminants in fashion, music, interior decorating etc... and of course... politics Ask the campaign team from 44th President of the United States and they will tell you how Big Data oiled their campaign.

As soon as better tools for structuring and analysing Big Data become available and as soon as visionary analysts are capable of integrating Big Data in regular BI architectures the revolution will grow in breadth and depth. Some authors state that entirely new skills will be needed for this emerging market. If I were to promote training and education I'd say the same. But from where I stand today I think the existing technological skills in database and file management may need a little tweaking say a three or five day course but no way is there a need for an MBD (Master in Big Data) education. On the business side of things there may some need for explaining the works of semi structured and unstructured data and their V's which already add up to seven. I believe it is going in the same direction as the marketing P's where Kottler's initial four P's were upgraded to over thirty as one professor of marketing churned out this intellectual athletic performance. Let's sum them up and see if someone can top them: Volume: a relative notion as processing and storage capabilities increase over time Velocity: ibid. Variety: also a relative notion as EBCDIC, ASCII, UTF-8 etc... are now in the company of video and speech thanks to companies like Lernhout and Hauspie whatever the courts may have decided on their Language Development Companies, Volatility: I have added this one in an article you can find on the booksite on "Business Analysis for Business Intelligence" because what is true today may not be true tomorrow, so it is not about the time horizon you need to store the data as some authors claim because that will be defined by the seasonality. The problem with these data is there might not be any seasonality in them! Veracity: how meaningful are the data for the problem or opportunity at hand? Validity: meaning Big Data can only be useful if validated by a domain expert who can identify its usefulness. Value: what can we invest in recording, storing and analysing Big Data in return for what business value? This is one of the toughest questions today as many innovative organisations follow the Nike principle: "Just do it". And that, professor Gordon is just what all the pioneers did when they introduced the car and the aeroplane to their society, ignoring the anxious remarks from horse breeders and railroad companies. Remember how the first cars where slower than trains and horses? I rest my case.

Time to muse over “holiday”

Greetings from a 30°C office, in the north of Belgium, Flanders. It’s late in the evening, a good time for some lighter philosophising before packing the suitcase.

The DATE dimension is a very simple flat table (although some designers and DBA’s prefer third normal form). It contains surrogate keys, the date, the name of the weekday, month, day number, week number, month number, quarter, trimester and semester number, IsLastDayOfMonthFlag and what have you. There will also be a IsAHolidayFlag somewhere and in case you only work in a few countries, you simply add a column per country.

In case you operate on a global scale  you may want to snowflake a bit to support truly global analytics and compare apples with apples.

Example of a snowflaked Holiday model

Holidays can explain a lot

Imagine you are comparing last year’s June sales with June of this year. Last year, June had no holidays and 21 working days. This year it has 20 working days and one holiday, i.e. 19 working days. In case your revenue is directly linked to the number of working days, this explains for a revenue drop of no less than ten percent in June this year! And in case you sell holiday related products like ice cream or beer it may well run into the other direction explaining a sales increase of over twenty percent or more…

So, here’s my advice: enjoy your holiday but don’t forget to integrate it in your analysis.

When will transaction systems and analytical appliances converge?

 There is no harm in being a bit visionary sometimes…

For the last three or four decades, the gap between online transaction processing (OLTP) and business intelligence(BI) was impossible to cross. Neither technology nor the architecture of both systems allowed integration on data level. So cumbersome and expensive technical infrastructure was needed to extract, transform and load (ETL) data into data warehouses and data marts to exploit the information assets, hidden in the records of the OLP systems.

One of the main problems was that OLTP applications were never conceived with a view on BI. For example, the first Customer Relationship Management (CRM) Applications stored the customer address  as an attribute of the CUSTOMER entity instead of treating it as an entity in itself. When the first geographical information systems and geographical analytical systems came along, CRM developers remodelled the customer database and made ADDRESS a separate entity.

Imagine…

… a greenfield situation where you could develop any OLTP application from scratch and with an added BI perspective. What would it look like? What would be the guiding design principles to develop an OLTP application that plugs in seamlessly into a BI infrastructure.

Because, don’t get me wrong, there will still be a need for a separate and dedicated BI infrastructure as well as a specific BI architecture. But the ETL , the master data management (MDM) and data quality (DQ) management should no longer be a pain.

What would it take to relieve us from ETL, MDM and DQ chores?

Or how the simplest things are the hardest to realise. It would take:

• A canonical target data model and subsequently,
• A function in any OLTP application to drop off its data in that target format for the BI infrastructure to pick it up, be it via an enterprise service bus or via a bulk load procedure,
• A hub and spoke system where the master data objects are managed and replicated in both the OLTP and the BI applications,
• A uniform data quality policy, procedure and checking of accuracy, consistency, conformity and being in line with business rules of lower order (e.g. “a PARTY > PHYSICAL PERSON must have a birth date older than today”), but also of a higher order (e.g. a CUSTOMER who is under age should have a parent in the customer database whose co-signature must be on the ORDER form”)

So why is this not happening?

One explanation could be that application vendors claim they have full BI functionality built in in their solution. I remember a client with an ERP solution that contained over 300 standard analytics and reports of which the client used less than 3 %, eight to be exact. Not a great BI achievement I guess, but certainly an attempt to have 100% account control...

Another explanation could be the that we are still far away from a canonical model integrating transaction processing with BI, although data warehouse appliance vendors will tell you otherwise. Nevertheless, the dream has been kept alive for decades. I remember a data warehouse guru from the nineties who developed what he called verticals for BI to be used in telco, retail and finance, among others. They were just target models in the third normal form to set up a corporate data warehouse, so no OLTP included. He managed to sell them to a database vendor, bought a nice sailing boat (or rather a yacht) and disappeared from the BI stage. The database vendor peddled these models for a few years but in the years after 1999 I never came across these templates. And the vendor never mentioned their existence since then. Conclusion: we're still far away from a one-size-fits all transaction and analytics model.

What is the second best choice?

I can come up with a few ideas that will still be hard to realise.  Think of an industry wide data type standard for transactions and BI purposes with the transformation rules documented. For example: a timestamp in a transaction database should always be in the form of 'YYYY-MM-DD HH:MM:SS' and two numbers are added for BI purposes: the standard day number, counting from an internationally accepted date like 1900-01-01 and the second number, which is a figure between 0 and 86.400 to represent the lowest grain of the time dimension.

The next step is to design industry dependent star schemas for basic analytics every organisation needs in that industry. There is after all, a growing body of knowledge in retail analytics, telco, finance, production, supply chain etc… If we can already achieve that, I will probably see my retirement date coming which is 2033-01-04 00:00:00.