The Eternal Business Intelligence Conundrum

Finding an Optimum between a Manageable BI Architecture and Business Agility

This is the second post in a series of three about programme management in Business Intelligence (BI). In the previous post we positioned project- and programme management in BI and the latter’s relationship with BI architecture.

In this post, we discuss the universal and eternal problem, conflict, dialectics,… (call it what you want) between the business who wants a decision support solution here and now, no matter what the consequences for the IT department are and the IT guys who want to steer the team and the infrastructure into calm waters. “Calm waters” meaning a strict architectural, TOGAF based approach to managing the BI assets. 

Head for the Cloud!

I won’t describe the situations where the IT guys –according to the business- waste time with the introduction of new tools and the business strike a direct deal with a vendor, using the tool completely outside the managed environment. 

Figure 2 The data mining tool Data Maestro is an example of a powerful cloud based tool

Needless to say that many cloud based solutions offer a solution with a small IT footprint: all the business needs is a browser. Well, that’s what the business thinks. Issues like data quality, data governance and data security are not always handled according to corporate standards and legislation on data privacy and data security is becoming stricter and more repressive every so many years.

What Business Stakeholders Need

As I pointed out in the section “Managing Strategy” (Business Analysis for Business Intelligence p. 66 – 71) business stakeholders need decision support for their intended strategy as well as emergent strategies (note the plural in the latter). To support analysis and monitoring of intended strategies (i.e. the overall business plan or a functional strategy as described in a marketing-, HR- or finance plan for example) a balanced scorecard (BSC) does the job. If well done, a BSC aligns all parties concerned around a well-designed causal model breaking down strategic priorities into critical success factors and key performance indicators as well as a project plan, a data model and an impact study on the existing analytics architecture. But to capture, evaluate, monitor and measure the impact of emergent strategies is a different ball game. The business intelligence infrastructure needs an agile approach to produce insights on the fly. Some vendors will suggest that all can be solved with in-memory analytics. Others suggest the silver bullet is called “Self Service BI” (SSBI) Yet even the most powerful hard- and software is a blunt and ineffective weapon if the data architecture and the data quality are in shambles.

Sometimes new tools emerge, producing solutions for niches in finance, marketing or production management which cause the business to urge IT for adopting these tools. This ends with either a mega vendor acquiring the niche player or the niche player broadening its offerings and competing head on with the established vendors. In any case, if the IT department happens to have standardised on the “wrong” technology partner, there will be bridges to cross for both…

Other than issues with new software “interfering” with IT’s priorities, most of the troubles are found in the data architecture. The reason is simple: if not all BI projects are backed by an enterprise wide data architecture that is connected with BI programme management, new information stovepipes will emerge. This is quite ironic as the initial reason for data warehousing was to avoid the analytic stovepipes on transaction systems. So here’s my advice to the business:

Whatever business you are in, make sure you have an enterprise view on the major information objects for your analytic projects

Without it, you are destined to waste money on rework, on incomplete and even false information.

What IT Stakeholders Need

IT management has many constraints do deal with. Keeping up with business requirements, while getting the biggest bang for their buck means pooling skills, facilities and technology components to optimise license cost, education and training and hard- and software performance. The final objective is to provide high service levels and keeping their customers happy at a reasonable budget. But if ”happy customer” means: acquiring new, exotic software, training new skills and insourcing expensive tech consultants from the vendor to explore new terrain without experience or knowledge of best practices, then IT management may be at the short end of the stick.

Take the example of data visualisation ten years back. Business had a point that the existing vendors weren’t paying too much attention to good visualisation to produce better insights in data. Even the most common tools had problems creating a histogram, let alone sophisticated heat maps or network diagrams. Then came along vendors like Tableau Software selling end user desktop licenses at affordable rates, educating the business to enjoy the benefits of visual exploration of data. The next step in this “camel’s nose” or “puppy dog approach” is getting the organisation to acquire the server for better management, performance and enterprise wide benefits of the technology.

So here’s my advice for IT Management:

If a new technology becomes available, it will be used. Make sure it is used in a managed and governed way instead of contributing to information chaos.

Don’t fight business intelligence trends that have a pertinent business case, fight BI fads only.

A Governance Decision Model for Conflicting Interests

I don’t like dogmatic thinking in management but when it comes to governance in BI, I will defend this dogma till the bitter end: only duopolistic governance will produce the best results in analytics.

That a business monopoly won’t work was clear after a consulting mission where I found a data warehouse with no less than six (6!) time dimensions. This extreme situation can only be explained by what I call “the waiter business analysis model”. Without any discussion, counterarguments nor suggestions, the analyst-waiter brings the ordered tables, cubes and reports to please the business. If the business funds the projects solely, then accidents will happen.

Figure 3 The BI Waiter Model: don't argue with the customer, bring him what he wants, no matter what...

But IT monopolies also are a recipe for failure in BI. At another client’s site, the IT department repeats over and over “x unless…” (x is a well-known BI tool provider). As it happens, this tool provider is lagging seriously in data mining and visualisation functionality so the business is wasting money on external service providers who do the analytics off line. Another source of waste are business managers installing software on their private PC to explore new ways of analytics at home.

In a duopolistic governance model, decision makers from both sides have to consider five key governance decisions. This will result in a better mutual understanding of each other’s concerns and priorities as well as provide a roadmap towards a managed analytical environment.

The Five Key BI Governance Decisions

(from my book Business Analysis for Business Intelligence, page 300 -301)

1.       BI Principles decisions:

a.       In what measure do we value data quality in the transaction systems?

b.      If we have a trade off between security issues and potential gains from better distribution of information, which direction do we choose?

c.       Do we choose a proactive or a reactive attitude towards our BI users, i.e. do we deliver only the required information or do we make suggestions for enhancements?

2.       BI Architecture decisions

a.       Do we follow the general architecture policies or is there a compelling reason to choose an alternative route?

b.      If we need alternatives, where will they be of importance: in databases, ETL tools, BI server(s), client software,…?

3.       BI infrastructure decisions

a.       What are the shared IT services the data warehouse will use?

b.       What part of the infrastructure will be organised per department or business unit?

c.      What are the access methods for the information consumers: local client PC, PDA, web based, VPN,…?

4.       Business Application needs

a.      Specify the business need

b.      Specify the urgency

c.      Present alternative solutions

5.       Prioritisation of investments in BI

  a.        How will we evaluate the priorities?

  b.        Who will handle conflicting interests?

  c.        Which user profiles will be served first?

  d.        Which subject areas will be tackled first?

Figure 4 More on BI Governance in this book, available in all major bookstores

In the next post I will have a look into the next generation of information design and architecture. Comments are welcome!