Business Intelligence Programme Management: Optimising Time to Market with a Manageable Architecture

In this series of three posts, I will address some typical aspects of programme management for decision support. The first post will define programme and project management in Business Intelligence (BI) and its relationship with IT architecture.

The second post will deal with some issues in governance, the tensions that arise between business and IT and how to deal with them.

The third post will propose a new way of designing systems for both transaction  and decision support to improve the organisation’s effectiveness further.

A project holds the middle ground between routine and improvisation to produce a product

This is the essence of project management: managing unknown and unfamiliar risks and uncertainties using experience with proxies and lessons learnt to produce something within a time frame, a budget and within a certain quality range while delivering correct management information for the steering committee about the progress and the resources needed.

In Business Intelligence, this product orientation sometimes leads to overly focussing on the deliverables while ignoring valuable opportunities along the way. Many BI projects are described in terms of delivering x reports on y KPIs or delivering OLAP cubes and reports on x sources to describe what I call “stocks and flows” of business processes. I am not arguing against this approach but project steering committees should be aware that a tunnel vision can be a costly liability in BI.

Already in 1994, Séan Kelly (not the Irish cyclist nor the politician, but the datawarehouse manager from Eireann Telecom) stated that the business user can’t formulate correct requirements at the start of the BI project and changes these requirements during the project’s lifecycle. Twenty years later, many BI project teams haven’t learned anything from Kelly’s observation. Either they stick to the initial product description or they reiterate the analysis –design –build cycle at extra cost and throughput time. The major reasons are a lack of BI maturity in the organisation and the lack of an overall BI programme vision, at best an incomplete one.  In the next section I describe BI programme management and how this contributes to more effective BI projects.

Projects deliver products, programmes have outcomes

There is certainly some truth in this dictum on one condition: BI programme management should focus on favourable outcomes for decision making. The BI programmes can focus on improving decision making processes, on information objects like customer, product, channel or on broader outcomes like knowledge sharing, improved positioning, improved competitive capabilities etc.

The decision on the programme scope is not trivial. Sometimes programmes can define too broad an objective for the organisation’s maturity. Not everyone sees the relationship between a new column store and improved competitive positioning.

Some authors and practitioners see a BI programme as a collection of projects, a higher hierarchical level from tasks via projects to programmes. I strongly disagree. In my experience, BI programmes have links with other programmes on HR, IT infrastructure, marketing and sales, and other functional or strategic endeavours to produce a better outcome for the BI project portfolio.

The matrix below shows a few examples of how BI programme management interacts with other business programmes in the organisation.

Functional strategies and programmes	BI Programme interaction	Dependencies outside the BI programme
Marketing: improving customer knowledge	CRM and ERP systems deliver data to the customer analytics programme	The organisational question of customer ownership needs to be addressed when multiple divisions deal with the same customer.
Marketing: improving direct communication response rates	A customer MDM programme is initiated to improve data quality for BI and CRM processes	The customer logging processes,( e.g. in the contact centre) needs improvement initiatives
Finance: reducing days receivable outstanding	Within the customer analytics programme, a data mining project is initiated to predict payment behaviour	The invoicing process needs updating
HR: reducing absenteeism	HR and ERP systems deliver data about potential influencers on absenteeism as well as customer analytics to examine the impact of customer behaviour on absenteeism	The organisation’s management style needs adjustment for the new economy

Table 1 How BI Programmes interact with other business programmes

From these simple examples you can see clearly that the outcome of a BI programme affects other programmes and in its turn is affected by other programmes. In the case of BI projects we can also distinguish dependencies with other projects but these dependencies are usually smaller in scope and easier to manage in the steering committee.  You will probably recognize some of these quotes:

I can’t get the data from the HR department, it’s classified information,

They say I have to take the matter to the architecture board,

The infrastructure needs upgrading,

The license negotiations are slowing down the project, etc…

Now that the link with external programmes is clear it is high time to look what is inside a BI programme. This is where the link with architecture becomes clear.

BI programmes should have an overarching view, vision, business case and target setting for the principal contributors to better decision making.

In Kimball terms: the conformed dimensions, in Linstedt terms: the data vault’s hubs and satellites. No matter which solution approach you choose, a vision on the principal information objects is quintessential to the success of individual BI projects. 

From Information Objects to BI Architecture

Thinking about information objects like customer, channel, product, location, is thinking about the way they are created, stored, updated and deleted in the various information systems of the organisation. It also relates to the business processes using these information objects to produce context for transaction registrations like information request, complaint, order, payment etc..

Thus, answering the what, where, why and how questions as the Zachman Framework does is talking architecture. The illustration below is a classical BI situation. With the advent of complex event processing and big data technology on Hadoop, things are changing but for 99% of the organisations, classical BI is still the modus operandi.

Figure 1 A generic architecture of contemporary information systems: transaction systems and decision support systems are separated systems where information objects pass from the transaction systems to the decision support systems via an Extract Transform and Load Process.

Some comments with the above schema in Archimate modelling language:

Business drivers such as “end to end support for the order to pay process” define various business processes which in turn are supported by transaction registration systems. These systems create transaction lines like order, order confirmation, bill of material, manufacturing data, shipping bill, delivery note, customer receipt registration, invoice,  customer payment registration etc…

All these transactions relate to information objects like date and time, product, customer, shipment mode, etc..

Via the Extract Transform and Load process these objects are scrubbed, normalised and made ready for publishing in analytical environments and reports. That’s when they become decisional data objects: facts and dimensions are always the end result, no matter what intermediate storage you use: a data vault, an anchor model or an enterprise data warehouse in the third normal form. The facts are the measures in the reports and the dimensions are the perspectives on these measures. Usually you read the facts per dimension, i.e. the sales per region, per outlet, per account manager,…

In data mining projects the facts and dimensions will be flattened in a matrix for offline analysis and combined with semi structured and unstructured data in Hadoop files systems. In streaming analytics temporary snapshots are compared with the scoring model which is derived from the facts and dimensions as well as semi structured and unstructured data in Hadoop file systems.

With new Big Data technologies, new architectures will emerge but that is for another post.

Suffice it to conclude that managing enterprise wide facts and dimensions as well as semi structured and unstructured data is both the task of programme management and BI architecture.

BI Architecture and Programme Management: See the Picture

Programme managers need to see the entire picture to set priorities, look for synergy between projects and initiate data management projects to fill the gaps between the BI projects as required by the business.  An example can make this clear.

Let’s take the first example from table 1: “Improving Customer Knowledge” driving a programme to consolidate all static and dynamic information about customers and their behaviour.

Conferring with the enterprise architecture board, the programme manager discovers that the geographical coordinates of each customer are valuable information for the logistics department to optimise delivery schedules. Though it is outside the scope of marketing, the programme manager will initiate a project to add geolocation data to the customer dimension. Later in the process, the marketing manager discovers the potential of geomarketing.

Of course, the interaction can also work the other way around: the BI architecture review board evaluates programmes and projects and readjusts priorities and project scope on the basis of availability and cost of data capture. Sometimes the replacement or adjustment of a source system can impact the BI programme heavily. It always boils down to defining a business case that sticks. Excuse me for hitting the nail over and over but it all starts with business analysis for business intelligence that sets the scene. Too many projects and programmes have failed in BI because of a gung ho approach of the designers and builders who cannot wait till the specs are thought through after thorough analysis of the strategy process and the data landscape.