This third
post in a series of three on BI programme management looks at a new way of
designing systems for both transaction and decision support to improve
the organisation’s effectiveness further. I will examine the concept of BI
architecture further and give hints of how BI programme management can evolve
towards an ideal architecture which merges transaction and decision support
systems in a powerful ensemble, ready for the new economic challenges.
I propose
an “Idealtyp” knowing that no existing organisation can achieve this in less
than a decade for reasons like sunk cost fallacies, the dialectics of progress
and simply resistance to change.
But new
organisations and innovators who can make the change will notice that the
rewards of this approach are immense. They will combine architectural rigidity
with business agility and improve their competitive power with an order of
magnitude.
Why a BI Architecture is Necessary
I am a fan
of Max Weber’s definition of “Idealtyp”[i],
which has direct links with architecture in information technology. BI
architecture is an abstraction of reality, and as such an instrument to better
understand a complex organisation of hardware, network topologies, software, data
objects, business processes, key people and organisational units. All these
components interact in –what appears to outsiders- in a chaotic way. An
architectural framework brings order to the chaos and provides meaning to all
the contributors to the system.
Architecture
is used as a benchmark, a to be situation by which the present state of nature
can be measured. It is a more crisp and more manageable concept than CMM-like
models which express maturity sometimes in rather esoteric terms. For a quick
scan, this will do but for in-depth managing of the above mentioned BI assets,
an architectural framework is better for BI environments.
CMM Level
|
BI symptoms
|
Principal risks
|
Initial
|
A serious case of “spreadsheetitis”: every
decision maker has its own set of spreadsheet files to support him in his
battles with the other owners of spreadsheets. Everyday tugs of war over who
has the correct figures.
|
Your project may never take off because of
political infighting and if it does, there will be a pressing need for change
management of the highest quality and huge efforts will have to be invested
in adoption tracks.
|
Repeatable
|
The organisation uses some form of project
management, in most cases inherited or even a carbon a copy of systems or
application development
|
The project management method may
be totally inadequate for a BI project leading to expensive rework and
potential project failure in case everybody remains on his position.
|
Defined
|
The organisation has a standard
procedure for the production of certified reports. These can connect with one
or more source systems in a standardised way: direct connection to the source
tables, import of flat files, or some form of a data warehouse.
|
Resistance to change.
This depends on the way the
organisation has implemented the data warehouse concept and how reversible
the previous efforts are in a migration scenario.
|
Managed
|
The development processes are
standardised and monitored using key performance indicators and a PDCA cycle.
|
The iterative and explorative
approach of BI project management may frighten the waterfall and RAD fans in
the organisation. Make sure you communicate well about the specifics of a BI
development track.
|
Optimising
|
The development processes only need
fine-tuning.
|
Analysis
paralysis and infighting over details may hamper the project’s progress.
|
Table 2 Example of the BI version of the Capability Maturity Model as described in Business Analysis for Business Intelligence on page 202. In the book, it is positioned as a tool to help the BA with identifying broad project management issues
No matter what industry you look at, market leaders fulfil their basic marketing promise: provide stability, predictable behaviour and a very high degree of CYA (google it) to the buyer. But that doesn’t mean the purchase decision is the best possible decision for future use. Market leaders in IT are also very keen on “providing” vendor lock-in, disallowing the client to adapt to changing requirements.
Why this "Idealtyp" is not Easy to Achieve
Proposing an
ideal BI architecture is one thing, achieving it, another. I will only mention
three serious roadblocks on the path towards this ideal BI architecture that
unifies transaction systems and decision support systems: the sunk cost
fallacy, the dialectics of progress and resistance to change.
The sunk cost fallacy is a powerful driver
in maintaining the status quo; organisations suffering from this irrational
behaviour consider they have invested so much effort, money, hardware,
training, user acceptance and other irretrievable costs that they should
continue to throw good money at bad money. And sometimes the problem is compounded when
the costs were spent on technology from market leaders.
No one ever got fired for buying…
(fill in any market leader’s name)
No matter what industry you look at, market leaders fulfil their basic marketing promise: provide stability, predictable behaviour and a very high degree of CYA (google it) to the buyer. But that doesn’t mean the purchase decision is the best possible decision for future use. Market leaders in IT are also very keen on “providing” vendor lock-in, disallowing the client to adapt to changing requirements.
As a
footnote: today, buyers are more looking at the market cap or the private
equity of the Big Data technology providers than at their actual technical
performance and their fit with the organisation’s requirements. Yes, people
keep making the same mistakes over and over…
At the
other end of the spectrum are the dialectics
of progress: this law was discovered
by the Dutch journalist Jan Romein who noticed that gas lights were still used
in London when other European capitals already used electricity. This law suggests-and I quote an article on
Wikipedia- that making progress in a particular area often creates circumstances
in which stimuli are lacking to strive for further progress. This results in
the individual or group that started out ahead eventually being overtaken by
others. In the terminology of the law, the head start, initially an advantage,
subsequently becomes a handicap.
An explanation for why the phenomenon occurs is
that when a society dedicates itself to certain standards, and those standards
change, it is harder for them to adapt. Conversely, a society that has not
committed itself yet will not have this problem. Thus, a society that at one
point has a head start over other societies, may, at a later time, be stuck
with obsolete technology or ideas that get in the way of further progress. One
consequence of this is that what is considered to be the state of the art in a
certain field can be seen as "jumping" from place to place, as each
leader soon becomes a victim of the handicap.
(From: https://en.wikipedia.org/wiki/Law_of_the_handicap_of_a_head_start)
(From: https://en.wikipedia.org/wiki/Law_of_the_handicap_of_a_head_start)
As always, resistance to change plays its role.
New tools and new architectures require new skills to be trained, new ways of
working to adopt and if one human species has trouble adapting to new
technologies it is… the tech people. I can produce COBOL programmers who will
explain to you that COBOL is good enough for object oriented programming or IMS
specialists who see nothing new in the Big Data phenomenon…
What is BI Architecture?
Here’s architecture
explained in an image. Imagine Christopher Wren would have disposed of modern
building technologies. Then either the cathedral, based on the architecture “as
is” would have looked completely different, with higher arches, bigger windows,
etc… Or,… the architecture could have evolved as modern technology would have
influenced Wren’s vision on buildings.
Exactly
this is what happens in BI architecture and BI programme management.
 |
Figure 5
On the left: architecture, right: a realisation of architecture as illustrated
by Wren’s Saint-Paul’s Cathedral
|
Architecture
descriptions are formal descriptions of an information system, organized in a
way:
- that supports reasoning about
the structural and behavioural properties of the system and its
evolution.
- These descriptions define the
components or building blocks that make up the overall information
system, and
- They provide a plan from which
products can be procured, and subsystems developed,
- that will work together to
implement the overall system.
- It thus enables you to manage
your overall IT investment in a way that meets the needs of your
business.
It is also
the interaction between structure, which is requirements based, and principles
applicable to any component of the structure.
What is the Function of BI Architecture?
BI Architecture
should reflect how the BI requirements are realized by services, processes, and
software applications in the day-to-day operations. Therefore, the quality of
the architecture is largely determined by the ability to capture and analyse
the relevant goals and requirements, the extent to which they can be realized
by the architecture, and the ease with which goal and requirements can be
changed.
 |
Figure 6
The Open Group Architecture Framework puts requirements management at the
centre of the lifecycle management. The connection with business analysis for
business intelligence is obvious.
|
Reality Check: the Two Worlds of Doing and Thinking
Now we have
established a common view on BI architecture and programme management, it is
time to address the murky reality of everyday practice.
Although
Frederick Taylor and Henri Fayol’s ideas of separation between doing and
thinking have been proven inadequate for modern organisations, our information
systems still reflect these early 20th Century paradigms. You have the
transaction systems where the scope is simply: execute one step after another
in one business process and make sure you comply with the requirements of the
system. This is the world of doing and not thinking. Separated from the world
of doing is the world of thinking and not doing: decision support systems. The
business looks at reports, cubes and analytical results extracted from
transaction and external data and then makes decisions which the doers can
execute.
What if the
new economy were changing all this in a rapid pace? What if doing and thinking
came together in one flow? That’s exactly what the Internet is creating, and I
am afraid the majority of organisations are simply not ready for this
(r)evolution. Already in 1999, Bill Gates and Collins Hemingway[ii]
wrote about empowering people in the digital age when they gave us the
following business lessons:
- q The more line workers understand the inner workings of production systems, the more intelligently they can run those systems.
- q Real-time data on production systems enables you to schedule maintenance before something breaks.
- q Tying compensation to improved quality will work only with real-time feedback of quality problems.
- q Task workers will go away. Their jobs will be automated or combined into bigger tasks requiring knowledge work.
- q Look into how portable devices and wireless networks can extend your information systems into the factory, warehouse and other areas.
I am afraid
this advice still needs implementation in many organisations. The good news is
that contemporary technologies can support the integration of doing and
thinking. But it will require new architectures, new organisational and
technological skills to reap maximum benefits from the technology.
The major and most relevant BI
programme management decision criterion will be the answer to the question: “Which
quality data yield the highest return in terms of competitive advantage?”
Bringing IT Together: Design from the Data
What if we
considered business processes as something that can change in 24 hours if the
customer or the supplier wants it? Or if competitive pressure forces us to
change the process? What if information systems would have no problem
supporting changing business processes because the true cornerstone, surviving
any business process is data? This could be a real game changer for industries
that still consider data as a product of a business process instead of the
objective of that process.
The schema
below describes a generic architecture integrating transaction and decision
support systems in one architectural vision. Let’s read it from left to right.
Any
organisation has a number of business drivers, for example as described by
Michael Porter’s generic strategies: be the cost leader, differentiate from the
competition or focus on a niche. Parallel with the business drivers are
decision making motives such as: “I want complete customer and product insight”
and finally, the less concrete but very present knowledge discovery driver to
make sure organisations are always in the lookout for unpredictable changes in
the competitive environment. These three drivers define a number of business
objects, both static and dynamic. And these entities can be endogenous to the
organisation (like customer, channel, product, etc..) or they can be external
like weather data, currency data, etc…. These business objects need to be
translated into data objects suitable for transaction and decision support
 |
Figure 7 This is the (condensed) target architecture of an
integrated “Big Data Warehouse”: combining batch and stream processing
using low latency for operational intelligence and aggregate data for tactical
and strategic decision making. Built from the ground up using data in
stead of business processes as the analytic cornerstone.
|
Conclusion:
an integrated view on transactions and decision making will improve BI
programme management supported by this architectural vision. The major and most
relevant BI programme management decision criterion will be the answer to the
question: “Which quality data yield the highest return in terms of competitive advantage?”
And thus, which project (whether on the transaction or decision support systems
need the highest priority in allocation of resources?
[i]
According to the excellent website http://plato.stanford.edu/entries/weber/ this is the best description of Max Weber’s
definition:
“The methodology of “ideal type” (Idealtypus) is
another testimony to such a broadly ethical intention of Weber. According to
Weber's definition, “an ideal type is formed by the one-sided accentuation of
one or more points of view” according to which “concrete individual phenomena
… are arranged into a unified analytical construct” (Gedankenbild); in
its purely fictional nature, it is a methodological “utopia [that] cannot be
found empirically anywhere in reality”. Keenly aware of
its fictional nature, the ideal type never seeks to claim its validity in terms
of a reproduction of or a correspondence with reality. Its validity can be
ascertained only in terms of adequacy, which is too conveniently ignored by the
proponents of positivism. This does not mean, however, that objectivity,
limited as it is, can be gained by “weighing the various evaluations against
one another and making a ‘statesman-like’ compromise among them”, which is often proposed as a solution by those sharing Weber's
kind of methodological perspectivism. Such a practice, which Weber calls
“syncretism,” is not only impossible but also unethical, for it avoids “the
practical duty to stand up for our own ideals”.”
What is less known is that Weber used the concept also
in decision making theory when he analysed the outcome of the Battle of
Köninggratz, where Von Moltke defeated the Austrian-Bavarian coalition against
Prussia and its allies in 1866, an important phase in the unification of Germany.
[ii] “Business at the Speed of Thought” Bill Gates and Collins Hemingway, Penguin Books, London England, 1999 pp 293 -294
