Relevance Lost 2.0

When Robert Kaplan and Thomas Johnson wrote about the fall of management accounting systems and pleaded for new perspectives and approaches to strategic process control they wrote a most relevant statement that still stands today. But as organisations are implementing performance measurement systems as applications of their business intelligence infrastructure, news risks of losing the most relevant part of strategic management arise. There is a clear need for a new, extended approach to the balanced scorecard (BSC). If you wonder what this may be, then read more.

In this article I describe the generic strategy process, compare this with the performance management process, how it is implemented in some organisations and analyze the aspects of this process related to the organization’s leadership style and thus the consequences for the balanced scorecard’s effectiveness.

The Strategy Process: Formation and Formulation

Every organisation has its own way of forming and formulating a strategy. It is in the organizational DNA: the way strategic objectives are grounded on a clear problem statement, SWOT analysis, competitive analysis, etc… and evaluated, proper to that same DNA.

Fig 1. The Strategy Process The strategic objectives entail the position of the company for mid- and short term, wanted behavior from clients, associates and partners. These objectives are then dissected, evaluated, scored and appreciated according to the organization’s values and beliefs, the organizational culture and past experience. 

The last phase is the decision phase where the objectives are translated into concrete actions and measurable products and outcomes of these actions.

As the illustration shows, the strategy formation process is usually done in four process steps:

• definition, 
• dissection, 
• evaluation and 
• decision

In the definition process, the strategic problems and opportunities are described and expressed in statements like: “We are not well positioned in the high end market”. Input for this process step is SWOT analysis, as well as analysis of the competition and the customer base. 
In the dissection process, expressions like “We are not well positioned…” are set in context and translated into critical success factors, e.g. “In order to improve our position in the high end market we need to upgrade our image, educate the sales force, identify more high end customers.”
The eva luation process is mainly concerned with prioritizing the dissected elements and looking for causal relationships between these elements to prepare a cohesive and consistent strategic plan which can be communicated to all parties concerned.
Finally in the decision process, the prioritized and feasible targets and ways to achieve these targets with global action plans are chosen. Some targets are based on decisions with tangible and measurable results and others are based on long term decisions with immaterial and almost immeasurable outcomes. 
The following decision points are ranked from very concrete and “hard” targets to more “soft” and less quantifiable decisions:

• which critical performance indicators will receive priority in the action plans and budget allocation,
• which actions are needed in the market place,
• which organizational behavior needs adjusting
• what needs to be improved in the organization’s culture

In the strategy formulation process steps, communication exercises will adjust these elements to match them with the various target groups: associates, clients, suppliers, shareholders, government officials, press, etc…

The Degrees of Collaboration in the Strategy Process

If we imagine a continuum in leadership styles, ranging from autocratic to democratic leadership, the four strategy formation process steps will be subject to shared inputs, transformation and outputs on a scale of zero to hundred percent.

In an autocratic leadership all four process steps are in the hands of the leader. Only the leader decides what is important in the environmental scan, what the objectives are, how the actions  are prioritized. Some autocratic leaders have problems formulating the strategic plan as they still hold up the adage “knowledge is power”. What they gain on control may get lost in the execution phase when their subordinates try to interpret their ambiguous communication.

Long time ago, I studied the biographies of dictators (which names I won’t mention because they deserve to be forgotten) and it struck me that often they communicated very vaguely about their strategic vision, priorities, objectives and the way to accomplish these objectives. Zealous and ambitious subordinates would then translate these cryptic messages into complete (and often horrific) action plans which would then be meticulously and ruthlessly executed. After which the dictator either rewarded the zealot or had him sent in exile because he became too popular or was a liability for the regime to the outside world.  Nevertheless, autocratic leaders can be highly successful in process industries, retail, service organisations and this leadership style emerges everywhere there is a crisis and fast response times are more important than a well pondered decision making process, balancing all market factors, interests and wishes in the organisation.

 In the democratic leadership all four process steps may be in the hands of the entire team. Lengthy discussion and negotiation may require a lot of resources but the upside is that everyone in the organisation is on the same page and the execution phase has less need for control and clear instructions as the organisation members act in a more autonomous way, responding faster to changes on the terrain. Organisations of professionals prefer democratic leadership as knowledge and competence are far more important than rank and power and the status that go with it.

Finally in mixed forms, the definition process may be initiated by the strategic apex, shared with the ranks and business analysts may be called in to dissect the statements into  manageable chunks.

The evaluation and the decision process, depending on the level of democratic leadership, may be done by the leader, a management team or a management team extended with staff members and analysts.

Today, as organisations become flatter and more democratic leadership styles are proper to new industries, there is a need for faster feedback loops to combine the advantages of autocratic leadership with the responsiveness of democratic organisations.

The Performance Management Process

The strategy process, seen from a performance management perspective, is a machine-like approach to define, monitor and manage the actions as defined in the strategic plan.

Let’s see how the strategy process is broken down into the performance management process. The performance management process breaks down the strategy formation process into smaller chunks to decompose the formation (definition, dissection, evaluation and decision) into nine steps.

This is a top down exercise:

·        Analyze the situation (SWOT, competition,…) (Definition Process)

·        Determine the objectives after the analysis (Definition Process)

·        Define the critical success factors (CSF) (Definition & Dissection Process)

·        Derive the critical performance indicators (CPI) from these CSFs (Definition & Dissection Process)

·        Mapping the CPIs on the organizational units down to the individual associate (Dissection & Evaluation Phase)

·        Adapting the HRM policies to these mappings (if trade unions allow, of course) (Evaluation Process)

·        monitor, manage and readjust the CPIs (Execution process)

·        monitor, manage and readjust the CSFs (Execution process)

·        Adapt the objectives to the new SWOT results (Execution process)

Remark how implicit the decision process is embedded in the dissection and evaluation process steps of the performance management approach.

Strategy automation?

In the past ten years I have worked on IT-support for balanced scorecards (BSC) in a university, a bank, an insurance company and a manufacturing company. In all of these cases, a poignant conclusion was unavoidable: “If all you use is a hammer, everything starts looking like a nail”. The ICT tool became a substitute for the strategy process and forced a freeze on the organisation. Let me explain this.

The BSC was used as an instrument to implement a top down strategic governance of the organisation. The strategy decomposition as described above is then modeled in the ICT tool creating links and correlations between the various CSFs and CPIs. Identifying these cause and effect chains is not a trivial matter. If sales go south, all sorts of explanations may present themselves to the organisation. E.g.: Are lower sales due to:

·        … lower consumer confidence?

·        … a competitive move?

·        … a government announcement?

·        … simple seasonality or a gradual shift in seasonality?

·        … the weather?

·        … all of the above?

Or can the sales slump be explained by a factor we can influence like the number of sales training hours received by the sales reps? But then the question arises if there is a correlation between the amount and quality of the training received and employee satisfaction? Or is it the other way around:  because our employees are not very satisfied with their job, they respond poorly to the training received[i]?

But that is not all. As we all know, strategic management is about adapting to the environment. If the ICT tool does not capture  the environmental change either bottom up or top down, then what? There is also another side of the coin: if the strategic decomposition leads to individual targets, personal development plans and other HRM tools how does this affect the flexibility of the organisation to adapt to change?

People act accordingly to the incentives from management: either they integrate the CPIs in their work planning and their approach to the job or they look for ways to beat the system. I remember sales people holding back order forms for a yearly publication to “smoothen” the CPI measures of bookings per month since management did not take seasonality into account when the performance indicator was defined.

Strategy Dialectics Are the Way Forward

It is clear that the latter is unwanted behavior but those who conform with the system should be rewarded, shouldn’t they?

The answer is an ambiguous “Yes and No”. “Yes” if their response, steered and governed by the performance indicator, is in sync with customer demand. And “No” if this is not the case. Needless to add that any strategy which is not sanctioned by your customers is not worth the paper it is written on.

But how are the designer, the monitor and the manager of the balanced scorecard to know this? Henry Mintzberg (1994)[ii] makes the distinction between intended and emergent strategies and the way I see it and experienced it, the balanced scorecard is an almost perfect tool for managing intended strategies. It uses a negative feedback loop, just like a thermostat. And just like a thermostat it sometimes oversees the efforts needed to keep everybody in line with the intended strategy. So people who don’t meet their targets are stimulated to do so or they are made redundant if they are not likely to comply with the desired behavior. But as John Lennon so rightly said “life is what happens while you are making other plans”. Management may have misinterpreted the signals from the environment or changes in the market may be unnoticed by management. In that case,  emergent strategies may provide the answers to these situations as there is some form of “wisdom of the crowds” in the collective response from front office workers and anyone else who is in contact with customers, competitors, prospects, suppliers, researchers and government officials, to name a few. To capture these emergent strategies, the system needs to provide positive feedback loops to reinforce unplanned but successful behavior, even when it is non compliant with the intended strategy. In other words, if top management makes a mistake, it will get noticed in three to five years but if the front office worker makes a mistake, the organisation has an acute problem.

This calls for a special form of management, allowing dissidence in the ranks and considering experimenters and contrarians as assets instead of a liability. “Is this May 1968 all over again, when it was forbidden to forbid?”, I hear you say. No, thanks.

But imagine an organisation form where the exchange between the hierarchy and the ranks is formalized, open, unbiased and where everyone’s fact findings and opinions are accessible to everyone for discussion, refining and leading to decisions and actions.

Imagine a special form of knowledge management which goes further than a glorified chat room and text mining.

Imagine a system supporting both bottom up and top down strategy processes, using the collective wisdom of the entire organisation. Technology may be able to design and build such a system but if management is not prepared to adapt its ways of developing, forming and formulating strategies then the developers needn’t bother.

Knowledge Management and Performance Measurement Systems in Modern Organisations

Remember the initial point I was making: intended strategy is only a partial explanation of the realized strategy because emergent or grassroots strategies contribute to-or reduce- the results of the intended strategy. Since no entrepreneur or manager likes to be only partially in control, we need a new approach to the balanced scorecard implementation. Maybe that won’t be enough, maybe we need to extend the scorecard’s toolset.

What if the exchange process were more important than the results of it?

What if the true outcome of the dialectic strategy process were -other than a plan with measurable results:

·        enhanced motivation because people see the context, the bigger picture and have  contributed to it,

·        a shared vision and sense of direction that enhances group cohesion,

·        a higher level of entropy, turning each individual into autonomous decision making entities without the usual chaotic side effects,

·        increased responsiveness to changing conditions or unexpected phenomena in the market?

  

What if the strategy process became a strategy dialogue?

What if the system could capture the dialogues between the workers and:

•                 middle- and top management,

•                 the customers,

•                 the suppliers,

•                 consultants,

•                 academics,

•                 opinion leaders,

•                 government officials,

•                 the data warehouse,

•                 the external information sources?

What if this dialogue were supported by a tool requiring almost no extra effort from the organisation?

Let’s examine the actions people perform in an office which are –often without knowing- valuable strategic information bits and are already captured partly or wholly by the existing systems.

Searches on the Intra- and Internet,

Consulting information providers ranging from Wikipedia to academic and government sources,

Sending and receiving e-mails,

Creating and reading documents like meeting notes, documentation, process descriptions, …

Handling customer complaints

Dunning customers,

Online and offline meetings, chats,

Analysis and decision making,

etc…

All these activities leave traces in some or another information system. What if you could combine the most relevant words and constructs into input for your strategic plan, supported by a balanced scorecard approach but avoiding a rigid approach to the strategic process management? Let me have a go at specifying such a system and check if the technology is already available.

Performance Measurement Systems 2.0., a Functional Specification

The short description of this system is: ”A Collaborative Strategy Process Manager”   The architectural view is visualized in the schema below.

Fig. 2. The Core Architecture of a Collaborative Strategy Process Manager The architectural picture consists of three interconnected pillars: strategy, individual development and knowledge support. For simplicity reasons we leave aspects like servers, APIs and user interfaces out of the schema. Only the relevant functional blocks are listed. Remark that the planning and execution of the strategy is not in scope. Tools like ERP and CRM can provide the necessary support for that part.

The Organizational Strategy Management Process: this is basically the support for the balanced scorecard with a link to the personal development plans of the people needed to execute the strategy. We refer to vendors like QPR, SAS, and others to discover the features of a balanced scorecard software. But the link to personal development plans (PDP) needs to be established. Imagine this PDP as a database with relevant and “nice to have” competence development plans which are maintained during appraisal and evaluation interviews. If the HRM application can make the link with salary scales, a proper analysis can match the desired competences with the future wage cost trend.

The Individual Development Support: this is a personal balanced scorecard where the interconnections between interests, competences, knowledge and the track record inside and outside the organisation are managed. Imagine a kind of personal LinkedIn with extra depth in the competence area. Instead of generic labels like “marketing” one would find hierarchies like “Marketing > Market Research>Qualitative Market Research>Focus Groups>Brand Experience”

The Knowledge Support System: this pillar is not readily available of the shelve, but some components are.

The first block Search Engine Logs and Ratings bundles the information search behavior, the rating of the results but also the ratings of colleagues search results, and all other communication and processed information. It also includes ratings for decisions to be made by a group or an individual be it multicriteria analysis, simple voting, rule based decision making or more complex algorithms like ELECTRE. Bundling the information search and decision behavior can yield interesting results for the knowledge management team answering questions like “What information did he/she (or didn’t!) look for before making a decision?”

The Object Database is the engine behind the object aware user interface, suggesting hyperlinks whenever users integrate (potential) knowledge objects like information sources, notions, definitions, persons, etc… in their communication. This forces users to be clear about what they communicate and to what ontologies[i] their concepts relate to. These objects, ready for the object aware user interface are structured and edited by a knowledge manager or by the group, depending on the configuration which supports both autocratic and democratic environments. But the object database will also act as a repository for unstructured data which can then be presented to the group or the knowledge manager by “emerging publishers”.

The Object Database also disposes of easy configurable agents which respond to events or trigger events themselves, e.g. “Present the weekly list of most used ontologies”.

The Knowledge Modeling System is where ontologies, learning blocks and documentation blocks are created and managed. These learning blocks and documentation blocks are complete sentences or groups of sentences whether or not combined with illustrations to create the basic material for documentation like ISO process descriptions and procedures, help files and learning material for distance learning or learning on the job.

Finally,  the E-Learning System is where the previous material is used in learning paths, documentation maps and presented in a presentation layer which can be simple text and image, sound and/or video. The material can be used on purpose or pop up spontaneously whenever users struggle with knowledge gaps.

The three connections: the first connection is between the organizational strategy management and individual development tools. With this link, important questions can be answered and reality checks become possible.  A sample of these questions and checks:

                     •                 “Do we have the competences in house to deliver the desired actions?”

•                 “How big is the knowledge gap we need to close”

•                 “Where are biggest obstacles for change management?“

•                 “What unused competences suggest opportunities for new strategic directions?”

The second connection, between individual development and knowledge support tools makes knowledge management more manageable by delivering information to questions like:

• “What knowledge objects are most used by individuals, groups, teams, divisions,… ?”
• “When and where does who contribute to the development or the extension of knowledge objects?”
• “What e-courses have been successfully taken by whom?”

The third connection, between the strategy management support and the knowledge support challenges assumptions and probes for answers to questions like:

·        “Do we have the necessary capacity for the personal development plans?”
·        “Have we made the CSFs and KPIs sufficiently operational for the workforce to understand, adopt and apply them?”
·        “What level of comprehension of the strategy matches with a certain level of compliance to the proposed KPIs?”

·        “What new information can influence adjustments to the initial strategy?“What new information can influence adjustments to the initial strategy?”

If the “What” and the “Why” are implemented properly, the “Where” and the “How” will become easier to manage in an adaptive way because management control 2.0. will increase self control with individuals and groups but leave enough room for new initiatives to respond better to consumer demand.

Fig. 3 The Strategy Execution Process As the illustration suggests, there are two major directions to take when executing a strategic plan: changing the rules of the game or the organisational behaviour in case the results are disappointing or optimize the existing, successful strategy.  The “how” to this “what” can be both shock therapy or an incremental “frog in the pot of tepid water” approach. ultimately it will lead to managing the organisational competences, be they individual or group competences. The “where” is in projects or in processes, i.e. in new ventures or in routine things people do in the organisation. The “why” is not in this picture as it is supported by the collaborative strategy process manager as described above.

Conclusion

Strategy management is a slightly more complex phenomenon than the cybernetic view some scholars and managers have. An organization is a living thing, the environment is something even the biggest organisations can’t control (unless they are in a socialist island republic). Therefore, adaptive strategy management is the way forward.

Strategy management 2.0. will be adaptive or it will become obsolete in a flattened society, where successful organisations in the new economy have exchanged the hierarchical, top down, cybernetic management paradigm for a customer centric, responsive and adaptive organisation where people are motivated, empowered and share a clear vision, a sense of purpose and understand the general  direction the organisation is heading. Only that way, these organisations can face the challenges of a mobile, fragmented and volatile generation Z and build a sustainable business.

---

[i] I use the definition from Tom Gruber from Stanford University: “In the context of knowledge sharing, I use the term ontology to mean a specification of a conceptualization. That is, an ontology is a description (like a formal specification of a program) of the concepts and relationships that can exist for an agent or a community of agents. This definition is consistent with the usage of ontology as set-of-concept-definitions, but more general. And it is certainly a different sense of the word than its use in philosophy.” From: T. R. Gruber. “A translation approach to portable ontologies.” Knowledge Acquisition, 5(2):199-220, 1993

---

[i] In many cases, this cause-effect identification process is a matter of preparing the strategy formulation phase which has the implicit message: “This is how we see things” or a misused word in the business jargon: the often overstated “paradigms”.

[ii] From: Mintzberg, Henry: “The Rise and Fall of Strategic Planning” pp. 24-27, The Free Press 1994.

About Ends and Means, or Beginning and Ends...

It has been a while since I published anything on this blog. But after having been confronted with organisations that –from an analytics point of view- live in the pre-industrial era, I need to get a few things off my chest.

In these organisations (and they aren’t the smallest ones)  ends and means are mixed up, and ends are positioned as the beginning of Business Intelligence. Let me explain the situation.

Ends are the beginning

A metaphor for a critical look at reporting requirements is like watching heavy drift ice 

and wondering whether it’s coming from a land based glacier or from an iceberg...

Business users formulate their requirements in terms of reports. That’s OK, as long as someone, an analyst, an architect or even a data modeller understands this is not the end of the matter, on the contrary.

Yet too many information silos have been created when this rule is ignored. If an organisation considers report requirements as the start of a BI project they are skipping at least the following questions and the steps needed to produce a meaningful analytics landscape that can stand the test of time:

• New information silos emerge with an end-to-end infrastructure to answer a few specific business questions leaving opportunities for a richer information centre unexplored.
• The cost per report becomes prohibitive. Unless you think € 60.000 to create one (1) report is a cinch…
• Since the same data elements run the risk of being used in various data base schemas, the extract and load processes pay a daily price in terms of performance and processing cost.

Ends and means are mixed up

A report is the result of an analytical process, combining data for activities like variance analysis, trend analysis, optimisation exercises, etc.. As such it is a means to support decision making; so rather than accepting the report requirements as such, some reverse engineering is advised:

What are the decisions to be made for the various managerial levels, based on these report requirements?

You  may wonder why this obvious question needs to be asked but be advised, some reports are the equivalent of a news report. The requestor might just want to know about what happens without ever drawing any conclusions let alone linking any consequences to the data presented.

What are the control points needed by the controller to verify aspects of the operations and their link to financial results?

Asking this question almost always leads to extending the scope of the requirements. Controllers like to match data from various sources to make sure the financial reports reflect the actual situation.

What are the future options, potential requirements and / or possibilities of the required enhanced with the available data in the sources?

This exercise is needed to discover analytical opportunities which may not be taken at the moment for a number of reasons like: insufficient historical data, lacking analytical skills to come up with meaningful results… But that must not stop the design from taking the data in scope from the start. Adding the data in a later stage will come at a far greater cost than the cost of the scope extension.

What is the basic information infrastructure to facilitate the above? I.e. what is the target model?

A Star schema is the ideal communication platform between business and tech people.

Whatever modelling language you use, whatever technology you use (virtualisation, in memory analytics, appliances, etc…) in the end the front end tool will build a star schema. So take the time to build a logical data star schema model that  can be understood by both technical people and business managers.

What is the latency and the history needed per decision making horizon?

The latency question deals with a multitude of aspects and can take you to places you weren’t expecting when you were briefed about report requirements. As a project manager I’d advise you to handle with care as the scope may become unmanageable. Stuff like (near) real-time analytics, in database analytics, triple store extensions to the data warehouse, complex event processing mixing textual information with numerical measures… But as an analyst I’d advise you to be aware of the potentially new horizons to explore.

The history question is more straightforward and deals with the scope of the initial load. The slower the business cycle, the more history you need to load to come up with useful data sets for time series analysis.

What data do we present via which interface to support these various decision types?

This question begs a separate article but for now, a few examples should make things clear.

Static reports for external stakeholders who require information for legal purposes,

• Reports using prompts and filters for team leaders who need to explore the data within predetermined boundaries,
• OLAP cubes for managers who want to explore the data in detail and get new insights,
• A dashboard for C- level executives who want the right cockpit information to run the business,
• Data exploration results from data mining efforts to produce valid, new and potentially useful insights in running the business.

If all these questions are answered adequately, we can start the data requirements collection as well as the source to target mappings.

Three causes, hard to eradicate

If your organisation shows one or more of these three causes, you have a massive change management challenge ahead that will take more than a few project initiation documents to remedy. If you don’t get full support from top management, you’d better choose between accepting this situation and become an Analytics Sisyphus or look for another job.

Project based funding

Government agencies may use the excuse that there is no other way but moving from tender to tender, the French proverb “les excuses sont faites pour s’en servir” [1] applies. A solid data and information architecture, linked to the required capabilities and serving the strategic objectives of a government agency can provide direction to these various projects.

A top performing European retailer had a data warehouse with 1.500 tables, of which eight (8!) different time dimensions. The reason? Simple: every BU manager had sovereign rule over his information budget and “did it his way” to quote Frank Sinatra.

Hierarchical organisations

I already mentioned the study of Prof. Karin Moser introducing three preconditions for knowledge co-operation: reciprocity, a long term perspective for the employees and the organisation and breaking the hierarchical barriers. [2]

On the same pages I quote the authors Leliveld & Vink and Davos & Newstrom who support the idea that knowledge exchange based on reciprocity can only take place in organisational forms that present the whole picture to their employees and that keep the distance between co-workers and the company’s vision, objectives, customers etc. as small as possible.

Hierarchical organisations are more about power plays and job protection than knowledge sharing so the idea of having one shared data platform for everyone in the organisation to extract his own analyses and insights is an absolute horror scenario.

Process based support

Less visible but just as impactful, if IT systems are designed primarily for process support instead of attending as well to the other side of the coin, i.e. decision support, then you have a serious structural problem. Unlocking value from the data may be a lengthy and costly process. Maybe you will find some inspiration in a previous article on this blog: Design from the Data.

https://ba4bi.blogspot.com/2015/07/the-future-of-information-systems.html

In short: processes are variable and need to be flexible, what lasts is the data. Information objects like a customer, an invoice, an order, a shipment, a region etc… are far more persistent than the processes that create or consume instances of these objects.

 [1]    Excuses are made to be used

 [2]    Business Analysis for Business Intelligence pp. 35 -38 CRC Books, a Taylor & Francis Company October 2012

Roadmap to a successful data lake

A few years ago, a couple of eCommerce organisations asked my opinion on the viability of a data lake in their enterprise architecture for analytical purposes. After careful study the result was 50 – 50: one organisation had no immediate advantage investing in a data lake. It would become just another data silo or even a data junk yard with hard to exploit data and no idea of the added value this would bring.

The other -€ 1 bn plus company- had all the reasons in the world to start exploring the possibilities of a repository for semi-structured and unstructured data. But it would take them at least two years to set up a profitable infrastructure. Technology was not the problem: low cost processing and storage as well as the software -mainly open source- was no problem. They even had no problem attracting the right technical profiles as their job offers topped everyone in the market. No, the real problem was integrating and exploiting the new data streams in a sensible and managed way. As I am about to embark on a new mission to rethink an analytical infrastructure with the data lake in scope, I can share a few lessons from the past and think ahead for what’s coming.

Start from the data and work your way up to the business case

Analyse the Velocity, Variability and Volume of the data to meet the analytical requirements

Is it stable and predictable? Then it’s probably an indication that your organisation is not yet ready for this investment. But if there is a rapid growth rate in at least one of these three Vs, you better get planning and designing your data lake.

Planning:

•         What time do we need to close the skills gap and manage a Hadoop environment professionally?
•        What is a realistic timeframe to connect, understand and manage the new semi-structured and unstructured data sources?

Designing:

•         Do we put every piece of data in the lake and write off our investments in the classical BI infrastructure or do we choose a hybrid approach where only new data types will be filling the lake?

o   In case of a hybrid approach, do we need to join between the two data sources?

o   In case of a total replacement of the data warehouse, do we have the proper front end tools to make the business users exploit the data or do they have to rely on data scientists and data engineers, potentially creating a bottleneck in the process?

•        How will we process the data? Do we simply dump it and leave it all to the data scientists to make sense of it or do we plan ahead on some form of modelling on the Hadoop platform, creating column families which are flexible enough to cope with new attributes and which will make broader access possible?
•        Do we have a metadata strategy that can handle the growth, especially from a user-oriented perspective?
•        Security and governance are far more complex in a data lake than in a data warehouse. What’s our take on this issue?

Check the evolution of your business requirements

It’s no use to invest in a data lake when the business ambitions are on a basic level and stuff like a balanced scorecard is just popping up in the PowerPoints from the CEO.

Some requirements are very clear on their data needs, but others aren’t. It may take a considerable amount of analysis to surface the data requirements for semi-structured and unstructured data.

And with legislation like the GDPR, some data may be valuable but also very hard to get as the consumer is more and more aware of his position in the data game. That’s why very fine-grained opt-ins are adding complexity to customer data management.

Develop a few winning use cases

“A leader is someone who has followers” is quite applicable in this situation. You are after all challenging the status quo and if there’s one thing I’ve learned in 30 years in analytics and ICT in general: a craftsman is very loyal to his tools. Managing change in the technical department will not be a walk in the park. It may require adding an entire new team to the department or at least have some temporary professionals come in to do the dirtiest part of the job and hand over the Hadoop cluster in maintenance mode to the team.

To enable all this, you need a few winning use cases that appeal to the thought leaders in the organisation. Make sure you pick sponsors with clout and the budget to turn PowerPoints into working solutions.

There certainly will be use cases for marketing, finance and operations. Look for the maximum leverage and get funded. And by the way, don’t bother the HR department unless you are working for the armed forces. They always come last in commercial organisations…

How to make progress in a political organisation

Why Business Analysis and politics don’t mix.

After thirty years of practice in all sorts and flavours of organisations there’s one that stands out as a tough conundrum for any business analyst and by extension enterprise architect as well as project managers. It’s the political organisation, so eloquently described by Henry Mintzberg. 
The problem with these organisations for a business analyst, project manager or enterprise architect is identical: setting priorities to determine the first iteration of the development cycle. This lack of priority ranking may lead to scope creep, projects that never deliver the product or a user community that is not on board, etc…

Forces in a political organisation

Wouldn't we all like to work in Tom Davenports Analytical Organisation?

In the paragraph “Decisions, Teams, and Groups at Work, Classification of Decision-Making Environments, I use a simple matrix to describe decision-making contexts for BI projects. But, believe me, you can use it for any project type.

You don’t need much time to determine if you’re in a political organisation. Look for committees that make the ultimate decisions, look for a lack of accountable individuals, slow decision making processes and a track record of projects that failed to deliver the intended product. Of course government bodies are by definition political but you will also find them in the private sector.

How to recognise a political organisation before you’re even at the reception desk?

Maybe this table can help:

Political organisations, by definition, don’t have shared goals. Each alderman, state secretary, each manager, wants to score his goals without letting the team take any credit for it. Because re-election or promotion matter… And political organisations always differ on the cause and effect chains which shows clearly in analytical projects.

Setting priorities in a political organisation

You can imagine that this is the toughest conundrum to solve; if you can’t prioritise “because everything is important” you can’t even start an analysis track. Unless you simply want to sell billable hours… And prepare for a debriefing and passing the buck, dodging any responsibility.
But if you’re a hired gun that may be exactly why you’ve been hired: to take the blame for the organisation’s ineptness to take responsibility and make choices even if they go against some members of the team. (I use “team” for want of a better word in a political organisation)
In this post, I am giving you a few tips and tricks to force the “team” to come up with priorities.

But first some context. The organisation is looking for a new way to analyse structured and unstructured data; Therefore it needs a modern data architecture. Your job as business analyst (and by extension project manager and enterprise architect) is to know what the strategic priorities of the organisation are.  This needs to match with the available data and information needs. You need to check the feasibility and then choose the first iteration to deliver analytical results.  A best practice is to check the organisation’s strategy, its initiatives to improve the organisation’s position in case of a commercial entity or the level of societal utility in case of a governmental or non for profit organisation.
Imagine the first intake with the project sponsor, the product owner and any other stakeholder who has been identified in the project structure.

Here’s the dialogue:

Business Analyst: At the kick off of this analysis track, I’d like to determine with you the first iteration: where we start analysing, designing and building the first deliverables.
The “team”: (silence)
Business Analyst: Do you have a project portfolio and do you use program management to prioritise the management actions? Do you have mission and vision statement for this project?
The “team”: We thought you could formulate the vision and the mission for the project. And no we don’t have a project portfolio. We do have an Excel sheet with a list of all the projects and their status.
Business Analyst: Could we infer from the status what the priorities are?
The “team”: No.
Business Analyst: What if we look at the budget per management project. Maybe the size says something about the priority? Or what if look at rejected project proposals and the reasons? Maybe that says something about the criteria.
The “team”: Not necessarily. First of all, all management project requests are answered positively and funds are allocated to these projects. Some projects may have big budgets but that doesn’t indicate anything about their importance.
Business Analyst: What about the number of full time equivalents allocated to each project?
The “team”: A high number may indicate something about the complexity or the scope but that doesn’t tell you what priority the project has.
Business Analyst: I think this one may help us out: have you indicated the origins of leakages and losses in your business processes and could those numbers give us a hint of what’s important to the management team?
The “team”: Leaks and losses are handled by the management team and as such are equally important.
Business Analyst: Does the amount of data, the connection with business processes and the variety in the data give us a clue where we should start the project?
The “team”: That’s we are hiring you as a Business Analyst.

Now it gets tricky and you make the call, as The Clash sing: “Should I stay or should I go”

Here are few of the killer questions and remarks that will lead you to the exit:

• What projects will get or got the most press coverage?
• What if you had to choose, right now?
• Do you expect me to deliver a successful end result if you don’t know what you want?

 

More on decision making contexts in the book “Business Analysis for Business Intelligence” p. 203 – 213 

Is there way out? Maybe.

The only escape route I can think of is to start with a stakeholder analysis. Try defining the primary stakeholders and map them on a RACI matrix. If that works, you can develop your first iteration with some confidence, knowing that danger is always on the road ahead..

Example of a stakeholder analysis that turns out well: the CEO’s desk is where the buck stops.

If a stakeholder analysis is inconclusive, there must be someone who’s not involved in the official decision making unit (DMU) who is the primary influencer. Now you’ll have to get out of your comfort zone as an analyst and start thinking like an account manager.

I was lucky to have training in the Miller Heiman Strategic Selling method as well as the Holden Power Base Selling method. It sharpened my skills for identifying and influencing these hidden decision makers. So here’s my advice: check out these two books. They will increase your efficiency in political organisations with an order of magnitude.

The new strategic selling is an update of the original, worth reading for any novice in business analysis and project management.
This is Jim Holden’s original book. Of course, as things go in this business, there were many to follow up on his success. Start here anyway.

Getting practical: How Analytics Can Drive the Information Architecture Development

Does the theory presented in the previous article work in practice? That is the theme of this post where I present an (anonymous) case from a project I did for a customer.

But before I proceed, a quick reminder from my book “Business Analysis for Business Intelligence”.

What every organisation needs to know boils down to four C’s. It is information about the customer, the cost, the competition and the competences of the organisation, the latter also represented by a higher level of abstraction: the capabilities.

The illustration below shows how these four C’s are the foundation of a balanced scorecard. But a balanced scorecard measures only the intended –or planned- strategy, not the emergent strategies. Therefore, this 4 C framework has a much broader scope and includes decision support for emergent strategies. 

To develop a shared knowledge of the customer, this organisation needed to embed a business rule in the data namely that contacts are associated with an account. This, because the organisation is an exclusive business-to-business marketing machine selling to large corporations. A contact without this association was registered and kept in a staging area, waiting to be associated with an account. In other words: only contacts related to an organisation were of use to the business. At least, in the present context. 

Today, this rule is cast in stone in a monolithic CRM application but the CIO wishes to migrate to a service factory in the near future. This way, when the business rule would change or when the company would move to a B2C market, the CRM processes would be easier to adapt to the new situation. A transition plan for all customer data needs to be developed.

Lingua Franca used the following phased approach:

1. ·        Mapping the customer data in a data portfolio
2. ·        Study the ASIS
3. ·        Link capabilities to analytics
4. ·        Map the capabilities on the data portfolio
5. ·        Define the information landscape
6. ·        Make the mapping analytics – transactional data
7. ·        Define the services
8. ·        Decide on the physical architecture

Mapping the customer data in a data portfolio 

A lot of customer data is of strategic value and a lot isn’t. That led us to use a modified version of McFarlan’s portfolio approach to information systems which can just as well be applied to data.

Variant on: McFarlan, F. W. (1981). "Portfolio approach to information systems.

"Harvard Business Review (September–October 1981): 142-150

The analytics version of this schema translates the four quadrants into workable definitions:

Strategic Data: critical to future strategy development: both forming and executing strategy are supported by the data, as well as emergent strategies where data might be captured outside the exiting process support systems.  The reason is clear: process support or transaction support systems are designed and tuned for the intended strategy. 

Turnaround Data: critical to future business success as today’s operations are not supported, new operations will be needed to execute. These data are often not even in scope of the emergent strategy processes. They may be hidden in a competitor’s research, in technological breakthroughs, in new government regulations or in consumer outcries against abuse to name a few sources.

Support Data: Valuable but not critical to success

Factory Data: critical to existing business operations: the classical reports, dashboards and scorecards

In this case, the association between account and contact was considered factory data as it describes the way the company is doing business today

As the illustration below in the Archimate model shows, there is a cascading flow of business drivers and stakeholders that influence the business goals which in their turn impact the requirements that are realised by business processes. These are supported by legacy systems and new software packages or bespoke applications. The result of this approach is a dispersed view on the data that are used and produced in these applications. What if not processes but data would be at the base of the requirements? Would this change the organisation’s agility? Would it enhance responsiveness to external influence? That was the exercise we were preparing for. 

Data dispersion in a classical IT landscape

.

Study the ASIS

Today, the business process of account and contact registrations is as follows:

The present CRM monolith supports this process but future developments like the takeover of a more consumer oriented business may change the business model and the business process drastically. Thus, the self-service registration process should make the link between contact and account optional and the validation process should only deal with harmonising data to make sure the geographical information is correct and contact data are uniform as far as (internal) naming conventions and (external) reference data are concerned. It is already a great step forward that the company uses a master data management system to separate data management from process management. This enables a smoother transition to the new information architecture development method. 

Link capabilities to analytics

Therefore an extensive inventory of all potentially needed business capabilities is undertaken and linked to the relevant business questions supporting these capabilities.

In this example we present a few of these present and future business questions:

What is the proportion of contacts from our B2B customers that may be interested in our consumer business?

Which accounts may experience a potential threat from our new consumer business unit?

Which contacts from the B2C may become interested in our B2B offerings?

Which products from the B2C unit may prove sellable via the B2B channels?

By listing all the relevant present and future business questions, it becomes clear that the account validation process as it is defined today may need to change and what is considered factory data today may get an “upgrade” to strategic and turnaround data to deal with the challenges.

Map the capabilities on the data portfolio

In this diagram, the entire data landscape of the account – contact association is charted and managed via five methods. 

Operational business metadata describe the context in which data is created, updated and deleted as well as the context in which it is used. A minimum deliverable is instructions and training for the people who perform the CRUD operations.Process metadata relates the business process (present and future) to the business context to provide the process stakeholders with information and motivation: the what, why, when and who of the process and the data captured.Business Intelligence metadata describes the decision support possibilities in the present and future clients: dashboards, reports, cubes, data sets for further examination,…Process alignment: it describes what is often a mutual adjustment between a monolithic application and the business process it supports. Some market leaders in OLTP software present their process flows as best practice. As if all businesses should converge in their way of doing business…ETL Architecture documents the lineage from source to target, the transformations, quality measures, as well as the technical aspects of the process i.e. parallel or sequential loading, dropping of indexes and rebuilding them, hashing, etc… 

Define the information landscape

Even in this simple customer – account relationship some thinking needs to be done about a holistic view on the essential elements defining the relationship. By “essential” I mean the minimum attributes and levels of detail that need to be shared outside the context of CRM to be used in other business functions like HR, operations, finance,…

Here are a few of the considerations to be made:

How long is a customer considered as such? If the average buying frequency of your product is twice a year, for how many years do you keep the relationship active if for three years no order has come in? How do we compare the account performance in case of mergers? Does an account always need a DUNS number? Or a VAT registration? What about informal groups regularly doing group purchases? Discussing these and many other issues lays the foundation for a data governance process. 

Make the mapping analytics – transactional data

This phase is crucial for the quality of your decision support system and is very much like the business analysis process for analytics. Start with high level concepts and descend to the lowest grain of attributes and transaction records as well as external sources like social media, open data and market research data.For instance: “customer loyalty” is expressed as “a constantly high share of wallet over an average historic period of time of three years and a projected future loyalty period of another three years”.

Can you imagine the data needed to make this definition work? 

The exercise at this customer’s site produced 87 different data types coming from the ERP and CRM systems as well as external data like Net Promotor Scores, contact centre chat data, e-mails and response to LinkedIn posts. It sparked new ways of customer interaction procedures: new sales and order processing methods as well as new aftersales initiatives, the organisation would never have come up with if it hadn’t done this exercise.

Define the services

To move from the monolithically based approach to a more micro service oriented architecture, we needed to decompose the monolith into distinctive reusable services and data components. This approach forces a strict quality management for the data in scope as errors or poor quality will reflect on an enterprise scale. On the other hand, this “do it right the first time” principle avoids replication of work and improves the quality of decision making drastically.The schema below needs some explanation. The intake service triggers the validation service which checks the contact and account data with reference data, Chamber of Commerce data and, when finished, triggers the registration service which in its turn triggers the master data update service. MDM contact is now a superclass of this contact and will be used enterprise wide. Four services now ensure reusability for not just the CRM application but for all other use cases in the organisation. And the data quality improves drastically as the “do-it-right-the-first-time” principle is easier to fund for enterprise wide data. 

Data landscape for a CRM and customer MDM application

Decide on the physical architecture

The classical approach using at least two environments is becoming obsolete for organisations that want to stay ahead of the competition. The separation between transaction processing and analytical processing will go out the window in the next few years. Not only because of the costly maintenance of Extract Transform and Load (ETL) processes between the transaction systems and the data warehouse but first and foremost because of the lack of integration with unstructured data that are in Hadoop Distributed File Sets (HDFS) or streaming data that are caught in Resilient Distributed Datasets (RDD)

The organisation needs a significant leap forward and is now examining the Vector in Hadoop solution, a database that combines the classic SQL environment with NoSQL. The reasons are supported by objective facts: a rapidly scalable full ACID SQL database based on HDFS. It supports modify, insert and updates using a patented technology developed at the University of Amsterdam: the Positional Delta Trees (PDT). More on this in their paper which is published here. The short version of PDT: a separation between the write and read store where updates are merged into the write store at run time using the row index for a correct positioning of the modify/insert or update. The result? Online updates without impacting the read performance. Since the database can also access Spark’s parallel processing capability combining Spark RDD architecture accessed from the SQL perspective so that queries that were previously impossible to consider, this system combines the very best of three worlds: ACID based transaction support, complex event processing and HDFS support for unstructured data analytics with a flexible approach to changing data influx –provided you do your homework and define the column families in the broadest possible sense to fit your analytical needs. 

Data loading – if that is the purpose - can be achieved at a rate of around 4TB per hour comprising four billion ‘120 column’ tuples per hour on a 10 node Hadoop cluster – or around  500 billion columns per hour in total! (many caveats apply but it is still a remarkable performance.

The advantage of this architecture will be exploited to the maximum if the data architecture is connecting transaction data, which are by definition microscopic and consistent, to analytical concepts which are macroscopic, flexible and fuzzy. So here is –finally!- my sales pitch: do your proper business analysis for analytics well. Because the cost of preparing for a well thought through system is a fraction of the license-, hardware- and maintenance cost.

Epilogue: an initial approach

A first attempt to map the various data ingestions to Vector H and the consumers of the data was made as illustrated below. This has a few consequences we will discuss in the next few paragraphs.

A more in depth example of Vector H’s power

One aspect will be along the Spark line – the ease of facilitating combined queries that incorporate data that is held in Hadoop with managed structured data in a way that standard BI tools simply query the database in the same way that they do a standard SQL database. I.e. the user does not need to use ETL or ELT separately from the actual BI query for ad-hoc queries once they have defined the external table as referencing the Hadoop data. It is hard to define the simplification this brings.  In its simple form – it’s like the data really is inside the Vector database. This brings the advantage that current solutions – including off the peg turnkey applications can access this data.

This example shows the declaration made by the DBA, once this is done, the end users’ business layer will simply see ‘tweets’ as a table that can be joined to actual tables 

CREATE EXTERNAL TABLE tweets

(username VARCHAR(20),

tweet VARCHAR(100),

timestamp VARCHAR(50))

USING SPARK

WITH REFERENCE='hdfs://blue/tmp/twitter.avro',

FORMAT='com.databricks.spark.avro' 

This command will select tweets that are made which are from customers only, those from non-customers will be ignored:

SELECT tw.username , cust.firstname, cust.lastname, tw.tweet 

FROM tweets  tw,

            Customers cust

WHERE tw.username = cust.username 

AND      tw.Timestamp > TIMESTAMPDIFF( SECONDS, 120, CURRENT_TIMESTAMP )

\g

Similar queries can track non-customer queries.

Where possible restrictions will be pushed down to the Spark ( Map Reduce and Scala level ) in Hadoop to be answered. The data never needs to be stored. Of course some data may be required to be added to the structured data. I already applied this in a customer analysis project where I illustrated how the results from Big Data analytics can be transformed to dimensions in the “classical” data warehouse:

To conclude: will hybrid architectures make data modelling obsolete?

I can’t yet generalise this for all hybrid databases but at least from Vector H we know that there is a serious chance. It uses a partition clause that distributes data on the hash of one or more columns that have a minimum of 10X unique values evenly distributed as the number of partitions you are using.

Vector H is therefore the most model agnostic data store I know. You simply create a schema, load data and run queries. There is no need for indexing or some form of normalisation with this technology.
Whereas the need for 3NF, Data Vault or Star schemas may become less important, governing these massive amounts of data in a less organised way may become the principal issue to focus on. And metadata management may become the elephant in the room.