Roadmap to a successful data lake

A few years ago, a couple of eCommerce organisations asked my opinion on the viability of a data lake in their enterprise architecture for analytical purposes. After careful study the result was 50 – 50: one organisation had no immediate advantage investing in a data lake. It would become just another data silo or even a data junk yard with hard to exploit data and no idea of the added value this would bring.

The other -€ 1 bn plus company- had all the reasons in the world to start exploring the possibilities of a repository for semi-structured and unstructured data. But it would take them at least two years to set up a profitable infrastructure. Technology was not the problem: low cost processing and storage as well as the software -mainly open source- was no problem. They even had no problem attracting the right technical profiles as their job offers topped everyone in the market. No, the real problem was integrating and exploiting the new data streams in a sensible and managed way. As I am about to embark on a new mission to rethink an analytical infrastructure with the data lake in scope, I can share a few lessons from the past and think ahead for what’s coming.

Start from the data and work your way up to the business case

Analyse the Velocity, Variability and Volume of the data to meet the analytical requirements

Is it stable and predictable? Then it’s probably an indication that your organisation is not yet ready for this investment. But if there is a rapid growth rate in at least one of these three Vs, you better get planning and designing your data lake.

Planning:

•         What time do we need to close the skills gap and manage a Hadoop environment professionally?
•        What is a realistic timeframe to connect, understand and manage the new semi-structured and unstructured data sources?

Designing:

•         Do we put every piece of data in the lake and write off our investments in the classical BI infrastructure or do we choose a hybrid approach where only new data types will be filling the lake?

o   In case of a hybrid approach, do we need to join between the two data sources?

o   In case of a total replacement of the data warehouse, do we have the proper front end tools to make the business users exploit the data or do they have to rely on data scientists and data engineers, potentially creating a bottleneck in the process?

•        How will we process the data? Do we simply dump it and leave it all to the data scientists to make sense of it or do we plan ahead on some form of modelling on the Hadoop platform, creating column families which are flexible enough to cope with new attributes and which will make broader access possible?
•        Do we have a metadata strategy that can handle the growth, especially from a user-oriented perspective?
•        Security and governance are far more complex in a data lake than in a data warehouse. What’s our take on this issue?

Check the evolution of your business requirements

It’s no use to invest in a data lake when the business ambitions are on a basic level and stuff like a balanced scorecard is just popping up in the PowerPoints from the CEO.

Some requirements are very clear on their data needs, but others aren’t. It may take a considerable amount of analysis to surface the data requirements for semi-structured and unstructured data.

And with legislation like the GDPR, some data may be valuable but also very hard to get as the consumer is more and more aware of his position in the data game. That’s why very fine-grained opt-ins are adding complexity to customer data management.

Develop a few winning use cases

“A leader is someone who has followers” is quite applicable in this situation. You are after all challenging the status quo and if there’s one thing I’ve learned in 30 years in analytics and ICT in general: a craftsman is very loyal to his tools. Managing change in the technical department will not be a walk in the park. It may require adding an entire new team to the department or at least have some temporary professionals come in to do the dirtiest part of the job and hand over the Hadoop cluster in maintenance mode to the team.

To enable all this, you need a few winning use cases that appeal to the thought leaders in the organisation. Make sure you pick sponsors with clout and the budget to turn PowerPoints into working solutions.

There certainly will be use cases for marketing, finance and operations. Look for the maximum leverage and get funded. And by the way, don’t bother the HR department unless you are working for the armed forces. They always come last in commercial organisations…

How to make progress in a political organisation

Why Business Analysis and politics don’t mix.

After thirty years of practice in all sorts and flavours of organisations there’s one that stands out as a tough conundrum for any business analyst and by extension enterprise architect as well as project managers. It’s the political organisation, so eloquently described by Henry Mintzberg. 
The problem with these organisations for a business analyst, project manager or enterprise architect is identical: setting priorities to determine the first iteration of the development cycle. This lack of priority ranking may lead to scope creep, projects that never deliver the product or a user community that is not on board, etc…

Forces in a political organisation

Wouldn't we all like to work in Tom Davenports Analytical Organisation?

In the paragraph “Decisions, Teams, and Groups at Work, Classification of Decision-Making Environments, I use a simple matrix to describe decision-making contexts for BI projects. But, believe me, you can use it for any project type.

You don’t need much time to determine if you’re in a political organisation. Look for committees that make the ultimate decisions, look for a lack of accountable individuals, slow decision making processes and a track record of projects that failed to deliver the intended product. Of course government bodies are by definition political but you will also find them in the private sector.

How to recognise a political organisation before you’re even at the reception desk?

Maybe this table can help:

Political organisations, by definition, don’t have shared goals. Each alderman, state secretary, each manager, wants to score his goals without letting the team take any credit for it. Because re-election or promotion matter… And political organisations always differ on the cause and effect chains which shows clearly in analytical projects.

Setting priorities in a political organisation

You can imagine that this is the toughest conundrum to solve; if you can’t prioritise “because everything is important” you can’t even start an analysis track. Unless you simply want to sell billable hours… And prepare for a debriefing and passing the buck, dodging any responsibility.
But if you’re a hired gun that may be exactly why you’ve been hired: to take the blame for the organisation’s ineptness to take responsibility and make choices even if they go against some members of the team. (I use “team” for want of a better word in a political organisation)
In this post, I am giving you a few tips and tricks to force the “team” to come up with priorities.

But first some context. The organisation is looking for a new way to analyse structured and unstructured data; Therefore it needs a modern data architecture. Your job as business analyst (and by extension project manager and enterprise architect) is to know what the strategic priorities of the organisation are.  This needs to match with the available data and information needs. You need to check the feasibility and then choose the first iteration to deliver analytical results.  A best practice is to check the organisation’s strategy, its initiatives to improve the organisation’s position in case of a commercial entity or the level of societal utility in case of a governmental or non for profit organisation.
Imagine the first intake with the project sponsor, the product owner and any other stakeholder who has been identified in the project structure.

Here’s the dialogue:

Business Analyst: At the kick off of this analysis track, I’d like to determine with you the first iteration: where we start analysing, designing and building the first deliverables.
The “team”: (silence)
Business Analyst: Do you have a project portfolio and do you use program management to prioritise the management actions? Do you have mission and vision statement for this project?
The “team”: We thought you could formulate the vision and the mission for the project. And no we don’t have a project portfolio. We do have an Excel sheet with a list of all the projects and their status.
Business Analyst: Could we infer from the status what the priorities are?
The “team”: No.
Business Analyst: What if we look at the budget per management project. Maybe the size says something about the priority? Or what if look at rejected project proposals and the reasons? Maybe that says something about the criteria.
The “team”: Not necessarily. First of all, all management project requests are answered positively and funds are allocated to these projects. Some projects may have big budgets but that doesn’t indicate anything about their importance.
Business Analyst: What about the number of full time equivalents allocated to each project?
The “team”: A high number may indicate something about the complexity or the scope but that doesn’t tell you what priority the project has.
Business Analyst: I think this one may help us out: have you indicated the origins of leakages and losses in your business processes and could those numbers give us a hint of what’s important to the management team?
The “team”: Leaks and losses are handled by the management team and as such are equally important.
Business Analyst: Does the amount of data, the connection with business processes and the variety in the data give us a clue where we should start the project?
The “team”: That’s we are hiring you as a Business Analyst.

Now it gets tricky and you make the call, as The Clash sing: “Should I stay or should I go”

Here are few of the killer questions and remarks that will lead you to the exit:

• What projects will get or got the most press coverage?
• What if you had to choose, right now?
• Do you expect me to deliver a successful end result if you don’t know what you want?

 

More on decision making contexts in the book “Business Analysis for Business Intelligence” p. 203 – 213 

Is there way out? Maybe.

The only escape route I can think of is to start with a stakeholder analysis. Try defining the primary stakeholders and map them on a RACI matrix. If that works, you can develop your first iteration with some confidence, knowing that danger is always on the road ahead..

Example of a stakeholder analysis that turns out well: the CEO’s desk is where the buck stops.

If a stakeholder analysis is inconclusive, there must be someone who’s not involved in the official decision making unit (DMU) who is the primary influencer. Now you’ll have to get out of your comfort zone as an analyst and start thinking like an account manager.

I was lucky to have training in the Miller Heiman Strategic Selling method as well as the Holden Power Base Selling method. It sharpened my skills for identifying and influencing these hidden decision makers. So here’s my advice: check out these two books. They will increase your efficiency in political organisations with an order of magnitude.

The new strategic selling is an update of the original, worth reading for any novice in business analysis and project management.
This is Jim Holden’s original book. Of course, as things go in this business, there were many to follow up on his success. Start here anyway.