Why do data quality projects fail?
On Data Quality Pro, most of the stories you will read are of success over adversity but of course in reality it’s not always that way.
I have had my fair share of failed projects and anyone who hasn’t is certainly the exception.
You may eventually win the war on poor data quality but along the way it’s inevitable that you’ll lose the odd battle too.
Over the years you start to realise that one of the primary reasons for failure is not that you failed to adopt the most capable technology or methodology but you failed to adopt a comprehensive approach to change management.
Managing change is critical to a data quality project. You need to take the organisation with you and this means getting the business community onside.
Why is change management so difficult?
One thing that always frustrated me was how one strategy in one part of the organisation would fail miserably in another. Even in the same business user group it’s not uncommon to get one bunch of business users totally bought into data quality and then another group running for the hills.
Why does this happen?
1) Time : One reason for change management failure is due to the speed of the project. Most projects are financially incentivised to go fast. Get the results, cut the staff burn rate to a minimum and get the project finished. A fundamental problem of course is that data quality management is not a project, it’s a corporate habit-system for implementing prime directives.
2) Support : Business users need to know senior management is involved with and committed to the process. All your efforts can be undone in an instant if the leadership of a group or department are privately undermining your goals with negative communication in the workforce.
3) Workload : If you ask an overworked business user to commit more time to activities that are not linked to his or her salary structure then the results are predictable. Business workers need to feel like they are not being punished for getting involved with your data quality improvement plans. Failing to realise this also results in a feeling amongst workers of a lack of support from management - see point 2.
4) Goal Conflicts : Business users may face goal conflicts when we introduce change. If we’re trying to change behaviours we need to be sure that these changes are not negatively impacting workers’ rewards systems or perceived roles. For example, a previous project of mine was to reduce headcount through data quality improvement. The idea being that if we improved the quality of processes, technology and data then the business unit could reduce its workforce. Right from the outset we hit a roadblock when the unit head openly admitted that their budgets were set based on the size of their team. It was also apparent that within this type of organisation your standing as a manager was influenced by the number of staff in your department. This resistance to change can be difficult to manage.
There are lots of reasons why change fails but these are some of the common ones I’ve witnessed in the past (if you can think of more just use the comments section below).
What is the right way to involve business users in your Data Quality Management initiative?
So we’ve learned of some change management failure points. But how do we overcome these and get the business engaged and supportive of the data quality process?
Here are some simple approaches I have found to be extremely useful:
Make your data quality assessments subjective (as well as qualitative)
When assessing data quality we tend to rely heavily on data quality technology. Issues are found which are relayed back to the business and together you work out a path towards a resolution.
A smarter approach is to seek their opinions from the outset, particularly around data quality dimensions.
For example, in ‘Introduction to Information Quality’ (Fisher, Lauria, Chengular-Smith, Wang) the authors use business users as the core of the data quality assessment technique. They even go so a far as listing different types of users (Consumers, Custodians and Collectors) to draw insight from the different perspectives each user group has based on their involvement in the information lifecycle.
As a result of this business inclusion many of their data quality dimensions focus on subjective metrics such as:
Help business users feel like experts
As a data quality practitioner it is important not to portray yourself as the ‘expert’. Taking such a superior stance can create a feeling of resentment from the business community and you may make workers feel as though their hard work is simply amounting to a catalogue of defects.
To get the business onside you need to practice humility. Openly acknowledge they are the experts. They are the ones who generate revenue or quality services for the organisation so don’t be critical of their processes and data. You need to practice humility at all times if you want to win them over.
An example of this was my very first project where we appointed key workers in the data entry team as specialist business process advisors. We emphasized their unique role on the project because, it was true, they really were experts of the data entry processes. As a result they took the lead in designing improvements and we simply supported their goals for the new approach.
Make sure that they have a career development plan that allows for data quality training if they want to include it in their career goals.
Help business users create their own solution
I have touched on this briefly but I always try and encourage the business community to create a solution to the problems we’ve jointly observed.
You may often have a gut feel about what the solution should be but if you want the business community to buy-in they have to feel like they’ve been instrumental in the choice of direction they take.
If you recommend that a tool be implemented to validate data automatically instead of performing costly and time consuming manual checks then you may be tempted to just forge ahead with that approach. But how will the business community react when they see their activities stripped away from them?
What if they arrived at that decision themselves? The result would be more positive because they are part of the solution. We did exactly this in one team. Instead of making one team members’ tasks redundant we simply trained all four members of the team in the correct use of data quality technology. With matrix management they became an asset to the whole company and went on to develop careers as data quality practitioners.
It’s easy to think of data quality management as some kind of IT-centric initiative because historically data has always been considered a technical asset.
As any practitioner will tell you, the business owns the data and information within the organisation and without their support you will face considerable difficulties in rolling out any kind of data quality improvement initiative.