Know Your Data!

In today’s world of increasing feature sets it is easy to become bedazzled by the latest instalment of new functionality. Data quality software is no different than other enterprise applications in this regard. However, perhaps more important than ever, it is critical that users of data quality software know the data they are analyzing in order to fully leverage the tool and truly deliver good quality analysis.

First things first

With all the advances in data quality software it is easy to become enamoured with the bells and whistles and thus lose sight of the fundamentals. As these rich feature sets enable the less technological oriented business users to participate in data quality exercises, the fundamentals of data analysis become more, well, fundamental.

One of my first questions when I start a new data quality project is, “Does anyone have a data model?”. Sometimes I get lucky and the DBA has one. Sometimes when I am really lucky the DBA also has a data dictionary. I don’t play lotteries due to my lack of being “really lucky”, if you know what I mean.

If I get a data model, I sit and examine it like a CSI agent does blood spatter. It is, after all, my roadmap to solving some mysteries. If there is not a data model available, I dig-in and create one. Most times I start out with pencil and paper (I know, how arcane!). Most data quality projects involve one or two main entities, the customer or a product. As such, I use this as my starting point.

For the sake of simplicity, let’s concentrate on a customer-focused data quality initiative. Commonly referred to as Customer Data Integration, or CDI, these projects involve the most critical person in any business; the customer! Customers are a complex animal, particularly from a data perspective. Organizations often focus on collecting as much data regarding customers as possible and rightfully so. As a result, there is usually a fair amount of data in, or related to, the customer entity.

What works for me is to start out with a big picture and narrow my focus with further analysis. My first picture step is to build what I call my “customer frame”. The customer frame consists of the customer entity and each entity to which it is related. In the figure below you can view some of my basic customer frame based on a typical instance of Microsoft Dynamics CRM (in pencil nonetheless).

Figure 1 Customer focused data model (click to enlarge)

The point of including such a crude example is to demonstrate the fact that this exercise is about learning the data and not being pretty.

What did I learn from the exercise? For starters we can see that:

• The ContactBase Entity has a relationship to the ContactExtensionBase entity but I didn’t find relevant/helpful data in the ContactExtensionBase table (which is why it contains only the ID field relating the tables)
  • This was useful in identifying what entities are essential and which are no
  • Since the ContactExtensionBase table is a storage place for custom defined details regarding a contact, it was critical to examine this entity so I could be sure I was not missing very specific contact details
• The ContactBase Entity can be related to the AccountBase entity by the ContactBase.AccountID <> AccountBase.AccountID
  • This allows me to, among other things, determine if there are active contacts associated with inactive accounts
• The ContactBase Entity can be related to the CustomerAddressBase entity via the ContactBase.ContactID <> CustomerAddressBase.ParentID
  • This allows me to relate a contact to their address on record. Addresses play a critical role in CDI projects so this is a crucial piece of information and a relationship I’ll know in my sleep before long
  • Knowing this relationship allows me to check for contacts with no address records or, worse yet, orphaned addresses (addresses without an association to a contact)

The list above is just a simple example of how studying the data model translates into practical knowledge which is critical to the success of a data quality initiative.

One of the more subtle points I touch on in this example is the identification of entities in the data model that are not necessarily useful. This is particularly true of Microsoft Dynamics CRM. Microsoft Dynamics CRM provides an “extension” table for just about every table in the database so that organizations can define and store data unique to their enterprise. As such, this is where to look for data that is “near & dear” to the hearts of users.

I came across a prime example of this on my last project when I discovered that my client was storing a unique identifier in one of these extension tables. This helped me identify a code that was akin to a social security number for each unique customer! On a CDI project, data that uniquely identifies a customer is a treasure well worth the time invested to discover it.

Conclusion

While I am definitely one to “geek-out” on new features of my favorite analysis tools, there is simply no replacement for knowing the data. Learning the basics of a data model doesn’t take very long but provides valuable insight into an organizations “data state”. I highly recommend writing up a cheat sheet like the one in the figure above and keeping it close to you as you define your data quality cleansing, standardization and matching routines.

About William Sharp

William is the founder of The Data Quality Chronicle, a site that attempts to capture the opportunities and challenges that exist as part of the various data quality initiatives encountered in the enterprise environment.  The Data Quality Chronicle tells real-life stories from data quality projects that can help members of the data quality community on their projects, give those not familiar with data quality more insight, and help students studying information technology learn more about the aspects of data quality.

William is a Senior Business Systems Analyst at Edgewater Technology, a technology management consulting firm providing a synergistic blend of specialty Information Technology services primarily in the North American market.

Connect with William on: LinkedIn : Twitter

Useful Resources

The Data Quality Chronicle

Integrated Modelling Method: An Introduction

Business Systems Modelling: Function Modelling (Tutorial 1)

Business Systems Modelling: Data Structure Modelling (Tutorial 2)

5th Normal Form: The Achilles Heel of ETL & Data Warehouse Projects

Necessity of Conceptual Data Modeling for Information Quality

Image credit: Jason Mulligan

This post provides some practical tips and advice for transforming data quality dimensions into engaging, customer-focused benefits.

How to Present Data Quality Dimensions For Maximum Impact

In the field of data quality we talk about data quality dimensions a great deal.

We frequently measure data against dimensions of accuracy, completeness, consistency, timeliness and even some more abstract dimensions - trust, believability, accessibility.

Dimensions have become a kind of "DNA" for data quality. By measuring certain dimensions we can even compare different data to see how the "quality of their DNA" compares.

I do have some issues with dimensions. I've witnessed practitioners and data quality vendors over-simplify their use. Poorly implemented they don't often align with what the business really needs out of their data. However, I accept that as an aggregation tool and reporting method they do have merit.

But I digress.

My main issue with dimensions is that we often focus on the features instead of the benefits, particularly when talking to potential sponsors, managers and other stakeholders.

What triggered this post was a white paper I read recently. Aimed at business people, this paper explained the benefits of a data quality assessment process. The paper stated that by following a structured methodology we can establish how complete, accurate, timely and consistent our data is. What I found completely missing from this paper was the focus on the end customer - the project sponsor.

Picture the scene if you will...

I'm a business manager and you're a data quality analyst. You have the unenviable job of convincing me all this data quality voodoo is really worth the pain of me releasing budget to do "some data stuff" I assumed was already adequately taken care of.

For the business manager just trying to hit their targets, you telling me that you can help our data possess greater accuracy, uniqueness, precision, timeliness, availability, completeness and consistency is a totally alien language.

However, turn these "data related dimensions" into "sponsor related dimensions" and suddenly everything takes on a new focus.

So, you're saying that you can HELP ME deliver more accurate decisions?  

I'm skeptical but tell me more.

You can HELP ME demonstrate that my team gets its work done more timely?

Interesting, this is a common complaint against my department.

You can HELP ME create more consistency in the business services we deliver?

This sounds promising, keep going.

You can HELP ME reduce duplicated effort across my team.

You're getting warmer.

You can HELP ME be more precise and accurate with those quarterly figures I present at the steering group?

Sold. When do we start?

See how we've switched those data focused dimensions to personal dimensions that push the buttons of the sponsor? It is so easy as a data quality practitioner to get lost in the features and technicalities of data quality.

Business sponsors don't care about technicalities. They really want to know what THEY will GET with their money AFTER you've done all the data quality voodoo.

Got a few minutes to spare, check out the recent interview with information quality expert Kathy Hunter, this is the reality of data quality in most companies:

Unfortunately, the business didn’t have the inclination to deal with their problems effectively so I had to wait to really get started. I spent nearly two years telling every senior manager I knew about the importance of information quality and the ways that poor quality information could be improved.

It took a very public embarrassment to focus senior management’s minds [...] Within two days I had a two-year budget of £2.5 million and three months later we had already found £10 million in lost revenue.

Kathy Hunter, Kynetika

As Kathy clearly demonstrates, most business leaders are far more interested in what all this means to them, it's simple human nature.

Without any tangible benefit to sponsors it can often be impossible to release cash for improvements but isn't it incredible how quickly organisations can find cash when corporate (read personal) embarrassment is a reality? There is always cash, you just need to discover the levers to release it.

Another common mistake is to focus on how data quality will improve life for the data consumer, not the customer. In many cases, the dimensions of data quality you are aiming to improve are completely at odds to my goals as the person funding your project.

"You want to help my team get rid of duplicate customers so they can manage service calls more effectively? But my annual staffing budget is set by the number of customers and you're telling me you want to eliminate them by 30%!"

The next time you are about to present a bunch of data quality dimensions to a potential sponsor, ask yourself if your presentation answers two simple words - so what?

Even if you think they are listening intently to your "features" story they will be totally focused on the personal benefits to them. Do some research, find out what emotional buttons to press and connect the dots.

If your data quality dimensions are not presented in a way that is compelling to them personally then you may be in for a long wait until a tipping point of embarrassing proportions pushes them into action.

Useful Resources

Expert Interview: Kathy Hunter of Kynetika

Do you struggle to create a compelling introduction to your DQ proposition? Learn how to tell stories that will boost your DQ opportunities.

Using Metrics to Assert a Business Case for Data Quality

Building Your Internal Data Quality "Business"

Suffering from feast or famine in your DQ business or career? Learn how to network and promote yourself more effectively with this 2-part tutorial.

5 simple activities to help sharpen your data quality sales performance

7 Productivity Tools for the Innovative Data Quality Leader

Does Your Business Suffer From a Data Quality Reality Gap?

Expert Interview: Kathy Hunter of Kynetika

Data Quality Pro: What route did you take prior to becoming an IQ consultant – how did you get started?

Kathy Hunter: When I was Service Delivery Manager at One2One (now T-Mobile), I found that most of the problems experienced by the different systems under my control were due to serious issues with the quality of information within those systems. I attended a workshop where I learned everything I needed to know to solve my problems. Unfortunately, the business didn’t have the inclination to deal with their problems effectively so I had to wait to really get started. I spent nearly two years telling every senior manager I knew about the importance of information quality and the ways that poor quality information could be improved.

It took a very public embarrassment to focus senior management’s minds. When the problem occurred, the months of gentle persuasion paid off and all those senior managers were ready to fund my IQ initiatives, called Project High IQ. Within two days I had a two-year budget of £2.5 million and three months later we had already found £10 million in lost revenue that was easy to recover. We’d paid for ourselves four times over!

Data Quality Pro: What skills have been most beneficial for developing your career?

Kathy Hunter: I would have to say that strong problem-solving and communication skills have been the most beneficial to me in my career. Being able to systematically analyse problems, potential solutions and then communicate the improvement strategies required to overcome issues is a big part of being successful in IQ (and so many other things).

Of course, it also helps if you have a knack with data. I always say that you needs to listen to the voice of the data when dealing with problems. If you tune your hearing to what the business needs from data within their systems, the messages will begin to be clear. This is especially important when profiling the data. Just reeling off the stats about data will not tell you whether the information the business requires is compromised.

Only by understanding the costs the business are experiencing will you do that. Once you hear the voice of the data, you then can use your communication skills to present the findings of the data profile in a way that resonates with the business people with whom you will be working.

Data Quality Pro: Do you find that organisations struggle to understand the role of an IQ specialist?

Kathy Hunter: I have found that organisations cannot understand information quality full stop if their senior managers are not in some sort of pain that they can relate back to a data problem.

It may seem obvious to people at the coal face but that does not automatically translate to an understanding at executive level. Even after the pain exists, it is essential to be able to communicate the role of the IQ specialist in a way that works for business people who may not have technical expertise.

After all, IQ is about business success or failure, not about implementing technical solutions. Technology will assist the IQ specialist in their job but it’s overcoming business obstacles that will ensure continuing funding for IQ.

Data Quality Pro: Is it still important to have technology skills or do you feel there are sufficient non-technical roles coming through now?

Kathy Hunter: To successfully implement IQ solutions, you will doubtless need some people with excellent technology skills. Some solutions will involve changes to systems and will need clever technical experts. However, many of the roles in IQ are not technology-based. Non-technical roles would include business analysts, project managers and process improvement experts. Even data analysts do not have to be terribly technical as many new analysis tools are geared towards non-technical individuals. In my experience there’s about a 50:50 split between technical and non-technical roles but that could even be 40:60.

Data Quality Pro: Are you finding any organisations creating a career path for IQ professionals or is there still a long way to go?

Kathy Hunter: There are some but these are often technical roles. Many in organisations that think that there is a magic bullet that will magically rectify their IQ problems. Very few organisations understand the level of organisational change required to implement truly long-lasting improvements. Only the most mature companies are looking at importance of non-technical roles in IQ. I think we still have a long way to go.

Data Quality Pro: What advice would you give to someone who is trying to get IQ off the ground within their organisation?

Kathy Hunter: My best advice would be to learn everything you can about the subject and then decide where and when to get started. Remember, it may not be the right time to start. However, if you think it is, start by gathering evidence to gain support and start building a business case.

You need input from the business to find out where problems occur. Here are some suggested ways of getting this - advertise on your intranet that you are looking for information quality problems and ask people to contact you to discuss their issues (stand back and wait for the onslaught!); conduct lunchtime forums in individual departments asking people to come and tell you about problems they are having; send out questionnaires to middle managers within the business asking them to answer questions about information quality problems and suggest they pass the questionnaires around to people that work for them.

When you discuss problems, find out whether there has been a specific senior manager that is feeling pain from all of this. This could be lost sales, customer churn, low profits, all the issues that were discussed in my business case presentation. Also, take this opportunity to get people to bring your evidence of the costs of poor quality. This will be invaluable in building a business case later.

Once you gathered the evidence, find out if one of the senior managers you’ve identified would be willing to sponsor your work. Make sure this senior manager understands that IQ is more than just IT implementing new tools or changes to existing systems. It will involve many areas of the organisation and will need more than just funding, it will involve real business change.

Data Quality Pro: What advice would you give someone starting out in our profession?

Kathy Hunter: Someone who wants to join the profession might want to start off working as a data analyst to gain valuable background knowledge or perhaps work in process improvement areas so you understand the whole idea of making improvements in a structured way. The IAIDQ are introducing a certification course that will provide practical skills and methodologies. With the right building blocks and a good head for data, a fulfilling career in IQ can be yours.

Summary of Points Raised

1. Be patient, plant the seeds of data quality and sow when the right opportunity presents itself.
2. Organisations always have spare budget for data quality initiatives, preventing future embarrassment is often a great business case driver.
3. Strong problem-solving skills and great communication techniques are pivotal for the modern data quality practitioner.
4. Focus on what the business needs from the data as opposed to just quoting stats and figures from your profiling apps.
5. Communicate data quality in a way that resonates with the business.
6. Senior management must be feeling the pain of data quality to fully appreciate its importance.
7. IQ is about business success or failure, not about implementing technical solutions.
8. Technology will assist the IQ specialist but overcoming business obstacles ensures continued funding for IQ.
9. There is no longer a requirement for IQ/DQ specialists to be experts in data quality software and technology.
10. Structured career roadmaps for data and information quality are still rare in most organisations.
11. Be proactive internally, reach out to workers who may be witnessing data quality issues.
12. Start to gather evidence to create a business case.
13. Hold open forums at lunchtime to enable people to share their data quality issues.
14. Distribute data quality questionnaires to middle management and ask them to complete and circulate.
15. Find the senior manager who is connected to the issue and identify the pain involved.
16. Explain how IT isn't the solution, whole business change may be required.
17. The data quality analyst role is a good launchpad for the IQ profession, the IAIDQ will at some point launch a certification scheme to provide practical skills/methodologies that will benefit this role.

About Kathy Hunter

Kathy Hunter has over twenty years information systems experience and more than twelve years experience in Information Quality Improvement. From information quality and data governance through to providing global data solutions and guidance she has attained a reputation for successful delivery in information management to her clients - some of the largest companies in the world.

A popular speaker at events, Kathy is known for her pragmatic approach to information management topics, providing helpful hints and practical examples in order to solve tough problems.

Kathy Hunter Expert Profile on Data Quality Pro

Kynetika Website

Useful Resources

See all interviews

Share Your Story on Data Quality Pro.com

How to Make Data Quality Improvements Stick: Expert Interview With Mark Eaton

The Importance of Change Management: Interview with Mary Gregory

Managing Change - Mary Gregory Revisited

How To Create A Data Issue Assessment Process: Expert Interview With Ken O'Connor

Master Data Management Interview with Charles Blyth of CPP

CIMP Certification Explored: Interview with Dave Wells of eLearningCurve

Interview with Jill Dyché of Baseline Consulting

Interview with Danette McGilvray, author of "Executing Data Quality Projects: Ten Steps to Quality Data and Trusted Information"

Interview with Larry English, Creator of TIQM

Data Quality Through a Metadata Strategy: Interview with Anne Marie Smith

Arkady writes on the importance of monitoring data quality along the information chain and in particular the many data integration junction points in an effort to ensure quality for our ever increasing demand for data.  

Faster or Better: The Ultimate Choice

The Data Needs

We live in the Information Age, meaning that information is our most important and valuable resource. We want more data and we want each piece of data to serve more purposes. Data warehousing is a great example. We have amazing technology in place to store, manipulate, and analyze unthinkable volumes of data. But the data does not magically materialize in the data warehouse. And so, data warehouses routinely get data from scores of source systems through numerous data interfaces.

As if the desire for more data were not enough, we want to get it faster. Monthly batch feeds are the long-forgotten past. Nightly feeds are the norm, and real-time data propagation is often promoted as the way of the future.

Of course, we do not want just any data, but rather we need high-quality data. When the data we get is incomplete, inaccurate, or misunderstood, the warm and powerful Gulfstream turns into a turbulent Maelstrom that turns the data from a great asset into an even greater liability.

The Problem

The problem is that in the real world the adjectives “more” and “faster” are typically in conflict with the equally important “better”. Asking for more data coming faster from more source systems is akin to asking a doctor to see more patients with different health problems while spending less time with each patient.

When confronted with the need to deliver high quality data from source to target systems at the breakneck speed, the data interface designers fall back on a simple philosophy that “the data quality must be managed at the source.” This paradigm, while widely heralded by the data quality professionals is unfortunately misunderstood.

It is certainly true that given a known data problem the best course of action is to perform root cause analysis and fix the problem at its root. This way we do not just reactively cleanse individual erroneous data elements, but rather proactively prevent all future problems of the same kind before they actually occur.

Regardless of the ideal, however, it is not practical or possible to ensure data quality at the source and guarantee that all data coming via interfaces to downstream systems is accurate. There are several important reasons.

1. New problems find their way into the source systems despite the best proactive efforts. The data gathering processes and business processes behind the data are just too complex and fluid to be fully controlled. Further, even with an adequate data quality management program in place, it takes time to identify new problems. Thus, when the interfaces pick up the data too soon the bad data come across before the problems are identified.
2. In reality, most source systems lack adequate controls. Sometimes it is due to ignorance, but often it is a financial decision. It is often deemed that existing data quality is adequate for the purposes for which the data is used within the source system and investing in data quality improvement has a negative ROI. Of course, such calculations ignore the impact of source data quality on downstream systems such as data warehouses, but it is the reality with which we must contend.
3. Data warehouses obtain data from multiple source systems. Oftentimes, the data coming from each source seems accurate when looked at independently from the other sources. It is only when data from multiple sources is put together that the inconsistencies can be discovered.
4. The structure and meaning of the data in the source systems change regularly to accommodate new data collection processes or as a result of system upgrades. Such changes may undergo regression testing within the system, but in most cases no mechanism is in place to test the implications on the downstream interfaces. As a result, the source data remains accurate but is incorrectly processed on the way to the downstream systems, creating an avalanche of data problems.

The Solution

In the world where scores of systems exchange huge volumes of data at breakneck speed, it is unwise to completely relegate data quality management to the source systems. Monitoring data quality in each interface is a necessary part of any data integration solution!

There are different types of data quality monitors. Error monitors look for individual erroneous data elements. Change monitors look for unexpected changes in data structure and meaning. The level of sophistication within these monitors may vary from simple attribute level screens to advanced data quality rules and statistical monitors. More comprehensive monitors will catch more data problems.

Of course, monitoring data quality in data interfaces is not free. Advanced monitors require greater investment of time and money. Most importantly, the more frequent the data feeds the less opportunity for data quality monitoring is afforded and vice versa. The ultimate decision is a financial trade-off. It requires analysis of the ramifications of bad data on the downstream systems, understanding the limitations of data quality monitoring for each level of feed frequency, and making a conscious comparison of the costs of data inaccuracy with the costs of data latency.

One thing is certain. If we continue to exchange more data at ever increasing speeds with disregard of its implications to data quality management, the next epoch will be known as Disinformation Age.

Author Profile

Arkady Maydanchik is a co-founder and Managing Director at eLearningCurve LLC. He is a recognized practitioner, author, and educator in the field of data quality and information integration.

Useful Resources

eLearningCurve - Online education provider for the information management sector

Data Profiling: Myth and Reality

Interview with Arkady Maydanchik of eLearningCurve and Data Quality Group

Data Quality Rules Tutorial: Rules for State-Dependent Objects by Arkady Maydanchik

Data Quality Rules: General Attribute Dependencies by Arkady Maydanchik

Data Quality Rules by Arkady Maydanchik - Tutorial 1 of 4: Attribute Domain Constraints

Data Quality Rules by Arkady Maydanchik - Tutorial 2 of 4: Relational Integrity Constraints

Data Quality Rules by Arkady Maydanchik - Tutorial 3 of 4: Rules for Historical Data

Data Quality Rules by Arkady Maydanchik - Tutorial 4 of 4: Rules for Event Histories

Rethinking Data Quality: The Need for a Data Quality Profession

How to structure a data quality team is a common question for organisations moving up the maturity scale.

As you move from data quality denial, through to fire-fighting and eventually onto a governed strategy you will continuously face the problem of how to shape and scale your data quality resources.

The simplest way I've found of demonstrating this is to apply the concept of an "internal data quality business"

Building Your Internal Data Quality "Business"

Defining the Vision

All businesses must start with a vision that sets a course for the future. 

• What is the goal of the business?
• What are we trying to achieve?
• What does future success look like?
• How do we know we've reached our objectives? 

This is equally applicable to your internal data quality "business" because without a clearly defined, publicly accessible vision it can often be difficult to foster change that everyone can get motivated behind. 

Some real honesty is required here.

If the vision of senior management is to cut staffing levels by reducing scrap and rework waste, then your fledgling data quality business is off to a shaky start.

Also, I would advise that the vision does not just focus on data but on the most important asset your business has - the customer. How will delivering your data quality goals impact the customer? Make sure this is a key component of your vision for your internal "data quality business"

We also need to work out what our mission is going to be for the business, fortunately Steve Sarsfield recently provides a neat article on how to do exactly that.

Next we need to assign roles to our business.

Appointing a Leader

The "CEO" role is an obvious starting point. Who is captain of your particular data quality ship?

Note that your leader does not necessarily need to be a DQ guru. (Many data quality practitioners I know would make terrible CEO's, none more so than myself).

We talked about the need for more data quality change agents recently. These people can make excellent Data Quality leaders as they have can deliver passion, drive and vision to set the business in the right direction.

Balancing the Books

Next we have the Finance Director. They are responsible for monitoring and reporting the financial performance of our business and helping the leader devise tactics and strategy.

In a data quality this is one of the most important roles but so often ignored.

A good data quality "CFO" will ensure that any service offered by the business is routinely monitored for bottom line gains so that the "shareholders" (ie. the sponsors and project customers) are receiving ongoing value for their investment.

Regardless of whether your sponsor asked for a business case, I always advise teams to create one anyway. A business case provides a "guiding light" for you to determine whether your efforts are still aligning with the business. Plus, you never know when you might get dragged into a budget meeting to justify your existence!

Managing Delivery

Next we have the "COO" of our internal data quality business. They are tasked with ensuring smooth delivery of service so this includes the initial fulfilment and any ongoing service assurance.

This may typically be someone from a more senior data quality background, a person who can lay down training procedures, coaching/mentoring frameworks, service designs, methodology structures and manage the individual service delivery units.

Promoting the Business

Next we have one of the most interesting roles - "Head of Sales & Marketing".

This person must spread the word and cross-sell the value of your data quality business to the wider organisation.

This is really an educational role in many ways. Most sales people in the data quality sector will tell you that they need to provide an element of education before they can make a sale and if you're looking to expand your data quality business internally then you will have to develop a comprehensive marketing, education and sales plan.

Need some ideas for tools? Here are 7 productivity tools that will help in this role.

Innovating Technology

The "CTO" will be responsible for matching the operational needs of the team with the appropriate technology.

They may have to provision custom built solutions or buy COTS products from the marketplace in order to help the business scale up and deliver the same level of high quality service.

Investing in People

The "Head of Human Resources" will be responsible for provisioning training and hiring suitable team members. They will be instrumental in attracting and retaining the best data quality professionals.

Benefits of "Data Quality Business Thinking"

I use this analogy a lot lately and I guess it's been driven by the credit-crunch and the need to demonstrate value and a professional approach.

I still witness data quality teams being disbanded and projects shelved because of a lack of perceived or realised value. I really believe that if you view your data quality team as a business entity it forces the agenda that you have to deliver value for money, every day.

What if your team only has one or two people?

It doesn't matter, you still need to assign these roles. My personal data quality media and consulting business has at any one time only 2-3 people actively involved but I still map out the various functions either to myself or other partners.

Even if your data quality team is just you, by thinking of "you" as a business that has to market, sell, deliver and sustain I think this is a far healthier approach than just thinking of yourself as the "data quality consultant".

It really pays to structure your data quality team as a capability that is so well organised and value driven it would be crazy to can it, no matter what the economic climate.

Next Steps

Think about your team - who is a natural leader? Who can define standards and frameworks for delivery? Who performs well in presentations? Who has the patience to document training manuals? Who is the "techie geek" who understands data quality technology? Who coaches new team members? Who calculates the value of what you deliver as a team? Who helps to sustain and monitor past deliverables? Identify the strengths and weaknesses of your team and make this an empowering exercise for team members to step up and think about what each role requires.

What do you think about "data quality business thinking"? Please add your comments below.

Useful Resources

7 Productivity Tools for the Innovative Data Quality Leader

WANTED: Data Quality Change Agents

What are your data quality goals for 2010?

8 Tips for Making Your Data Quality Resolutions Stick in 2010

Using Metrics to Assert a Business Case for Data Quality

WANTED: Data Quality Entrepeneurs

Creating An Internal Data Quality Community: Introduction (Part 1 of 4)

15 Tips for transforming knowledge-workers into a data quality task force

10 Tips to help data quality professionals boost their career prospects in the downturn

5 Simple Techniques To Differentiate Your Data Quality Service

5 simple activities to help sharpen your data quality sales performance

In this article, I want to explain why that is not necessarily the best approach and what you can do to create the perfect strategy for attaining your data quality goals.

Are You "Cookie-Cutting" Your Data Quality Strategy?

I routinely monitor the inbound search engine keywords for Data Quality Pro and it's clear that many people are looking for past data quality strategy documents to help shape their own projects. I also get occasional requests for project plans, strategy documents and other materials from people who are trying to shape future data quality strategies for their own projects.

One of the first interviews we published on Data Quality Pro was with Mandy Mackay who was then at the New Zealand Ministry of Justice. The interview also included the data quality framework that she had helped develop at the ministry and it became an incredibly popular download on the site for many months, still is in fact.

Clearly, there was a lot of value in this document and much of it could be adopted to other sectors, particularly local government.

However, documents like these are often very specific to that particular organisation and the unique data quality challenges they face.

Having worked both in consulting firms and on behalf of clients who are reviewing tenders from prospective consultancies I've often seen this issue. The consultancy creates their standard framework for data quality management and puts it forward as a showpiece on any new client proposals. The client, witnessing the depth and completeness of the framework may in fact grow confident that this must be the partner of choice for them but again, this is not a data quality strategy. This is typically a framework or blueprint for a generic process, not the strategy that is unique to that clients specific problem.

The best way to explain this is with an analogy of war.

Consider your data quality project as though you're fighting a war. The resources and processes found within your Navy, Army, Air force, medical units, administration divisions and all the additional governance that goes into running an army can be seen as your data quality framework.

This is very different to your strategy for war. A war strategy has to answer many questions:

• What is the desired outcome - how will we know we have won?
• What are the plans for action?
• How do these plans connect together and do we have any fallback plans?
• What is the timeline for each plan?

A military leader would be unwise to enter into a war simply using the same strategy they have adopted in previous campaigns. Each situation requires a unique perspective.

There is a similar danger here of simply adopting a data quality framework from a renowned expert and converting this into your data quality strategy.

To your peers and seniors a published framework may look detailed and thorough but is it right for your organisation?

Having read practically all the leading data quality books, and indeed spoke with many of the authors, it's interesting just how much their opinions and "strategies" differ.

This should be a warning sign, there is no "paint-by-numbers" strategy for data quality.

In fact, most data quality project strategies are doomed to failure before they even begin for this very reason.

They are trying to "paint by numbers" when they don't know what the numbers are yet.

How to Create a Bespoke Data Quality Strategy (Or How to Navigate Through Fog)

One of the most useful project management books I have read is Perfect Projects by Eddie Obeng (thanks  to Andy Brooks for switching me on to Eddie).

Central to the book is the premise that projects, IT projects in particular, will fail if they don't apply the correct project strategy to the type of change being implemented.

Most project strategies that I witness in terms of data quality have followed a fairly linear path. "We'll do A, then B, then C, then D, which will of course cost no more than X, delivering benefits Y and will delight sponsor Z ".

The fact is that for most organisations, unless you're way up the information quality maturity scale, this linear strategy is a recipe for disaster because you lack the experience of delivering these projects.

Eddie Obeng actually creates a formula for project success that may help you benchmark your own project -

Change with the right PURPOSE, matched to the PROJECT TYPE, kept on track through effective LEARNING and REVIEW, set up with good PLANNING and COORDINATION, plus effective STAKEHOLDER MANAGEMENT, shared amongst the TEAM, multiplied by (your) effective LEADERSHIP, equals SUCCESS.

Eddie Obeng, Perfect Projects

In the book, Eddie Obeng walks through each of the highlighted issues but the one most companies fall down on is adopting the wrong strategy for the PROJECT TYPE they are involved in.

Obeng states that projects are simply "chunks" of change but these chunks differ depending on the level of learning and experience we possess at the start of the project.

In a perfect scenario, according to Obeng, we would know:

Why we are doing what we are doing, what precisely the outcome will be, and understand how it is to be carried out - including the methodologies and technologies required.

Obeng points out that most projects fail this criteria and instead fall under the following categories:

• Clear goal - missing method
• Clear goal and method - lack of clarity on outcome
• No goal and no method and project exists because "something must be done"

We now have 4 different types of change which Obeng classifies as:

• Paint by numbers
• Quests
• Movies
• Fogs

So where most organisations are focusing on a paint by numbers strategy, they should instead by adopting a quest, movie or fog strategy because they have not yet developed the required learning and experience.

In many cases I believe organisations should be adopting a fog based strategy because they don't truly understand the direction and method required. Even if the method has come from a "guru", if it isn't based on learning and experience from their particular situation it may not be suitable.

They're typically reacting to a perceived need because "something must be done" about the data quality issues they are facing and the true vision and goals of the project are often poorly understood as a result.

The approach for a fog style project is to move forward in cycles of learning and review, much as you would do if lost in the fog.

You must inch forward testing the terrain and reviewing your strategy based on what the conditions are telling you. At each step you develop the learning and knowledge required to help you plan the follow-on phase.

In short, you are creating an iterative strategy that is far more likely to guide you to a successful outcome than a paint by numbers approach that can get you hopelessly lost because you're following the wrong method or the goal lacks clarity.

With a fog approach you are constantly asking the question - what is the true goal and outcome we require and what specific methods (based on our recent discoveries) will get us there?

(For detailed instructions on how to manage fog style projects and the other styles mentioned above then obviously purchase the Perfect Projects book, it's a sound investment).

I'm sure if you presented a business case to your sponsor that likened your strategy to wandering around in the fog you would soon be ushered/thrown out of the room.

However, in many cases this really is the best approach for companies who are starting out with data quality.

Yes, there may be fear and apprehension from the sponsors because you lack a complete set of steps from "cradle to grave" so in this situation you need to explain that this approach is likely to deliver far less risk and could in fact save money and time through a more pragmatic, staged approach where the strategy evolves and adapts based on what you discover along the journey.

Next Steps

When you next come to build your strategy for a data quality initiative, take a step back and ask some key questions:

• Do you really know the precise strategy at this stage?
• Are all the steps laid out perfectly before you so you can connect the dots and deliver a complete vision?
• Do you know exactly what you are doing, why and how every single task has to be carried out?
• How are you defining the methods for the project? Is your methodology adapted from frameworks in published texts or based on your learning and experience of the present situation?
• What are the real outcomes for the project? Are they known? Have they been validated?

Answering these type of questions will help you define the type of project you're involved in. If it's a paint by numbers style project, fine, create a linear strategy and go for it. However, if there is uncertainty and confusion on goals, method and outcomes then look at the alternative approaches provided above.

Only when you understand the type of project required can you start to develop the right data quality strategy for moving forward.

Useful Resources

Pentacle Website (they now publish other books by Eddie Obeng for free on their website, well worth a read)

IQM-CMM: Information Quality Management CapabilityMaturity Model, interview with Author Sasa Baskarada (useful for measuring your maturity and understanding the level of experience/learning in your organisation)

The Data Governance Maturity Model.pdf (download supplied from DataFlux in our resource centre, again useful for identifying your maturity)

How to create a data quality framework or data qualitymethodology:Essential resources to get you started (if you do want some templates for understanding the methods of data quality then this can help but be sure to heed the lessons above, develop a strategy that works for your organisation)

Creating a data quality framework? Learn how the NZ Ministry of Justice created their DQ framework and download the final framework document.

WANTED: Data Quality Change Agents

If you’re looking to mature your in-house data quality capability then at some point you will most likely need to recruit fresh resources. This is no mean feat, I speak from battle-hardened experience.

Despite the massive data quality problems that pervade the corporate and public sector there is still a serious shortage of capable data quality professionals on the open market. On several occasions, having hired seemingly exemplary candidates, I’ve later realised that sadly it just isn’t going to work out. The candidates just lacked that certain “spark” or “drive” to take the organisation forward on its data quality mission.

Over time I realised that the problem lies not with the traditional data quality skillset of the candidate but with the desire and capability of the recruit to invoke change.

In short, we often look for technicians but what we really need are change agents.

What does a data quality change agent look like?

Here we meet the first problem – data quality change agents are a rare and secretive breed, often laying dormant amongst us for many years.

Data quality change agents typically disguise themselves under a different role, often with no obvious connection to data quality.

Business analysts, software developers, database administrators, supply chain managers, call centre supervisors – there are no practical restrictions on who can evolve into a data quality change agent.

Age and rank are often irrelevant also. (I actually became a fledgling data quality change agent as a completely green 21 year old software developer fresh out of University, I told you data quality skills are not always vital!)

No, you won’t find a data quality change agent on any corporate employee register but chances are your organisation has several of them right now, ready to step forward, if you create the right environment for them to shine.

Data quality change agents are not defined by their title but by their attitude.

There is little difference in people, but that little difference makes a big difference. That little difference is attitude. The big difference is whether it is positive or negative. 

- W.Clement Stone

On being positive

The first thing you notice about change agents is that they are positive as individuals. They seldom look to the past, to point blame and to lament their (and others) failings.

In our silo-driven corporate culture it is far too easy to lay the blame for data quality on the “other department” or “the folks in IT” who supply the data and software we use in our daily roles.

Change agents do not question who is at fault. However they do question, and that is another of their most important characteristics.

Status Quo - There to be questioned

Take the typical information chains that flow through your organisation. I would hazard a guess that many of them have been cobbled together with scant regard to data quality management and business performance.

These information chains are so often the breeding ground for change agents. Frustrated with hearing “this is just how we do business here” they seek to question, probe, challenge and test the way things are done.

Change agents are a stubborn breed and they need to be because it can often take many months before they get a chance to demonstrate their next skill.

The art of persuasion

We thus have a kind of see-saw: first, pure persuasion leading to the conversion of a minority; then force exerted to secure that the rest of the community shall be exposed to the right propaganda; and finally a genuine belief on the part of the great majority, which makes the use of force again unnecessary

– Bertrand Russell

Okay, perhaps exposing your organisation to data quality propaganda is somewhat extreme (or is it?) but Bertrand Russell sums up nicely how the change agent operates. They focus on converting a small community of individuals in the organisation who also have passion, persuasion and a positive attitude to changing the status quo.

Here we have the early beginnings of a data quality culture forming.

Next this smaller community begins to promote their findings to the wider organisation, providing irrefutable evidence of the need for change. Ultimately this leads to the entire organisation evolving their data quality culture into one of "this is how we do business here".

Along this journey of persuasion there is another trait that the change agent displays, it's subtle but very profound. They typically look to innovate, not simply improve.

Innovation vs Improvement

Far too many organisations focus on simply "patching up" the information chain and performing minor step changes. This is not the goal of the change agent.

Change agents are often far more visionary and entrepreneurial in their attitude to change. They seek to positively disrupt and add far more value when compared to conventional improvements.

They are often unconcerned with the impact of failure on themselves personally and once afforded the opportunity to innovate they are seldom comfortable with "toeing the line" or meandering through their working life.

In cultures that thrive on preserving the status quo this frequently creates conflict which leads us to our final character trait.

Focus on achieving (continually)

Data quality change agents don't just want to make up the numbers. They want to achieve their goals. Note how I use the plural here.

Delivering one project seldom satisfies the inner passion of the data quality change agent. They want more and that can often lead to problems as organisations can sometimes fail to evolve at the pace dictated by the change agents' drive and passion.

Why Your Company Needs a Data Quality Change Agent

The short answer is simply that "they get things done".

The longer answer is that organisational cultures, both in the corporate and public sectors, are often designed to prevent change.

I can think of very few companies who have openly embraced the kind of upheavals that data quality maturity demands.

Change agents are pivotal for helping an organisation steer a course from data quality apathy to maturity.

You can hire all the skilled data quality consultants, technicians, data gurus in the world and throw the most advanced software at them but I truly believe real data quality change requires individuals with the kind of qualities presented in this post.

My advice for leaders in 2010 is to help create the environment within your organisation where change agents can emerge, develop and advance. If you are a leader or decision-maker reading this article I urge you to seek out these people at all costs because they hold the key to your long term vision of data quality success.

Tip: If you want to read about the life and times of a data quality change agent in action, drop by Jill Wanless' blog "Data Quality from the Ground Up", she is creating a valuable insight into this challenging role.

What other characteristics do you feel are important for an effective data quality change agent? Do you agree or disagree with the content of this article? Please add your comments in the section below.

Useful Resources

How to Make Data Quality Improvements Stick: Expert Interview With Mark Eaton

Managing Change - Mary Gregory Revisited

The Importance of Change Management: Interview with Mary Gregory

Introduction to Guerilla Data Governance: An interview with Mike Meier

How to create a data quality competency center: Expert interview with John Schmidt

Creating An Internal Data Quality Community: Introduction (Part 1 of 4)

Creating An Internal Data Quality Community: The Benefits (Part 2 of 4)

Creating a Data Quality Community: The Launch (Part 3 of 4)

WANTED: Data Quality Entrepeneurs

Are you managing change in your data quality initiative?

Struggling to sustain improvements in your data quality programs? 10 recommendations for organisational success

Are you sustaining change in your data quality initiatives? Lessons from leading change management specialists.

Insights from data quality professionals who have attained senior roles - Andrew Brooks (#2 in series)

Interview with Daragh O'Brien

Scaling the heights: Data quality professionals give their insight into what it takes to climb the data quality career ladder to the very top (#1)

2009 has witnessed a prolific rise in blogs relating to data quality, data governance and MDM matters. This round up highlights some of the "old battlers" who were there throughout the year and some of the "new recruits" who have joined the fray.

Thank you to everyone who got involved in 2009 and contributed their experiences, expertise and insights to make it a truly memorable year for our respective professions. 

	The World is Addicted to Data (and that's good for us) : Steve Sarsfield (@stevesarsfield) comments on the prevalence of data in our personal as well as corporate lives and what this means for those in the data quality profession.
	Data Chaos and Five Truisms of Data Quality : Phil Simon (@philsimon) launches his first post on the DataFlux community of experts with a great post looking at some of the common traits of organisations that fail to address data quality.
	Adventures In Data Profiling (Part 8) : Jim Harris (@ocdqblog) concludes his vendor neutral series on data profiling with a useful summary of what was covered and the value gained. Over on the DataFlux Community of Experts, Jim posted "The First Law of Data Quality" - urging us all to question the where, why, when, who, what, and how of our data usage.
	Education – a must have in Data Governance : Charles Blyth (@charlesblyth)examines some of the different forms of education that are critical for maturing data governance. Charles also kicked off his DataFlux Community of Experts channel with "Finding a home for MDM" - exploring the relations between MDM and BI.
	Making Data Governance as simple as possible, but not simpler - Part 2 : Dalton Cervo (@dcervo) with  a follow up to his earlier post discussing a proposed Data Governance Component Model.
	Business Intelligence without Data Quality : Julian Schwarzenbach (@jschwa1) presents an argument in this post - can BI exist without data quality and data governance?
	Principles of Data Governance : Mike Meier calls for a larger, more comprehensive system of data governance.
	Are we ready for all this data? : Rich Murnane (@murnane) poses some tough questions for organisations who are ramping up data volumes without care for data quality.
	Dear Mr Other ... : Graham Rhind ( @grahamrhind) provides some simple examples of how even data quality vendors can infuriate their customers with poor quality data.
	MDM Data Quality Processes : Lawrence Dubov of Initiate (@InitiateSystems) explores some of the techniques involved in delivering data quality through hub approaches.
	Information Quality – Every Little Helps : An interesting post from the IAIDQ IQ Trainwrecks site (@iaidq) that recounts the tale of how UK supermarket giant, Tesco, paid considerably more than the list price for a set of bicycles.
	Phony Phones and Real Numbers : Henrik Liliendahl Sørensen (@hlsdk) provides an overview of some of the data quality issues that can beset telephone numbers.
	Data Quality: Classify and Describing : Jackie Roberts (@jackiemroberts) provides some practical advice on setting up an Identification Guide drawing heavily from the ECCMA classification.
	The Prevention Paradox - Part 1: David Loshin (@davidloshin) presents an interesting result of adopting a rigorous quality strategy.
	Ways to 'Communivate' your Data Issues : Jill Wanless (@sheezaredhead) kicks off a Purple Cow approach to data quality by discussing how she communicates in an innovative way to create exposure for her continuing data quality efforts.
	You Build it, You Break It, You Fix It: Why Applications Must Be Responsible for Data Quality : Evan Levy with a great post that discusses the role of applications in data quality and why they should raise their game.
	Context For Data Quality : Vish Agashe (@vishagashe) opens up a debate exploring how context can be tracked around data quality issues to help with prioratization as well as justifying ROI.
	IAIDQ Information Quality Blog Carnival : Daragh O Brien (@daraghobrien) provides a roundup of the December IAIDQ blog carnival.

If I missed anyone, please drop me a line.

Useful Resources

Find all posts in : DQ Blog Roundup

• El Festival del IDQ Bloggers (promote your DQ posts or why not host it on your blog?)
• Data Quality Blog Roundup (November 2009) Edition
• Data Quality Blog Roundup (October Edition)
• Data Quality Blog Roundup (July 2009 Edition)
• Data Quality Blog Roundup (June 2009 Edition)
• Data Quality Blog Roundup (May 2009 Edition)
• Data Quality Blog Roundup (April 2009 Edition)
• Data Quality Blog Roundup (March 2009 Edition)

How can we help you reach those goals?

If you take a few minutes to share your goals and the areas you wish to improve I'll make sure the Data Quality Pro team focus their efforts on creating useful content, debate and support throughout the year to help you achieve these aims.

To share your goals simply add a comment to this post in the section below or add your thoughts anonymously to this quick survey (no more than 2 minutes of your time and I promise to convert each response into an action).

The more I know about your data quality goals and challenges, the more I can help steer Data Quality Pro resources in your direction. Your feedback really is critical to the content we produce here.

Here are just a few of the goals I've set myself to research and document in far more detail this year, what are yours?

• Deliver Insight into the Value of Data Quality: I want to research how organisations are successfully demonstrating the value of data quality. What is working right now given the economic situation? What triggers can be used for specific types of sponsor? What can we learn from people like Jill Wanless and her "guerilla" tactics? I think as a profession we need to explore and understand this topic in far more detail. My to-do list has a string of features and articles but what is your take? Perhaps you have some sage advice to offer and would like to contribute?
• Provide Techniques For Improving Data Quality Focus: Data volumes are going up but data quality budgets rarely follow suit - focus and prioritisation are vital techniques which we need to cover in far more detail. We need to get smarter at deploying often limited resources on the data quality priorities that really matter.
• Understand the Best-Practices for Data Quality Longevity: How can we sustain data quality improvements so we really gain the benefits for the long term. If we can create more stability in our careers then that is also a great goal to aim for. What techniques are companies adopting to keep the ball rolling with data quality? Who is doing it right? How do they structure their people and resources to make data quality a way of life across the enterprise? Lots of areas for exploration here so definitely a key goal for exploring throughout the year.
• Foster Greater "Connectivity and Accessibility" Within the Data Quality Profession: These are two goals but they are closely linked. The goal for Data Quality Pro has always been to make data quality knowledge far more accessible. I think in 2009 we went some way to achieving that but with DQ Directions (the new online DQ conference format), we hope to raise our game and create a truly global platform for professionals to demonstrate their expertise and make far more useful connections.

What about you? What data quality goals and challenges will you be facing this year?

Please add to the comments below or submit your goals to this survey so we can provide the right level of educational content for the year ahead.

Useful Resources

8 Tips for Making Your Data Quality Resolutions Stick in 2010

7 Productivity Tools for the Innovative Data Quality Leader

The evolution of the data quality recruitment market - where are we now and what do data quality professionals need to do next?

5 Resources For Finding Data Quality Jobs

Are your data quality skills up to scratch? Find out what employers are really looking for in a data quality analyst

15 Tips for transforming knowledge-workers into a data quality task force

10 Tips to help data quality professionals boost their career prospects in the downturn

Perhaps you are looking to improve your data quality practices next year but are concerned about how to make those habits stick?

This article gives you a list of possible data quality habits to focus on and a simple methodology for making sure you and your team stick to them.

Advertisement 

8 Tips for Making Your Data Quality Resolutions Stick in 2010

One thing I have learned over the years with data quality is the benefit of adopting simple habits that an entire team can adopt and get motivated behind.

Data quality often seems complex and difficult to tackle because there are often so many problem areas to address but the fundamentals are actually very straightforward. We often make it far more challenging than it really needs to be.

I find that by implementing "bite-size" techniques that can be embedded incrementally as habits is a far better approach compared to major programmes where too many initiatives are thrown at the workforce.

But what are some of these simple, best-practice habits?

Here are some of the features published on Data Quality Pro this year that may give you inspiration for forming new data quality techniques in the new year:

1. Do you struggle to create a compelling introduction to your DQ proposition?
2. Suffering from feast or famine in your DQ business or career? Learn how to network and promote yourself more effectively
3. Data Profiling for Beginners
4. 20 simple tips to spice up your data quality blog
5. 5 simple activities to help sharpen your data quality sales performance
6. Data Quality Rules Analysis Using Data Visualisation Tools: Tutorial #1 Historical Data
7. Essential data quality skills: Information Chain Management
8. 8 Tips for vetting data quality service providers
9. 8 Simple techniques to improve data quality root-cause analysis
10. "Information tagging" - the simple tool that can eliminate scrap and rework in data-driven processes
11. Lean techniques to help your data quality improvement initiative (Part 1: Time Value Maps)
12. Lean techniques to help your data quality improvement initiative (Part 2: Little's Law)
13. Data Profiling Tutorial: Discovering Dependency Rules
14. 10 Tips to help data quality professionals boost their career prospects in the downturn
15. Identifying Duplicate Customers (Part 1) : (Part 2) : (Part 3) : (Part 4) : (Part 5)
16. 15 Tips for transforming knowledge-workers into a data quality task force
17. How to create an online data quality rules repository
18. Reducing the need for scrap and rework with web data collection
19. Creating An Internal Data Quality Community: Introduction (Part 1) : Part 2 : Part 3 
20. How To Deliver A Compelling Data Quality Business Case
21. 7 Essential Skills for Effective Data Quality Leaders
22. The Importance of Change Management
23. ETL and Data Test Guidelines for Large Applications by Wayne Yaddow
24. How to set data quality goals any business can achieve
25. 5 Simple, Low Cost Ways To Improve Data Entry Data Quality
26. 7 Productivity Tools for the Innovative Data Quality Leader
27. Using Metrics to Assert a Business Case for Data Quality
28. How to Make Data Quality Improvements Stick: Expert Interview With Mark Eaton
29. Data Profiling Tutorial (With Free Data Quality Software) - Lesson 1 : Lesson 2

How do we make these new habits stick?

So once you have identified a desirable habit and included it in your data quality resolutions for the new year - how do you make it stick?

Inspired by Zen Habits, I've adapted a simple methodology for helping you to see those resolutions through another year:

1. Make a commitment. The key to commitment is going public. If you are going to improve a data quality habit then let the whole team know your aim and get others involved. Don't approach this half-hearted. In the words of Yoda - "Do or do not, there is no try".
2. Practice. Get used to carrying out your new data quality habit regularly. The only way you can embed a new technique or best practice is to adopt it daily where possible. Get comfortable with the new technique so it becomes second nature.
3. Motivation. Any kind of change can be de-motivating. Recognise that changing habits and creating new working patterns can be tough. Look at the rewards tip below for some additional pointers but make sure you have a motivational strategy in place.
4. Tracking. Log your progress. This will not only help with your motivation but if you can tie in some other business metrics then it can demonstrate the value that adopting better data quality habits can offer to the business. Also a useful tip for demonstrating what value you provide to the organisation, especially useful when your appraisal comes around.
5. Support. Create a support group and share the experience. It's easier to change behaviours and adopt better ways of working if you operate as a group. Why not create a blog and share your experiences. Many bloggers draw support from the global data quality community. (Data Quality From The Ground Up blogger Jill Wanless springs to mind here).
6. Rewards. Have daily, weekly and monthly rewards. If your team stick to a best-practice for a whole month then have a group reward. It is critical that you motivate yourself, particularly in those early days when it is so easy to slip back into those bad data quality habits.
7. Focus. For the first 30 days focus solely on one habit, far better to succeed at one habit than fail at many. Once the habit is "bedded in" it will get easier to focus on new challenges but initially you may need a great deal of focus and persistence to get comfortable with this strange new world of doing data quality the right way.
8. Positive thinking. Going cold turkey (seasonal pun intended) and shrugging off those old working practices is not easy but remember why you're adopting these new habits, they will benefit you and your colleagues if you stick through the process of habit forming. Habits help you develop stronger skills and abilities that in turn earn you greater career opportunities so plenty of reasons to be positive for 2010.

What data quality resolutions are you looking to adopt in 2010? Why not share your views in the section below.

Best of luck with your data quality resolutions for 2010 and thank you to everyone who has supported Data Quality Pro this year.

To find out more about this scheme and how it may benefit data quality professionals we recently interviewed the Director of Education at eLearning Curve, Dave Wells.

Advertisement 

CIMP Certification Explored: Interview with Dave Wells of eLearningCurve

Data Quality Pro: Please introduce yourself Dave and explain your role at eLearningCurve for the benefit of our readers.

Dave Wells: I'm the Director of Education for eLearningCurve. What that means is that I plan the curricula and seek out experienced instructors and high-quality courses for our education program. It is also my responsibility to define and shape the certification program. This is an ideal role for me because of my experience and interests. I've had a long career in information management, with my first IT work as a programmer starting in the late 1960's. In the 40+ years since then I've worked at one time or another in many different information management roles, ranging from technical to management. Then in 2001 I changed from being an IT professional to become a business manager. This is significant because I made the shift from provider of information services to consumer of information. What an eye-opener! I still don't know whether I came from "the dark side" or went to it. But I do know that it changed entirely my perspectives about information management.

I'm also a long-time IT and professional educator. I've been teaching, developing courses, and managing education programs since the 1980's. For those who know The Data Warehousing Institute (TDWI), I was the TDWI education director for several years. It was there that I first explored certification and developed the Certified Business Intelligence Professional (CBIP) program. I know that certification advances careers, and I still believe in the value of CBIP. But I know that BI is a narrow slice of information management. I'm excited to have a lead role in CIMP and to deliver a program that will bring certification to many more information management disciplines.

Data Quality Pro: Briefly describe the goals of CIMP, in particular how will certified professionals benefit?

Dave Wells: CIMP really serves three distinct groups - (1) the job seekers who are unfortunately a large population in this economy; (2) the career builders who want to reach new horizons as an information management professional, and (3) hiring managers and recruiters who are looking for ways to distinguish top talent from the rest of the crowd.

From the perspective of the first two groups - job seekers and career builders - the most apparent benefit is a credential that affirm your knowledge of a specific discipline. A credible certification (which is certainly the goal of CIMP) is based on difficult exams that require real study. When you achieve this level of certification, it says to prospective employers that you have the energy, drive, and commitment that is necessary for professional growth in a continuously changing field. Several studies show that certification makes a difference in salary, promotions, and career opportunities. It is also important to recognize the difference between product-specific certifications and tool-independent certification. The first affirms your ability to use a software tool; the second attests to your ability to understand concepts, analyze problems, and devise and implement solutions. Clearly, the latter is more valuable as the technology continues to change but these fundamental problem-solving skills are always in demand. CIMP is specifically designed to fit into the second and more valuable category.

From the perspective of hiring managers and recruiters, certification is particularly valuable to offer them access to a community of information management professionals who have demonstrated drive, commitment, knowledge, and ability. What recruiter seeking to hire information management professionals would not take advantage of a resource such as a CIMP registry?

Data Quality Pro: What are the first IM disciplines that will be available for certification and in what timescale?

Dave Wells: We'll begin with Data Quality for two reasons: First, because it is a certification that is much needed in the industry, and second because we have a robust line-up of courses that help to prepare for certification. The January 2010 launch also includes two other disciplines where we have curriculum ready to support certification - Data Governance and Master Data Management. Both are hot topics that are getting lots of attention. Shortly after the January launch (certainly within the first quarter) we will also have Information Management Foundations and Data Modeling certifications ready to go. Then later in the year we'll introduce certifications in Business Intelligence and Data Warehousing.

Data Quality Pro: How will the examination process physically be delivered?

Dave Wells: Exams are taken online using your web browser and eLearningCurve's learning management system. Each exam consists of twenty to thirty questions that vary in format and difficulty. The set of questions that are selected is unique to each exam. You can be assured that you will not receive all multiple-choice questions and you will not receive all "easy" questions. Exams are timed, expecting an average of two minutes per question to respond.

Data Quality Pro: What are the requirements for becoming CIMP certified?

Dave Wells: The requirement is to pass a total of five exams, each with a score of 70% or better. The five exams must include the fundamentals exam for the chosen discipline and three others from that discipline. The fifth exam is an elective that may be selected from among all of the exams available in any of the disciplines offered at eLearningCurve.

You may, of course, choose to earn certification in multiple disciplines. When you do so, some of the exams will apply in more than one of the disciplines.

Data Quality Pro: What sort of costs are involved?

Dave Wells: The base cost is $450 which includes an application fee of $150 plus five exams at $60 each. When you pay the application fee prior to purchasing exams, you are able to purchase the exams at a discounted price of $45 instead of $60 which brings the base cost down to $375.

It is likely, of course, that you'll want to take courses before taking the exams. Assuming five courses at an average cost of $325 each, the cost of education is $1675. The combined cost of education and certification with discounted exams is $2000. That's really quite a bargain when you consider that it includes five courses plus certification - especially when compared to a typical conference with registration costs approaching $2000, travel costs additional, and no certification opportunity. 

Data Quality Pro: Is there a pre-requisite to complete the ELC curriculum to sit the exam or will prior experience be sufficient?

Dave Wells: There is no requirement to take courses before exams. In most cases, however, I do strongly recommend the courses. The exams are difficult and challenging. Only the most experienced people are likely to pass without taking courses. Even then, they may not pass. Taking tests is a unique skill in itself. Some of us do it well while others struggle. It will take breadth and depth of experience, and being a good test-taker, to pass without taking the courses first.

Data Quality Pro: What benefits do certified professionals receive to help them develop their career and future opportunities?

Dave Wells: The certification itself has intrinsic value in career development. Numerous studies show that certified professionals find better job opportunities and receive higher pay than those without certification. Each person who earns CIMP is able to demonstrate that designation in several ways. Obviously you'll receive a certificate. In addition you receive a credentials letter in PDF format that describes the certification and the effort and demonstrated knowledge that is required to earn it. The credentials letter has much greater impact when attached with a resume or job application than a photocopy of the certificate.

CIMP holders are authorized to use the designation on business cards and correspondence, social networking profiles such as LinkedIn, personal and professional websites, author and speaker bios, and similar circumstances. You are also entered into a CIMP registry that is accessible to employers, recruiters, and agencies seeking talented Information Management Professionals.

Further benefits include discounts from eLearningCurve affiliates. For career advancement, perhaps one of the most valuable is from IT Resume Service, offering a complimentary resume review and 15% discounts for resume writing or career assessment services. Professional career development services combined with meaningful certification can really make a difference. To find out about this offer, visit www.itresumeservice.com.

We also recognize the value of networks, networking, and continued learning for career growth. We are developing several benefits in these areas including online "Ask the Experts" forums with eLearningCurve instructors, a LinkedIn group exclusively for CIMP networking,  and topical webinars at no cost to support the continued learning goals.

Data Quality Pro: Is there a re-certification process and if so what is involved?

Dave Wells: Recertification is essential in a rapidly changing field such as Information Management. CIMP requires that you recertify once every three years. There are a variety of ways to achieve recertification, each intended to show continued activity - work, study, and learning - in the field.  Work experience, continued education, writing, speaking, and teaching all earn credit toward recertification. Any of these activities or a combination of several can meet the requirements.

Data Quality Pro: Where can we find out more about CIMP or post further questions?

Dave Wells: You'll find lots of details at www.eLearningCurve.com - just click the Certification link. If you don't find the answers to your questions there, just send me an email at david.wells@elearningcurve.com.

Useful Resources

Review: Data Quality Education From eLearningCurve

Interview with Arkady Maydanchik of eLearningCurve and Data Quality Group

DQ Directions 1 will launch in Q2 2010 and finally offer a global platform for delegates to learn, share and connect. 

DQ Directions Conference - Call For Presenters

In Q2 2010 Data Quality Pro will host the first DQ Directions online event, the next generation in data quality conferences.

We have taken the very best elements of traditional offline conferences and created a unique online resource that provides incredible benefits for attendees and presenters alike.

Benefits for attendees:

More Presenters. Offline events have limited capacity for presenters but the online format means no limitations on presenter counts, we can therefore showcase far more expertise.

Globally Accessible. Online hosting means you no longer have to worry about the costs and disruption of travel - we bring high-value presentations straight to your desktop.

Affordable. Offline events are expensive with hotels, travel and event tickets making it difficult for many people to attend. DQ Directions eliminates these worries with a low-cost, affordable membership plan.

On-Demand Viewing. Offline conferences frequently create presenter clashes as you can't be in two places at once. With DQ Directions the attendees can view ALL the presentations.

For presenters, there are also many great benefits: 

Time Efficient. You pre-record the presentation in advance of the conference membership site go-live, this means no requirement for travel or time away from paying clients/employers.

Larger Audience. We are expecting far larger audiences for the first DQ Directions event when compared to the traditional, offline alternative.

Perfect Presentation. Your presentation will be recorded in advance using screen capture software and high quality audio, multiple re-takes are available if required.

Globally Accessible. The benefits of online hosting also apply to presenters, we can now showcase experts who wouldn't traditionally be able to attend an event.

Would you like to submit a presentation proposal?

We already have a confirmed line up of recognised thought-leaders, authors and highly experienced practitioners so why not join them and demonstrate your expertise?

Here are some other reasons to submit a proposal: 

• Remember that there are no expenses involved, we record your presentation using high-quality web capture technology at a time that suits you and well before the membership site opens to delegates.
• Topics will focus on data quality, data governance and master data management so there is a broad area for content.
• We will also release a number of free event passes for you to share your presentation with clients or colleagues.
• In addition, presenters will be given their own media portal to deliver your presentatio, showcase your experience and engage with the event delegates.

To submit a presentation proposal please follow this link: submit a proposal for DQ Directions 1.

How will the event be delivered?

Over the coming weeks we will record the presentations from leading experts in the field of data quality, data governance and master data management.

These videos will be hosted on DQ Directions in a custom designed media portal for each expert, enabling far more engagement and interaction with the attendees than ever before.

Unlike a traditional offline event, DQ Directions only requires a low fee to grant you long-term access to each event meaning you can study each presentation repeatedly.

We are planning several events per year to help attendees develop regular insights into the techniques, technology and approaches that are shaping the directions of the data quality profession.

Got a question about DQ Directions?

Check out the FAQ section of the DQ Directions website or contact us.

To submit a presentation proposal please follow this link: 

Submit a proposal for DQ Directions 1

How to Make Data Quality Improvements Stick: Expert Interview With Mark Eaton

Data Quality Pro: How do you describe the typical phases involved when organisations embark on an improvement initiative?

Mark Eaton:

There are really 3 main phases involved and these can be broadly defined as preparation, implementation and embedding.

• During the preparation phase the organisation is getting ready for improvement, setting down the objectives of the improvement programme and putting in place the resources to implement the changes.
• The implementation phase is fairly obvious, this is the physical processes involved when implementing the change and obtaining the initial improvements.
• The embedding phase is really a transition phase from ‘this is a new way of doing things’ to ‘have we ever done it any other way?’ This is otherwise known as the transition from having changed processes to the point of changing behaviours.

Data Quality Pro: How do organisations typically make transition between these 3 phases? Is there a further breakdown of activities?

Mark Eaton:

There are typically 6 transition points or crisis points as shown in the following chart.

Each point in this chart represents a critical stage in the improvement process. 

The 6 transition points are:

1. Decision to Improve
2. Strategic Planning
3. Preparation to Implement
4. The Noise of Implementation
5. Adoption of Improvements
6. Improvements Embedded

Data Quality Pro: It is quite common for people to react negatively to data quality improvement. In your experience, what kind of negative reactions are common at each of these transition points?

Mark Eaton:

During the Decision to Improve phase the typical reaction is that ‘this is not the right time’. There is often confusion about what needs to be done and therefore mixed messages from leaders which can lead to a drop in productivity

Strategic Planning often results in managers attempting to derail the process and move the focus from their own areas to other areas. There may be a lot of disagreement about the content of the strategy and attempts are often made to alter the scope and duration of the improvement programme.

During the Preparation to Implement phase, concern may start to grow within the organisation, although this may be partly balanced by some excitement about the change process ahead.

By the time we reach the Noise of Implementation phase some things will have gone well and others not so well. The ‘nay sayers’ will focus on the failures and some managers may often have a crisis of confidence that may lead to early termination of the programme.

During the Adoption of Improvements phase those not involved, or not positive about the implementation approach used, will try to undo the work done. Bad habits that existed before will still be there and will further degrade the achievements made if not effectively managed.

Finally, the Improvements Embedded phase will occasionally witness snipes by ‘nay sayers’ but broadly there is neither the will nor often the ability to go back to the old ways of working by this point.

Data Quality Pro: A recurring theme I hear in our industry is the situation where initial improvements have been made but gradually the gains are lost as old patterns of behaviour re-emerge. Why do so many improvements fail to embed themselves within the organisation?

Mark Eaton:

In identifying the negative reactions you will encounter during the improvement transition points, the following quote may be of use:

There are four things that hold back human progress; ignorance, lethargy, committees and inflexibility.

This quote is adapted from from Charles J C Lyall’s “There are four things that hold back human progress; ignorance, stupidity, committees and accountants.”

Ignorance is often accompanied by fear of the unknown. It can be tackled through effective (and ongoing) communication, involving people in the process of improvement and giving them the skills to know how to embed the improvements after they have been put in place.

Lethargic improvement programmes can result if there is not an effective ‘pace’ put on the programme. A lack of pace is normally indicative of unclear objectives for the programme and a lack of a sense of urgency from the senior team. In the words of Will Rogers, “Even if you’re on the right track you’ll get run over if you just stand there.”

Group thinking and committee domination needs to be avoided. Encouraging individual initiative and empowering leaders to make decisions and deal with issues (and then supporting them when they do) is the key here. If individuals feel threatened or at risk, they will not support the change.

Inflexibility to adapt the improvement strategy is a common cause of failure. The approach you started with may not be the most appropriate six months or two years later. Being blunt, there is no one single approach to making improvements work. Being prepared to experiment, learn from experience and broadly keep going is the key to success.

Data Quality Pro: How does the environment and culture of the workforce and organisation affect the progress of improvement and change?

Mark Eaton:

It’s absolutely critical and so often ignored.

There are environments that do not ‘allow’ people to comment constructively on the changes going on around them. Teams may operate along tribal lines with poor communication between them. In these situations the probability of success is very low, and both of these issues are directly related to the management environment that the organisational leaders have established.

The way leaders at all levels behave will also affect how teams react to the changes that have been implemented. Leaders who show no interest in the new ways of working, who actively push the team to work in the ‘old way’ or who, through words or actions, show that they disagree with the vision and objectives for the improvement programme are likely to lead to improvements that just slip away.

Where you might feel the desire to ‘by-pass’ difficult people and teams and perhaps implement the improvements without involving such people, I would refer you to the following quote,

“Without involvement, there is no commitment. Mark it down, asterisk it, circle it and underline it. No involvement, no commitment.” - Stephen Covey, author of ‘The Seven Habits of Highly Effective People’

Data Quality Pro: You and your team help large organisations implement successful change, what are some of the success elements that make this happen? Why do some organisations succeed where so many others fail?

Mark Eaton:

Several years ago we were involved in a research project to examine exactly why organisations failed to implement successful Lean initiatives. We identified eight areas that now form part of a diagnostic that we use to measure the readiness of organisations who are about to embark on major improvement initiatives.

These 8 phases form the acronym CRITICAL:

• Communications: Organisations tend to under-communicate both prior to and during the implementation.
• Resources: Improvement projects that have not succeeded are often found to have failed to allocate the right resources at the right time.
• Involvement: Failure to engage people in the improvement process and the failure to pass over ownership for the improvement to them are common issues.
• Training: Organisations often offer extensive training that results in people receiving information overload or they want to minimise the training and get straight into action.
• Implementation: Quite often implementations can cause upstream and downstream issues. Other problems relate to speed of delivery, either too fast or slow.
• Compass: Organisations that fail to implement improvements often state that they should have been clearer about what they wanted to achieve, how fast, with what resources and how they should engage and inform their staff.
• Achievement: Most people respond favourably to something that physically demonstrates the aspects of improvement that the organisation wishes to achieve.
• Leadership: This often includes a failure to show interest, deal effectively with ‘snipers’ or a failure to create the imperative for change. Difference of opinion amongst senior leaders is quickly identified by front-line staff as a reason to pull back from the improvement.

Mark Eaton is managing director of Amnis Ltd, a consultancy which specialises in innovation, transformation and organisational improvement, helping clients plan and deploy strategies for successful transformation. Its goal is to help clients not only deliver sustainable change but also to develop their capability to tackle their next challenges. Providing both consultancy and training services, Amnis’ team includes specialists in Lean/Six Sigma, organisational development, strategic planning, change management and systems thinking. He can be contacted via markeaton[AT]amnis.uk.com

Useful Resources

Struggling to sustain improvements in your data quality programs? 10 recommendations for organisational success

Are you sustaining change in your data quality initiatives? Lessons from leading change management specialists.

Are you managing change in your data quality initiative?

The Importance of Change Management: Interview with MaryGregory

Managing Change - Mary Gregory Revisited

Dr.Baškarada has an extensive background in the field of information and data quality so I recently caught up with him to find out more about his goals for the book and how he feels it will benefit organisations on their information quality journey.

IQM-CMM: Information Quality Management Capability Maturity Model, interview with Author Saša Baškarada

Data Quality Pro: Please describe your background. How did you get involved in information quality and what is your involvement in the sector at present?

Saša Baškarada: My background is in academia, research, and software engineering. I taught a wide range of software engineering and computer science courses at the University of South Australia (UniSA) and I also spent some time developing software for the Defence Science and Technology Organisation (DSTO) before I got involved in information quality through my doctoral research. I completed my PhD at the Strategic Information Management (SIM) Laboratory at UniSA, while also researching data quality for a large Australian Defence organisation. I am currently working for PricewaterhouseCoopers (PWC) in Melbourne, where I am part of the Risk and Controls Solutions group. Through my current position, I mainly provide consulting services in the areas of data management, data quality, and data analytics.

Data Quality Pro: What was your goal for creating the book?

Saša Baškarada: My PhD research, which focused on the development of an information quality management capability maturity model, received positive feedback from researchers and practitioners alike. In particular, the German Association for Information and Data Quality (DGIQ) showed great interest in my work and offered to sponsor the publication of the book. I saw this as a great opportunity to share my findings with the wider data quality community and to invite others to provide feedback and actively contribute toward further enhancements of the model.

Data Quality Pro: Can you explain the approach you took in developing the maturity model?

Saša Baškarada: The Information Quality Management Capability Maturity Model was developed in three stages.

Stage one involved six exploratory case studies and a comprehensive literature review. The aim of the first stage was to identify a wide range of candidate information quality management maturity indicators.

Stage two involved a Delphi study with information quality experts (practitioners and researchers), which was used to validate and group the candidate maturity indicators into staged evolutionary levels.

The final stage, stage three, was aimed at ensuring further external validation and enhancements though the application of the model in seven international case studies.

Data Quality Pro: What type of reader do you feel this book is aimed at? Who would typically be tasked with implementing your approach?

Saša Baškarada: The Information Quality Management Capability Maturity Model addresses technological, organisational/process, as well as social/people aspects of information quality management. As a result, this book should be of benefit to anyone with a strategic view of implementing a sustainable information quality management program in their organisation. More specifically, data/information architects, data/information quality analysts/managers, as well as data/information governance managers should greatly benefit from the approaches described in the book.

Data Quality Pro: What benefits will organisations obtain through their use of the Information Quality Management Capability Maturity Model?

Saša Baškarada: There are many reasons why organisations may wish to assess and enhance their capability maturity in information quality management, including minimising the costs associated with poor information quality and complying with government regulations and reporting requirements. The Information Quality Management Capability Maturity Model presents a set of evaluation tools, which are intended to assist with the identification of problems in the collection/storage/use of information and other information management practices. Furthermore, this tool is aimed at providing organisations with a measure of their capability maturity in information quality management, along with recommendations for increasing the level of their maturity, leading to enhancements in information quality. Thus, the model should help organisations with assessing their existing information quality management practices, developing improvement strategies, and benchmarking against best practice approaches as well as other organisations.

Data Quality Pro: You raised an interesting point in the last question regarding “…benchmarking against best practice approaches as well as other organisations…”, this strikes me as very progressive but quite hard to implement – do you have any thoughts on how this can be achieved?

Saša Baškarada: You are right in what you are saying Dylan – benchmarking tools and processes are definitely challenging to develop and implement.

Firstly, I would like to clarify that IQM-CMM does not aim to benchmark the quality of data between organisations, but the maturity of data management and data quality management processes. My goal with IQM-CMM has been to identify, develop, and document a set of ‘best practice’ approaches for information quality management. Thus, organisations can use IQM-CMM to assess their existing processes against the best practices specified by the model. Benchmarking against other organisations could be somewhat more difficult to achieve, since not all organisations may be willing to openly share the finding of their assessments. However, this hurdle could be overcome through the use of independent IQM-CMM auditors, who could anonymise any findings (like I did in my book) before sharing benchmarking information with any potential competitors.

Data Quality Pro: In the analysis you have undertaken so far, what were some of the common areas for improvement you have witnessed?

Saša Baškarada: The potential areas for improvement will of course depend on the maturity of the organisation with respect to information quality management. For instance, less mature organisations may enhance data quality simply by standardising and consistently enforcing their access control and change management processes. Gaps in access control may lead to fraud, vandalism, and lack of ownership and responsibility. Similarly, system or process changes, if not managed appropriately, may lead to issues with data consistency and completeness. On the other hand, more mature organisations may benefit form standardising and documenting their enterprise information architecture as well as from taking a risk based approach to information quality management. Needles to say, raising data quality awareness through appropriate training usually leads to quick and noticeable improvements in most organisations.

Data Quality Pro: What has the feedback been so far from organisations that have used IQM-CMM?

Saša Baškarada: The feedback has generally been very positive. Information quality management professionals seem to value IQM-CMM for its broadness and the detailed practical guidance it provides. Furthermore, organisations do not necessarily have to perform a complete IQM-CMM assessment in order to realise benefits – just reading through the model may help data quality professionals identify or generate new ideas.

Data Quality Pro: Do you find that organisations are generally surprised by the maturity rating they receive, either positively or negatively?

Saša Baškarada: That’s a very interesting question. I have noticed that organisations which are ignorant about information quality generally tend to overestimate the effectiveness of their processes. I guess their lack of understanding leads to the lack of awareness. On the other hand, more mature organisations tend to have a much better understanding of their information quality issues and thus they may often be much more self-critical.

Data Quality Pro: You mentioned your academic background with IQM, do you feel as an industry we should be doing more to promote the academic research into data/information quality? I sometimes feel that a lot of the great work carried out in academic circles rarely filters into the commercial world – what are your views?

Saša Baškarada: I definitely believe that closer cooperation between researchers and practitioners would lead to greater benefits for everyone. For instance, many doctoral researchers are eager to identify and develop new and better approaches, theories, and models; however, many such researchers also struggle to get access to real-world case studies. One of the reasons for the lack of integration between practitioners and researchers may be due to the fact that much of the research tends to be quite specialised and thus not always easy to understand. Luckily, there are now quite a few groups/forums – such as data quality pro, DGIQ, and IAIDQ – where practitioners and researchers can come together to establish networks and share ideas.

For more information on the book please visit Amazon at this link, there is also additional information here: http://www.scribd.com/doc/22835076/978-3-8348-0985-8-Baskarada-Information-Quality

If you are interested in adopting or developing the maturity model and wish to contact Saša Baškarada then please contact us at editor@dataqualitypro.com and we will forward on your details.

Dr. Saša (Sasha) Baškarada has more than ten years of experience in ICT, having spent time in academia/research (UniSA) and the Defence Science and Technology Organisation (DSTO). He is currently employed as a consultant at PricewaterhouseCoopers - Risk and Controls Solutions. Saša has provided advice to numerous Australian as well as international organisations in the areas of Information Quality Management and Strategic Information Management. He has published numerous book chapters as well as peer reviewed journal and conference papers.

Useful Resources

See all in: Methodology

Information Quality Management Framework (IQMF): An Overview

How to create a data quality framework or data quality methodology:Essential resources to get you started

Learn how the NZ Ministry of Justice created their DQ framework and download the final framework document.

MIKE2.0 and omCollab - Need to accelerate your own information management framework?

Using Metrics to Assert a Business Case for Data Quality

Money and resources wasted, sales missed, extra costs incurred. Recent research by industry analyst firm Gartner shows the shocking price that companies are paying because of poor quality data. And it adds up to a staggering $8.2 million annually.

That’s the average loss that the 140 companies surveyed put on it. But 22 per cent of them thought it was closer to $20 million and 4 percent even put the figure as high as $100 million. [1]

It’s a sobering thought. But do business managers really appreciate the scale of the problem? Not according to a recent report from The Data Warehousing Institute (TDWI). More than 80 per cent of the business managers surveyed by TDWI believed that their business data was just fine. Yet half of their own technical people took a very different view to their executives.[2]

The truth is that few business managers appreciate the extent to which data quality issues impact their company, since typically, no quantified measurements are made. This means proposals for data quality improvement projects fall on deaf ears. To have any chance of budget approval, data quality project proposals need an assertive business case. They need the backing of metrics that communicate evidence of a real problem that business managers readily understand; a problem that poses a risk to the business and to the key performance indicators (KPIs) by which its success is measured.

Data quality metrics

By developing a programme of data quality metrics and measuring and reporting regularly, organisations can build increased awareness of what data quality means for the business. Metrics can help demonstrate what risks/issues might be presented by any decline in data quality levels, and what opportunities might be gained by investing in improvement. Metrics also support objective judgement and reduce the influence of assumptions, politics, emotions and vested interests.

But there’s no point in measuring and reporting on all of an organisation’s data, or every aspect of that data. Be selective. A metric showing that nine percent of customer records in a marketing database lack a middle name is likely of little consequence to KPIs. But if five percent are missing a postal code, then it could be of some importance since if one million mailings are sent annually, then 50,000 would be returned. And if the metric referred to a billing database, it could make a strong business case indeed for a data quality project, as invoices worth millions of dollars might not be reaching customers, delaying or even threatening receipt of revenue.

Where to measure: key processes

For most organisations, business KPIs and the executive decisions aligned with them will most likely relate to cost, revenue, profitability, procurement, logistics, products, customers, suppliers and other important assets. Identifying the processes supporting these KPIs, the data required for these to operate effectively, and the quality of that data, enables organisations to determine the impact of poor quality in tangible terms. They are then much better placed to gain business understanding and support for building the business case for data quality.

For example, you might establish that:

• Revenue is typically 40 percent repeat sales, and half of repeat sales come through contact made by customer service representatives. Good contact data is vital to them. They cannot make follow-ups where customer contact information is missing or wrong.
• With direct print and mailing costs being significant and increasing, and the CEO keen to show ‘green’ achievements in the annual report, the current 100,000 plus pieces of mail returned per annum must be cut down. Customer address data accuracy is the key, and would appear to be an issue.
• In the last six months, eight percent of customers who purchased online had to wait longer for delivery than expected, despite the products being in stock. Product codes in the order system are inconsistent with the stock system requiring manual inspection and resolution.
• Senior management is considering rationalising product lines. But the sales ledger reports show discrepancies with the marketing department’s business intelligence system. Management cannot trust the cost/sales figures, delaying decisions and incurring unnecessary costs.

What to measure: dimensions

Having identified which data to produce metrics on, the next step is to define which of the many aspects of its quality to measure. These dimensions might include:

• Structure: Is the data in the right format for it to be usable?
• Conformity: Does it conform to critical rules?
• Accuracy: Does it reflect the real world?
• Completeness: Is business required information present?
• Timeliness: Is it sufficiently current?
• Uniqueness: Are duplicate records creating confusion?
• Consistency: Is the data the same, regardless of where it resides?
• Relevance: Is it useful to the business in its pursuit of objectives?

Defining which dimensions are important, prioritising them and producing data quality metrics that are meaningful for business owners is typically the job of one or more ‘data stewards’: individuals who understand the key business processes, the role of data in those processes and the intricacies of what makes ‘good data.’

How to measure: using rules

Having determined the data to be measured and by which of its dimensions, it is then possible to build a set of data quality rules against which to profile the data and compute compliance metrics. For example, if repeat sales from customer service representatives (CSRs) are key, and customer contact information must be present and accurate for the CSRs to make a follow-up/sales call, then perhaps ‘completeness’ of fields at original purchase is important, together with ‘accuracy’ and ‘structure’ of telephone numbers. If customers are waiting too long for the delivery of goods that would appear to have been in stock at time of order, then perhaps there is a disparity between product codes in the order processing system and the warehouse stock and dispatch system, so data ‘consistency’ should be measured.

In producing metrics, it’s probably best to be very focused at first, concentrating on just a few areas where data appears critical to business performance. It’s also better initially to produce just a small number of metrics on important characteristics that have real meaning to business managers in their roles and responsibilities. Indicating and proving that the business really cannot be totally confident in the data it relies on for certain important processes, decisions or compliance reports should quickly justify further investigation. Success in one area, can then be used as a reference to help communicate the value that could be won from metrics in other areas of the organisation.

Making data quality metrics relevant

Once the use of data quality metrics becomes accepted and management backing is secured, then the practice could extend to cover wider data sources and report in more detail. It’s then important to define who requires what level of reporting.

Management may desire a high level overview, so a data quality dashboard will be helpful. The dashboard should be able to group and aggregate scores into meaningful metrics associating data quality with key business processes or functions. It should also provide drill-down to multiple levels of detail and permit custom reporting allowing business owners, data stewards and data quality analysts to visualise data quality metrics and trends appropriate to their needs.

A facility to monitor and measure data quality over time is fundamental too. Only then is it possible to prove that investments in data quality are making a difference.

Data quality metrics, when aligned to business KPIs have the power to increase business user awareness, understanding and support for data quality investments. Using data quality metrics to identify where the real issues are that impact cost, revenue, profit or other important business metric will most certainly help drive the business case for data quality improvement.

About the author

Ed Wrazen is vp product marketing with Harte-Hanks Trillium Software, a leading provider of total data quality solutions. Working with customers, partners and industry analysts, Ed is responsible for product planning and release co-ordination. Ed has over 25 years experience working on IT systems, having started his career as a computer programmer on retail banking and travel reservation systems. He has been heavily involved in database and data management technologies as a product developer, consultant and lecturer, specialising in data architecture, performance design, data integration and data quality. Ed is a regular speaker at industry events worldwide and author on topics relating to data management. www.trilliumsoftware.com

Ed can be reached via www.trilliumsoftware.com

References:

[1] Gartner: Organizations Perceive Significant Cost Impact From Data Quality Issues, by Ted Friedman, 14^th August 2009

[2] TDWI Report Series: Taking Data Quality to the Enterprise Through Data Governance, by Philip Russom, May 2006

Useful Resources

See all posts in: DQ Techniques

How to create a data quality scorecard

Data Quality Rules Explained:

• Attribute Domain Constraints
• Relational Integrity Constraints
• Rules for Historical Data
• Rules for Event Histories
• Attribute Dependencies
• State-Dependent Objects