To help you kick-start your data quality initiative we’ve highlighted some existing data quality frameworks and data quality methodologies that are available both online and offline.
They should help you accelerate your own framework plus give you some pearls of wisdom from others who have trodden the same path.
If you know of any other useful resources please add them to the comments section and we’ll continue to update this post. Perhaps you have your own data quality framework or data quality methodology you would like to publish? If so, please contact us, we would love to help you publicise your work.
A word of caution when creating your own data quality framework
Data quality is a diverse discipline and this is reflected in the data quality methodologies currently available. You therefore need to be highly subjective in order to establish what components to include in your data quality framework.
For example, are you responsible for building a data quality methodology for your data warehouse facilities? If so this may require fairly detailed activity definitions of how to measure, assess and improve data at a practical level.
In contrast, are you looking to create a corporate governance strategy for data quality improvement? If so, you may not be interested in how individual teams deliver data quality improvements but instead may be concerned with how to build an overarching structure that delivers targets to data quality teams across the organisation for example.
Is your data quality framework focused on internal data quality improvement or is your goal to improve data quality across an industry or government sector? There may be major differences in the way these distinct frameworks are assembled.
When evaluating the following frameworks keep these previous points in mind.
Don’t simply rehash an existing framework without considering how it will be used on the ground.
You can take what you need from the frameworks below with relative ease but do think about how your employer or clients will use the framework and tailor it appropriately. Seldom will you find a perfect match.
Creating a Data Quality Methodology or Framework is not a One-Time Activity
Creating a data quality framework or data quality methodology for your organisation should be a continuous activity.
Your organisation will be learning, failing, succeeding and maturing with its approach to data quality all the time so this needs to be reflected in your framework.
Keep it well maintained, visible and accessible to the entire organisation. It shouldn’t be restricted or constrained to one particular area.
If you need support in this area please remember to use our forums and draw advice from the community here on Data Quality Pro.
List of Data Quality Framework and Data quality Methodology Resources
New Zealand Ministry of Justice Data Quality Framework
New Zealand Ministry of Justice Data Quality Framework
This covers the data quality lifecycle, the phases and activities of DMAIC, dimensions of data quality, causes and costs of data quality, checklists, assessment process flow and scorecards. Comprehensive and a good example of how a well structured framework should be constructed. The framework is built around the well-proven “Define – Measure – Analyse – Improve – Control” cycle and is an excellent example of how a framework should be tailored closely to the needs of the organisation.
CDQM: The “Complete Data Quality Methodology”
This is a framework presented by Carlo Batini/Monica Scannapieco in their book:
Essentially this framework consists of 3 core phases – state reconstruction, assessment and improvement. One of the most compelling aspects of this framework is the focus on business process and organisation services, this helps focus the efforts on the pain areas where costs are highest.
The book itself provides details of several data quality frameworks and a wide array of data quality concepts so is a useful resource when building your own data quality framework.
Data Quality Assessment Framework – Arkady Maydanchik
This book provides a complete data quality assessment and measurement framework that you will be hard pressed to find elsewhere in such detail.
Extremely detailed with clear examples and very much focused on practical techniques that can be easily adopted without the need for expensive tools. Essential reading for the practical data quality consultant who is looking to establish a framework for data quality assessment as part of a wider framework or simply on a per-project basis.
CIHI Data Quality Framework – Canadian Institute for Health Information
CIHI Data Quality Framework: This is the framework provided by the Canadian Institute for Health Information, it is available online.
This framework covers the various dimensions of quality the CIHI use to manage data assets across their databases. There is some good detail provided for each of the different data quality dimension and although obviously focused on the health sector, much of the content is applicable to most industries.
Enterprise Knowledge Management by David Loshin
This framework is provided by David Loshin of Knowledge Integrity and covers components such as building the economic framework for data quality, statistical process control, measurement and current state assessment, root-cause analysis, supplier management and how to implement business rules for data quality.
MIKE2.0 (Methodology for an Integrated Knowledge Environment): An Open Source initiative launched by Bearing Point that features an entire Enterprise Information Management framework, incorporating data quality.
There are a number of data quality framework components in the methodology and as it’s Open Source you can simply download the content and even build your own Wiki with the new omCollab facility.
An excellent resource, well worth investigating.
It culminates with a detailed 17 step plan for launching your data quality initiative. A very useful resource for shaping your own data quality methodology or framework.
Strategy for NHS Information Quality Assurance
Strategy for NHS information quality assurance – consultation draft: This report obviously focuses on the public health sector but it is particularly well detailed with respect to the governance and management processes involved in information quality assurance. Also has a number of great definitions of information quality that you could certainly use to educate senior management for example.
Ontario Ministry of Health and Long Term Care
Ontario Ministry of Health and Long Term Care: Data Quality Management Framework: Another useful framework focused on public sector data quality assessment and long term management.
The framework covers all key phases required in order to define, measure and report on data quality with considerable emphasis on health sector related KPI’s for data quality.
Danette McGilvray – Ten Steps Data Quality Framework
Danette details what is in effect a comprehensive data quality framework aimed at organisations who need to understand the logistics of executing well structured data quality initiatives. There is also a wealth of additional for free download at her supporting website for the book.
We have written a book review of the Ten Steps book in the past and I think it should definitely feature high on your wish list for data quality framework resources as it is thorough, practical and contains excellent supporting resources.
TDQM from MIT
TDQM: This is a joint effort among members of the TDQM Program, MIT Information Quality Program, CITM at UC Berkeley, government agencies such as the U.S. Navy, and industry partners. Activities in the TDQM program have evolved into four key areas:
- MIT Information Quality Program
- The International Conference on Information Quality
- The ACM Journal of Data and Information Quality
- Information Quality Certificate Program
TDQM stems from the Total Quality Management (TQM) methodology which developed in the manufacturing sector. To learn more about TDQMvisit this link, TDQM is also available in the form of anIQ Certification Programme.
TIQM from Larry English at Information Impact
TIQM (formerly TQdM) – Total Information Quality Management: This data quality framework was developed by Larry English of Information Impact International Inc.
TIQM is a comprehensive information quality framework that is detailed in what has to be one of the most popular data/information quality books of all time: “Improving Data Warehouse and Business Information Quality“.
Information Impact also provide a comprehensive set of free resources at their website. The TIQM methodology is well proven and many organisations have adopted TIQM for their enterprise-wide data quality initiative. TIQM is available as a certification and training program.
Implementing an Effective Data Quality Strategy (Australia | United Kingdom ):Daragh O’Brien has written a report that explores how organisations can define their own information quality strategy. It also scrutinises several methodologies for information quality so readers may also find it useful for helping to build their own data quality framework.
Data Quality Framework and Strategy for the New Zealand Ministry of Health
Karolyn provides additional components to the CIHI Data Quality Framework above and highlights the various data quality dimensions and components to be adopted by the New Zealand Ministry of Health.
Data Governance Framework by Data Governance Institute
Data Governance Framework: This framework is provided by the Data Governance Institute. Although this is not a data quality framework per se, there is obviously the need to manage data quality at an enterprise level data and many organisations are now realising the need to implement data governance strategies to sustain their data quality initiatives.
The Data Governance Institute is an excellent resource that combines community, content and education. Well worth a visit as there are a lot of very useful resources available that support the data quality process.
Defra RADAR Data Quality Framework
Defra RADAR Data Quality Framework: This framework is provided by the UK Department for Environment and Rural Affairs, it was published in 2004.
Relatively short and lacking in any real detail, it does however provide some useful templates and definitions for the various data quality dimensions that could prove useful, particularly in veterinary/health circles.
GS1 Data Quality Framework
GS1 Data Quality Framework: GS1 is a leading global organisation dedicated to the design and implementation of global standards and solutions to improve the efficiency and visibility of supply and demand chains globally and across sectors.
This framework has been written with the input from many retailers, manufacturers, industry associations, certification bodies and other organisations in the supply/demand chain sector. As such, it is encouraging to see so many organisations invest time and resources to compile this very thorough framework.
Although geared towards the supply chain industry, this is a good example of how any industry body or trade association could implement a governing framework for improving data quality standards in its member organisations.