If you’re keen to demonstrate your data quality skills in one of the most exciting and innovative information-driven organisations in the world, then the FT would like to speak with you.

Financial Times Seek Data Quality Manager, London, UK

I’ve been speaking with the Head of Database Marketing and Analytics at the FT for some time now and they are clearly innovating in a significant way. With major initiatives in business intelligence and master data management they are totally committed to data quality, in short – they get it.

Whilst this role will report to the Head of Database Marketing and Analytics, the opportunity to flex your data quality muscles across many other lines of business are huge.

We’ve been discussing the role and it’s clear to me they’re searching for the right balance of practical technical skill, methodology knowledge, management experience and a huge passion for data.

Given the management team involved and the responsibilities on offer, I think this is one of the most exciting data quality roles I’ve seen in the UK for quite some time.

If you are interested in this role, here is what you need to do:

1. Read the full job description below
2. Contact me here at Data Quality Pro: http://www.dataqualitypro.com/contact/
3. Supply your LinkedIn profile URL
4. Provide a brief description of your most recent data quality roles (be sure to check the job description below)
5. Provide a contact number and email address

If your background and experience are a good match then your details will be passed on to the relevant management team at the FT for review.

Job Description: Data Quality Manager – Financial Times

Background

The Financial Times, one of the world’s leading business news organisations, is recognised internationally for its authority, integrity and accuracy. Providing extensive news comments and analysis the FT Newspaper has a daily circulation of nearly 400,000 and a readership of 1.3 million worldwide. FT.com is an essential source of financial news, comment, data and analysis for the global business community. FT.com has over 2 million registered users and over 125,000 digital subscribers. The FT family of companies also includes FT Business which provides specialist information on the retail, personal and institutional finance industries, including the UK’s premier finance magazines Investors Chronicle, The Banker, Money Management and Financial Advisor.

The FT seeks to drive strategic business decisions from an in-depth understanding of its customer base and their demographics. This is facilitated by the Database Marketing & Analytics team (located with in FT.com), which develops operational and analytical reports for the various lines of business.

FT.com has recently embarked upon a major programme to implement a new Business Intelligence and Master Data Management solution that will provide the platform for all future marketing analytics.

The FT has its own IT organisation, but leverages managed IT services for several key business systems.

The Role

The FT recognises that appropriate levels of data quality and data governance are key enablers to its success in deriving business insights from marketing analytics. Consequently, the FT is looking for a full-time, hands-on Data Quality Manager to drive improvements in data creation, maintenance and integration across its core operational and business intelligence systems.

The role will report to Head of Database Marketing and Analytics, within FT.com, but will have the remit to affect data quality across all major FT lines of business.

The role will be based in the FT main office at 1 Southwark Bridge, London, with occasional travel to other FT sites.

The role requires a results-driven, practical, hands-on individual, who is self-motivated, proactive and resourceful, and who can undertake systematic analysis themselves and influence key business stakeholders to adopt improved data management practices.

Key Responsibilities

• To map the flow of information across FT business systems and processes, and identify the business critical data attributes upon which to focus resources and energy to gain the maximum return.
• To establish a light-touch approach to data governance and data quality improvement that will include: governing principles, an efficient engagement strategy, stakeholder management and communication, and a repeatable process for identifying and resolving data issues.
• To establish accountability for the quality of data within the relevant business groups, leading to greater data ownership and stewardship of the data at source.
• To drive issue resolution through business process change, system modifications and data cleansing, and by establishing the business case for action.
• To establish repeatable metrics that assess the impact of poor quality data on business costs and effectiveness.
• To establish and deploy, were appropriate, company-wide data standards.
• To provide data governance/quality training to FT business managers and information producers.

Skills & Competencies

Technical Skills

• Has domain expertise in media, publishing, and/or marketing in a global organisation.
• Is an expert in data governance and data quality practices, tools and techniques
• Can demonstrate recent successes in resolving data quality issues through business process change, system modification and data cleansing.
• Has hands-on and current knowledge of relational and dimensional data modelling, and data profiling, cleansing and enrichment tools (e.g. Dataflux, Trillium or IBM’s Datastage) are mandatory.
• Has hands-on and current knowledge of MS SQL Server and its toolset would be a clear asset.
• Knowledge of marketing analytics tools (such as FastStats) would be a preference.
• Can demonstrate successful project management. Experience of Agile methods would be a preference.

Personal skills:

• Has a passion for data
• Is motivated, proactive and resourceful
• Has excellent verbal and written communications skills
• Has excellent analysis, research and facilitation skills
• Works toward consensus
• Has the ability to work in cross-functional, geographically-dispersed teams
• Shows clear leadership and direction in a challenging environment
• Seeks to deploy consistent methodologies and techniques.

Education & Professional Accreditations

Mandatory: Good honours degree from leading university

If you are interested in this role, here is what you need to do:

1. Contact me here at Data Quality Pro: http://www.dataqualitypro.com/contact/
2. Supply your LinkedIn profile URL
3. Provide a brief description of your most recent data quality roles (be sure to check the job description below)
4. Provide a contact number and email address

If your background and experience are a good match your details will be passed on to the relevant management team at FT for review.

Please note: If you are interested in learning how to implement some of the concepts explored in this article, Mike will be hosting his popular Enterprise Data Governance & Master Data Management seminar on the 22-24 September 2010 in London.

Transitioning to Enterprise MDM: The Change Management Process by Mike Ferguson

There are typically three kinds of MDM system you can deploy:

• Registry Based MDM System
• MDM Data Hub
• Enterprise MDM System

The Registry Based MDM system uses Global Identifiers and data federation to create a virtual master data system of record (SOR) where no persistent master data store exists. Instead, the master data is integrated at run time and provided to requesting applications and processes that require it.

An MDM Data Hub is a system that propagates master data changes between disparate operational systems and can persist a complete integrated set of master data, in a master data store, as a system of record (SOR). In this case the operational systems that maintain the master data remain the systems of entry (SOEs).

The most advanced type of MDM deployment is an Enterprise MDM system which has the following characteristics.

• An Enterprise MDM system is both the system of entry (SOE) and the system of record (SOR)
• Supports master data integration (consolidation and federation) and master data synchronisation
• Fully integrated complete set of master data in a master data store
• The Master data store supports multiple master data entities
• All master data changes are logged and changes are propagated to other operational applications and data warehouses
• Can integrate operational master data, BI and related unstructured content associated with an instance of a core entity e.g. customer
• All master data entities are defined in a shared business vocabulary (SBV), i.e. a common enterprise wide data names and data definitions
• Full policing of master data integrity across the enterprise
• Full data quality firewall with DQ services to validate and clean master data
• Full set of enterprise data management tools for managing master data and metadata across the enterprise
• Common master data web services for access & maintenance of master data
• Tracks changes and holds to master metadata over time (history of metadata changes)
• Complete set of MDM business services around the master data

A diagram of Enterprise MDM is shown below.

Enterprise MDM (click to enlarge) 

One of the major differences with enterprise MDM is that this kind of MDM system is both a system of entry (SOE) as well as a system of record (SOR). This means that master data is updated centrally before changes are synchronised with other systems. Getting to enterprise MDM is a significant effort for many companies and may take several years if you are in a position where master data is heavily fractured and there is no single system of record. One way to evolve towards enterprise MDM is to start with a registry based MDM system, then progress to a data hub with a persistent system of record and then finally move on to an enterprise MDM system.

Admittedly, this is a lot easier to say than do. However, even if you have worked out how this can be done it is not just the mechanics of trying to centralise master data that is of concern to many organisations. One of the real challenges associated with deploying master data management is that of change management that may result from introducing an MDM system. In recent consulting engagements, and in talking to several of my clients, it has become clear to me that some organisations consider the integration of master data as the easy bit. What is actually causing the caution and the slow adoption of MDM is an even bigger issue that is much more difficult to get right and that is the change management process needed to correctly implement full enterprise MDM. In other words, working out what has to change and in what order when introducing an MDM system is what many are struggling with.

Looking at this problem more closely it is not a surprise that people are having difficulty, considering that an MDM system may result in:

• Changes to people’s roles and responsibilities
• Changes to documents
• Changes to user interfaces
• Changes to human/document workflow and system processes
• Changes to applications (operational & BI systems)
• Changes to data stores in disparate line of business application
• Changes to data movement and unstructured information management

The data piece is just part of the story here. It is often the case that data professionals may find they possess good skills for the mechanics of master data integration and data management but weaker skills in areas like changing processes, user interfaces, applications, documents and people’s roles and responsibilities. Certainly from the above list it is changes to people’s roles and responsibilities and changes to human/document workflow and system processes that are often the most challenging. One thing for sure is that you will struggle to implement the necessary change unless you know how your existing processes and applications work with master data.

In my opinion a significant contributor to success in MDM is that IT professionals tasked with implementing MDM must start an MDM project by leaving technology aside and taking the trouble to learn how their business works. It is necessary to conduct a business analysis of existing processes to identify what process activities are associated with master data. For example, this means understanding the processes associated with creating new customers. It may also mean we need to understand what operational and analytical applications, collaboration tools and documents are used in such processes.

The objective here is to identify all core business:

• Documents
• Process activities (tasks)
• Applications and application functions
• Application screens and on-line forms
• Reports
• Data integration jobs
• Etc, ……

that currently access and maintain disparate master data and also to identify all data stores holding disparate master data, e.g. Files, RDBMSs, cubes, content management systems etc.

The figure below shows an example order entry, fulfilment and tracking process with activities highlighted that are associated with master data (click to enlarge).

The MDM Change Process (click to enlarge) 

In order to do this well, a number of key questions need to be asked for each master data entity when following a business process. These include:

• Who performs each process activity?
• What documents and forms contain master data attributes?
• Which documents containing master data are used in which process activities?
• Which process activities access master data, e.g. customer data, product data, etc.?
• Which process activities maintain what master data, e.g. customer data, product data, etc.?
• Where do master data changes currently flow from?
• Where do master data changes currently flow to?
• What applications are used to perform each process activity and to maintain master data?
• What screens in the identified applications allow users to maintain master data
• What functions in what applications access and maintain master data
• What ETL jobs acquire master data and from where (what systems) do they get it from?

The objective of this kind of questioning is to create an inventory of things that may need to be changed when implementing an MDM system. This inventory could include documents, processes, application functions, screens, paper and on-line forms, data models etc. Once this has been done you should be able to identify overlaps, duplication and conflicts within this inventory and zoom in on who performs duplicate activities. The objective is to identify duplicate, overlapping and conflicting:

• People who are currently responsible for maintaining master data
• Documents containing master data
• Process activities associated with master data
• Screens and forms used to maintain master data
• Application functionality that maintains master data
• Reports containing master data
• Data integration jobs
• Data models
• Data stores and schema

Having got this far the next stage is to create a matrix to track conflicts, duplication and overlap so as to identify what must be changed in order to eliminate these. Then you can determine the order of priority in which changes need to occur. Factors influencing the order of priority on MDM change management include return on investment brought about by the change, change complexity, cost vs. budget available, human resources required and their availability and time to implement.

At this point you should be in a position of control over change management for MDM because you know what has to change and in what order. You may find that the ‘to do’ list you have created is somewhat daunting. However, at least you know what it is and that you can nevertheless make it doable. Transitioning systems to leverage centralised common master data may involve changing line of business SOE applications to remove/disable master data attributes from data entry screens and on-line forms. Once this is done new forms and screens will need to be created to directly maintain the master data in central MDM data hubs. This latter capability may be available out-of-the-box as some MDM systems ship with pre-built standard portlets that can run in any enterprise portal server to allow users to maintain master data directly from an enterprise portal user interface. We will talk more about this later in the article.

The big change is the move from many systems of entry (SOE’s) to one SOE. In order to make this happen you have to understand the impact of that kind of change on existing line of business applications. For example, line of business SOE systems may update more than master data in a specific transaction scope. Therefore, transaction logic in the line of business application may also have to change to remove master data changes from transactions. In addition duplicate or overlapping functionality in existing systems of entry (SOEs) has to be identified and removed/disabled to stop master data conflicts. There is no doubt that major transition begins once an MDM system shifts from being a system of record (SOR) to a system of entry (SOE) as shown in the figure below (click to enlarge).

MDM Transition Process

When this shift happens you need to already have a thorough understanding of the impact an enterprise MDM system will have. It may be the case that changes may need to be made to:

• Data
• Applications
• Processes and workflows
• User interfaces
• Documents
• People

Copyright ©Intelligent Business Strategies, 2010 – All Rights Reserved

Do you want to gain a detailed understanding of the potential change impact on each of these topics and how to deal with them?

Mike Ferguson will be running a 3 day workshop that provides practical advice on how to address these areas and much more at his popular Enterprise Data Governance & Master Data Management seminar, 22-24 September 2010, London.

For full details follow this link: http://www.intelligentbusiness.biz/EnterpriseDataGovernance&MDM.htm

About the author

Mike Ferguson is Managing Director of Intelligent Business Strategies Limited.

As an analyst and consultant he specialises in business intelligence and enterprise business integration. With over 28 years of IT experience, Mike has consulted for dozens of companies on business intelligence strategy, technology selection, enterprise architecture, and data management. He has spoken at events all over the world and written numerous articles. Mike is a resident expert on the Business Intelligence Network, providing articles, blogs and his insights on the industry. Formerly he was a principal and co-founder of Codd and Date Europe Limited – the inventors of the Relational Model, a Chief Architect at Teradata on the Teradata DBMS and European Managing Director of Database Associates. He teaches popular Enterprise 2.0 master classes in Operational Business Intelligence and Performance Management, Complex Event processing (CEP), Master Data Management, Information Integration and Service Oriented Architecture (SOA)

Recommended reading:

• Enterprise Compliance - Do you trust your data enough to tell it to a judge? White Paper by Mike Ferguson
• Data Governance Q&A with guest Steve Sarsfield
• Live Q&A with Jill Wanless discussing guerilla data governance
• How To Create A Data Issue Assessment Process: Expert Interview With Ken O'Connor
• Master Data Management Interview with Charles Blyth of CPP
• Interview with Jill Dyché of Baseline Consulting
• Data Quality Through a Metadata Strategy: Interview with Anne Marie Smith
• Identifying Duplicate Customer Records: An MDM Case Study With Dalton Cervo of Sun Microsystems
• MDM Insight Series: Expert Interview With Dan Power of Hub Solution Designs Inc.
• Interview With David Loshin, Author of "Master Data Management"
• A beginner's guide to MDM (Master Data Management)
• How to Make Data Quality Improvements Stick: Expert Interview With Mark Eaton
• Managing Change - Mary Gregory Revisited
• The Importance of Change Management: Interview with Mary Gregory
• Introduction to Guerilla Data Governance: An interview with Mike Meier

We discussed a range of topics including organic governance approaches, challenges of defining data governance, industry adoption trends, policy enforcement vs legislature and much more.

Runtime: approximately 45 minutes

Episode 4: Data Governance Q&A with guest Steve Sarsfield

Special thanks to everyone who helped promote this session, in particular Jim Harris who did an excellent job of live tweeting the event.

Runtime: 45 minutes

About Steve Sarsfield

Steve is a data governance business expert, speaker, blogger and author of the book "The Data Governance Imperative".

His popular blog, Data Governance and Data Quality Insider has provided regular industry commentary, advice and techniques since 2007.

Steve currently works for Talend as the Data Quality Product Marketing Manager.

Recent Episodes of Data Quality Pro LIVE

• Episode 1: Live Q&A with Jill Wanless discussing guerilla data governance
• Episode 2: Data Visualisation Techniques for Improved Data Quality
• Episode 3: Data Quality Career Bootcamp: Defining Your Value

This developing story should act as a stark warning to any services company in the social media age.

Bad news sparked by poor data quality travels faster and further than ever before. 

Enbridge Gas and the reality of data quality disasters in a social media age

According to the article penned by Ellen Roseman the underlying issue appears to stem from a classic data quality initiator, a major data migration project:

The utility upgraded its computerized billing system last fall, creating a few hiccups along the way. Direct Energy, which bills customers through Enbridge, upgraded its computer system at the same time.

Some customers didn't get bills for months. Some were overcharged for Direct Energy services (such as water heater rentals) and didn't get corrections for months, long after they'd been told the charges would be removed.

original article

As a result of earlier system issues, there was a failure in the process that adjusted customer monthly payments to the correct amount. Many customers started to receive far larger bills than expected when the shortfall was corrected later in the year.

Monitoring Data Quality Where It Matters

What is interesting is why this issue wasn’t picked up earlier.

According to Enbridge spokeswoman Lisa McCarney “There was no indication that such a large number of customers would have a balance owing or a credit.”

This points to a lack of comprehensive data quality management within Enbridge Gas, this issue has clearly taken them by surprise.

The reason for this may be down to an observation by Ellen Roseman: “Enbridge looked at the overall budget billing plan, which tracked closely to expectations, and didn't realize that many customers were being undercharged.”

So, it appears that the top level figures looked good, possibly as a result of a large quantity of people claiming that their accounts were in excessive credit (indicating over-billing). With under-billing and over-billing activity the top level figures may have looked reasonable but clearly there were issues at a more granular level.

There is a simple lesson here in that you need to manage and monitor quality of the data being shipped to your downstream data consumers, in this case the gas customers.

Using a top-level, aggregated report like the one that appears to have been used at Enbridge can easily mask problems.

A data quality monitoring check to observe fluctuations in pricing based on previous year estimates would have been quite trivial in this situation and perhaps prudent given all the earlier systems and billing issues but based on the facts provided in the article, it does not appear to have been implemented.

“Perfect Storm” of Negative PR

The knock-on effect of this data quality mishap is a large volume of customer complaints, some very stretched staff and a perfect storm of damaging PR for Enbridge as news breaks that they may have leaked up to 1 million gallons of oil into a local river, potentially threatening the health of locals.

Social media is clearly playing its part in this story.

With the article being posted online in The Star.com many disgruntled customers have decided to vent their frustration online. The story went live yesterday and already there are 51 comments, many of them from very dissatisfied customers.

A search for Enbridge Gas on Google is also starting to list front-page negative stories, from several media outlets.

Welcome to Data Quality in the Social Media Age

In this example, Enbridge Gas no doubt implemented their pay-monthly billing plan to help improve their cash flow and reduce the administrative burden of late payments. As a result of what appears to be a series of data quality errors they can surely expect to see a migration of customers back to a traditional quarterly billing cycle. This clearly hits their bottom line and as some of the commenters in the The Sun.com article pointed out, there are other suppliers available. Many customers may obviously vote with their feet as frustrations rise.

This is the new reality for any service company that continues to ignore data quality management in the age of “new media”. Your customers now have a voice that stretches far beyond their local community. They can form their own communities, connect, share stories and become a force that damages your reputation and development.

In stark contrast to the scale of rising frustration, it appears that the data quality controls required to prevent this issue occurring would have been quite trivial.

A practical lesson for all companies sitting on the data quality fence.

Recommended reading:

• Trust and the Negative Data Quality Feedback Loop
• Enterprise Compliance - Do you trust your data enough to tell it to a judge? White Paper by Mike Ferguson
• Expert Interview with Jill Wanless (author of the "Data Quality from the Ground Up" blog)
• How Are You Creating a Pull for Data Quality in Your Organisation?
• How to Present Data Quality Dimensions For Maximum Impact
• Building Your Internal Data Quality "Business"
• WANTED: Data Quality Change Agents
• 8 Tips for Making Your Data Quality Resolutions Stick in 2010
• How to set data quality goals any business can achieve
• 7 Essential Skills for Effective Data Quality Leaders

By focusing solely on what you do, instead of what the client gets, practitioners may be placing themselves at a disadvantage in the job market.

In short: are you focusing on your features or your benefits?

Data Quality Practitioners: Are you focusing on your benefits or features?

I recently launched the Data Quality Career Bootcamp, an online course designed to help our readers and members accelerate their career using some practical techniques.

If you missed the first online training session you can watch it on-demand here. If you want to get on the second session (spaces are limited and running out) please register here.

In the first session, one of the issues I discussed is a common problem many people experience when talking about their data quality skills and service, they forget to articulate the true value of their service.

I did this, then I did that, then I did this…

When we’re trying to impress a would-be client, partner or employer, the curriculum vitae (CV) is one of the preferred tools of the trade. Having recruited for data quality roles in the past I’ve witnessed my fair share of lacklustre CV’s. One of the main reasons is that the individuals focused on what they did (features), instead of what they actually delivered (benefits).

Let’s take this real life example of a Global Head of Data Quality, describing their latest role:

1. Developed organisations capabilities concerning Data Quality by introducing tools, techniques and software solutions.
2. Acted as chair to global data quality forum and actively supported the embedding of data quality roles and responsibilities in the business organisation.

Looks impressive?

The individual clearly has broad skills but what real benefits has the person delivered? They are nowhere to be seen. Instead, the person has opted to fill their bio with fairly standard features in their hope of impressing the visitor.

I took this one stage further and reviewed scores of data quality practitioner bios on LinkedIn. The result? Not one came close to explaining the benefits and true value of their skills and experience.

What is a benefit?

A benefit is typically what the company is left with AFTER you delivered your feature set of skills and services.

A classic example:

• The AFTER Benefit a client is really interested in:
  • A Hole
• How we typically interpret what the client wants to hear:
  • In my last job I used the world’s most powerful drill
  • I have 10 years experience using different types of drill
  • I have completed 10 training courses on how to use various types of drill
  • I have a number of highly effective drill bits that make me an expert in using the drill

Far too often, we focus on the different features of our expertise instead of clearly articulating what the client or employer actually got AFTER we "waved our data quality wand".

Some quick examples:

Senior Data Quality Analyst working for a manufacturer…

• Original Bio Feature: I delivered a data quality initiative for a global pharmaceutical manufacturer.
• New Bio Benefit: I helped a global pharma decrease their lead times by 5% resulting in a 1st year ROI of 85%, this was achieved by: (bullet-list your data quality features).

Junior Data Quality Analyst…

• Original Bio Feature: I rolled out a data profiling/monitoring initiative for all customer-facing web forms
• New Bio Benefit: I reduced customer-churn by an estimated 25% and increased new customer acquisition by 10%. I delivered this by (bullet-list features here)

In Summary

The key to a successful career is not to possess the most skills but to out-perform your peers searching for the same role.

You can have many years experience in all the right data quality methodologies, technologies and philosophies but if you can’t articulate your value in simple, clear terms then you’re at a major disadvantage.

If you want to learn how to accelerate your data quality career I recommend that you take part in the new online career tips course. The techniques covered are highly practical and quick to execute plus they will differentiate you in an increasingly crowded market. (The course is currently free for members and non-members).

DQ Career Bootcamp: Session 2: Building your online base : Thursday August 5th (reserve your seat)

Recommended reading:

See all in: Personal Development : Running a DQ business

• DQ Career Bootcamp: Session 2: Building your online base : Thursday August 5th (reserve your seat)
• Data Quality Career Bootcamp: Defining Your Value (first session of course)
• Learn how to tell stories and boost your DQ opportunities.
• Suffering from feast or famine in your DQ business or career? Learn how to network and promote yourself more effectively with this 2-part tutorial.
• 5 simple activities to help sharpen your data quality sales performance
• 10 Tips to help data quality professionals boost their career prospects in the downturn
• 20 simple tips to spice up your data quality blog

Are you maximising shared opportunities in your organisation?

Cash-Strapped Procurement? Try Sharing Data Quality Software

This post was triggered by the news that a group of UK police forces comprising of Essex, Cambridge, Bedfordshire, Hertfordshire, Norfolk and Suffolk together with the Ministry of Defence have decided to share access to data quality software as part of a project to create a single unified database across the forces.

With public sector funding in the UK heading for a major clampdown in light of the budget deficit it appears that creativity with licensing and procurement is now a reality across public sector organisations.

Local Knowledge Rules

Another interesting facet of this story is just how much weight public sector organisations place on the sector experience of solution providers.

The leader of the initiative, Chief Inspector Andy Gratrix, Chief Information Officer, Cambridge Constabulary, cited the domain experience of the provider as a key benefit.

Looking at the solution providers website, their primary focus is on the public sector and it has clearly been an advantage in this case over other technology providers offering matching technology.

Vendors Need to go Niche

As data quality technology matures and vendors begin to offer comparable products I believe it will increasingly come down to domain expertise and maturity of specialised partnerships to win these type of projects.

(Regular readers may remember that I talked about this recently in the WANTED: Data Quality Entrepreneurs article).

There is huge potential for niche solutions, both service and technology, across the public and private sector but the message I continue to hear from our members who have recently launched software evaluations is that many vendors are failing to understand the data requirements of their niche and are therefore being rejected from the final procurement round.

Is the Time Right for Data Quality in the Cloud?

Another angle here is the potential for cloud-based data quality services to solve these type of shared service requirements. Within public sector organisations there will always be the concern over security and moving data onto systems technically outside of their control but I think we’re heading for a major battle amongst the data quality cloud technology providers next year, particularly in the corporate sector.

A cloud based delivery model would make this type of data matching service particularly effective across disparate departments, systems, countries and silos. With several vendors ramping up their cloud offerings I think we’ll start to see a lot more sales activity in this area over coming months.

Companies Need to Consolidate Data Quality Functions

Finally, one of the biggest problems I witness is a huge waste of data quality resources, both in the form of people and technology in many organisations.

It is not uncommon in my experience to find multiple data quality products, disparate data quality teams and no unified strategy.

There are massive gains to be had by utilising an enterprise data quality team in the form of a central competency centre that can provide a well defined set of services to the entire organisation using one license framework.

What’s your take on this? Is your organisation benefitting from a shared license framework? How are you delivering value to the organisation in light of reduced funding?

Recommended reading:

• WANTED: Data Quality Entrepreneurs
• How to create a data quality competency center: Expert interview with John Schmidt
• How To Deliver A Compelling Data Quality Business Case
• 10 Tips to help data quality professionals boost their career prospects in the downturn
• 5 Simple Techniques To Differentiate Your Data Quality Service
• 15 Tips for transforming knowledge-workers into a data quality task force
• How to create a data quality framework or data quality methodology:Essential resources to get you started
• Are you managing change in your data quality initiative?
• Do you struggle to create a compelling introduction to your DQ proposition? Learn how to tell stories that will boost your DQ opportunities.
• Suffering from feast or famine in your DQ business or career? Learn how to network and promote yourself more effectively
• 5 simple activities to help sharpen your data quality sales performance

The “Data Quality Bootcamp” is a 4 part online course designed to help you maximise your career potential by delivering some practical exercises and techniques to help you build your personal brand, create an online base, promote your skills, create a high-value network and connect directly with organisations in need of your expertise.

Week 1 focused on helping attendees define their value and differentiate themselves from the rest of the crowd.

This post provides an on-demand video of that first live session.

Episode 3: Data Quality Career Bootcamp: Defining Your Value

Running time: 40 minutes

Recommended reading:

• Do you struggle to create a compelling introduction to your DQ proposition? Learn how to tell stories that will boost your DQ opportunities.
• Suffering from feast or famine in your DQ business or career? Learn how to network and promote yourself more effectively with this 2-part tutorial.
• 5 simple activities to help sharpen your data quality sales performance
• 10 Tips to help data quality professionals boost their career prospects in the downturn
• 20 simple tips to spice up your data quality blog

According to Mark, the figure quoted at the conference was 60-90% of data is valueless but is this a realistic value?

Are You Afraid to Say Goodbye to Your Data?

Based on a number of data migration projects I’ve been involved with I would have to wholeheartedly agree with the kind of figures quoted in Mark's article.

As an example, when identifying data in scope for a migration I typically start from the premise that ALL data is out of scope unless someone can justify its existence. (This forces the emphasis back on the business to justify their use of the data).

In most cases at least half of the information was found to have limited value and could be cut from the target system, typically to significant cost savings as every item of data incurs modelling, data quality, data mapping, transfer coding and extensive validation.

The Causes of Growth

Why have data volumes grown so excessively? There are plenty of reasons:

1. Storage is affordable and accessible
2. Data warehouses need feeding with data (but how much of that data is transformed to actions?)
3. Applications are designed that have attributes/data structures that are bloated and in many cases redundant
4. System silos lead to replication of corporate data
5. Mergers and acquisitions are commonplace, data often comes with the deal
6. There is no archive strategy

I believe the main reason data volumes are growing though is simply because of the last point, organisations are not very good at developing an archival strategy to remove stale data.

The impacts of this growth are numerous:

• Increased staffing costs to tune and manage the data
• Additional cooling/infrastructure costs
• Reduced query performance
• Backup windows become compromised
• Data integration and data migration become far more complex and costly
• New IT projects take longer and are more prone to failure
• Slower performance lowers knowledge worker productivity and increases costs

The Impact of Stale Data on Data Quality Management

There is a danger in assessing data quality across stale data as it can dramatically skew your findings.

If the data quality was found to be poor historically (perhaps there was a lack of completeness in the past but now there are far less data “gaps”) we may incorrectly assume that our improvement process is working correctly.

I recall an organisation that upon receipt of their new data profiling tool pointed it at their billing system.

They were horrified to find thousands of historical errors in tariff coding, product code allocation and many other issues. The problem was that the company had shifted their business model from offering products to focusing far more on services. In addition, many of the customers incorrectly billed in the past had terminated their accounts. By taking a data quality assessment of this historical data the company were in fact providing no real insight into data quality across their current business model.

Yes, they discovered they had badly designed processes but a workshop with the knowledge workers confirmed the same insight within a few minutes. What they should have been focusing on originally is how data quality impacts their business TODAY.

Designing an Archive Strategy - Getting re-use from the Data Quality Team

There are a number of techniques common to the data quality practitioner that can play a useful role in the decommissioning of your corporate data:

• Information chain mapping: Help identify the flow of information across the enterprise so that any downstream data consumers can be assessed for potential impact from decommissioned data
• Data profiling: Analysing the statistics of data elements (records/attributes) can help identify redundant data that can be eliminated
• Data matching/relationship discovery: Can help identify dependent data in disparate systems so that a synchronised process of data removal can take place
• CRUD analysis: Identifying which applications Create, Read, Update or Delete data is of great importance when determining which datasets can be archived

So What Next?

Archiving data is nearly always initiated by IT. If you’re on the business side start the discussions now and play your part because there are significant benefits to the business community in archiving off data and by waiting for something magical to happen without your involvement means it will simply never get done.

Note that we’re talking about archiving, not deleting, typically on a readily accessible medium.

The data can still be maintained in an offline store for compliance or reporting requirements but particularly if you want to reduce the costs of your data quality management efforts and create a more effective workforce, it may just be the time to collaborate with your IT colleagues and begin the essential activity of creating an archive process.

If you find after several months that on no occasion did you need to dip into the archive to retrieve some past information it may be time to archive to tape, store offsite and cut loose for good.

In particular I want to discuss how poor data quality leads to a cycle of increasing mistrust and gradual data quality degradation.

Trust and the Negative Data Quality Feedback Loop

First, some background. The UK National Health Service (NHS) wants over 50 million people in England to agree to their records being included in an individual Summary Care Record (SCR) database. The aim of this program is to help improve the quality of care provided by hospital staff and any out-of-hours doctors by releasing information that is typically held only by a person’s general practitioner (GP).

Following a pilot project in Birmingham it was revealed that 1 in 10 patient records loaded onto the national Summary Care Record system had not been correctly updated, leading to concerns from health experts that fatalities could result from applying inappropriate drugs or treatments.

This story is all the more significant in light of the recent case of a German out of hours doctor who administered an incorrect dose of a drug to a patient which caused fatality. Although the doctor in question did not use the Summary Care Record system the results of what can go wrong are all too apparent.

The results of the recent pilot have prompted the British Medical Association (BMA) GP Committee to pass a motion suspending the database until the safety issues have been fully investigated and resolved.

With 10% error rates being cited it would appear that any resolutions would take considerable time and effort, if indeed they are at all feasible.

NHS managers apparently claimed that “…the problems were insignificant because all care records came with a ‘health warning’ that they should not be relied on”.

This begs the question of trust.

If the perceived trust of the data is so low – how will the system function with a live, national rollout?

Why is a lack of trust so damaging?

A lack of trust does not directly create poor data quality, it is the events surrounding a lack of trust that ultimately generate poor data quality.

For example, a utilities engineer goes to install a new piece of equipment. The wiring schematic from the CAD based system is found to be inaccurate. New equipment is missing and the power rating is incorrect.

If the trust in the system is strong then the engineer is likely to maintain the system correctly but if the trust is low then the engineer is likely to store some basic information in the system and then perhaps add their own personal records using their own local spreadsheet.

On every data migration project I’ve been involved with, local records were discovered and one of the overriding causes was always because the workforce did not trust the system to store and retrieve the information they required in a format that was right for their role.

As a result the data quality worsens, caught in a continuously degrading feedback loop.

What is the answer?

In the example of the NHS database, trust is critical to the success of the system. People have the right to opt-out of the system and if the press surrounding the quality of data in the system continues to be negative then allergy sufferers and people on specialised drugs for example may certainly question their involvement in the scheme.

The answer is an obvious one.

Don’t try and resolve data quality once the system is live, it is simply too late. “Fixing the data in the target” is a classic data migration tactic but one fraught with risk because although we may eventually be able to recover data quality levels it can take far longer to recover trust.

The main focus of the project leaders would appear to be completion first, quality second.

If the reports of 10% defect rates in patient records are accurate then the entire integrity of the system is in jeopardy. If that figure does not dramatically improve there may be no way to retain and grow public trust in the system moving forward.

Which is truly sad because the system, as a concept, presents an excellent example of how data of the highest quality can save lives.

Recommended reading:

• NHS unique ID failure leads to duplicate newborn patient records (Data Quality Pro)
• Patients' medical records go online without consent (The Telegraph)
• NHS database raises privacy fears, say doctors (The Guardian)
• ‘Opt out’ fear on electronic medical record system (Times Online)
• Firm that hired fatal overdose doctor Daniel Ubani hopes for takeover (The Guardian)

The episode closes with details of a practical tutorial and free trial software you can use to build your own experience in this discipline that has so many applications within the data quality process.

Episode 2: Data Visualisation Techniques for Improved Data Quality

Running Time: 64 minutes

Trial software discussed in this episode:

• http://visokio.com
• http://tableausoftware.com
• http://qlikview.com

Practice Tutorial (provided by Data Quality Pro):

Data Quality Rules Analysis Using Data Visualisation Tools: Tutorial #1 Historical Data

Recent episodes:

Episode #1: Live Q&A with Jill Wanless discussing guerilla data governance

 The Information Difference is carrying out research into how Data Governance can link Master Data Management and Data Quality.

If you can spare about 10 minutes to complete the survey it will really help to extend industry knowledge in this area. Anyone who completes the survey will receive a free summary of the survey results and enter a draw to win a free vendor profile worth $495.

Complete the survey at this link: http://www.surveygizmo.com/s/304307/qijy6

How Data Governance links Master Data Management and Data Quality Survey 2010

This survey investigates the relationships between data governance, master data management (MDM) and data quality (DQ). Many authors are highlighting in the media the crucial importance of data governance and data quality initiatives to ensuring the success of MDM implementations. There is, however, scant information on the approach being adopted by organizations that have implemented or plan to implement MDM, or indeed those who have chosen not to implement data governance.

The aim of this Information Difference survey is to explore the linkage between data governance, master data and data quality, and to discover how organizations are tackling this area in practice. We also wish to understand the scale, scope and success rates of data governance programs.

All information provided will be used in aggregate form only and will be kept strictly confidential. The survey has about 20 questions on the topic and should not take more than 10 minutes to complete. In return for a fully completed survey you will receive a free summary of the analysis of the survey results. Additionally your name will be entered in a prize draw and the first five winners will receive a free vendor profile (worth $495) of their choice.

Here is the link to complete the survey: http://www.surveygizmo.com/s/304307/qijy6

The survey is sponsored by Ataccama, Data Foundations, Kalido, Lumendata, Microsoft and Teradata. Data Quality Pro, Information Management (formerly DM Review), TechTarget, The Data Governance Institute and DAMA are media sponsors.

Thank you in advance for your help with this survey.

Andy Hayler

Please take the survey: http://www.surveygizmo.com/s/304307/qijy6

Many Thanks!

The Information Difference Team

Jill is author of the Data Quality From The Ground Up Blog and in this episode shares nearly an hour of practical advice, offering techniques on a wide variety of data quality and data governance topics.

Episode #1: Live Q&A with Jill Wanless discussing guerilla data governance

To view this episode, please click on the viewer below. If you have any questions please post them in the comments section at the end of this post. 

Recommended reading:

• Data Quality From The Ground Up Blog – Jill’s popular data quality/governance blog
• Expert Interview with Jill Wanless (author of the "Data Quality from the Ground Up" blog) (earlier email based interview with Jill)
• Expert Interview: Kathy Hunter of Kynetika
• How to Make Data Quality Improvements Stick: Expert Interview With Mark Eaton
• IQM-CMM: Information Quality Management Capability Maturity Model, interview with Author Sasa Baskarada
• Managing Change - Mary Gregory Revisited
• The Importance of Change Management: Interview with Mary Gregory
• How To Create A Data Issue Assessment Process: Expert Interview With Ken O'Connor
• Master Data Management Interview with Charles Blyth of CPP
• Interview with Jill Dyché of Baseline Consulting
• Interview with Danette McGilvray, author of "Executing Data Quality Projects: Ten Steps to Quality Data and Trusted Information"
• Interview with Larry English, Creator of TIQM
• Data Quality Through a Metadata Strategy: Interview with Anne Marie Smith

Are you expecting too much of your data quality leader?

Data Quality Director Required - Must Possess Powers of Invincibility

For many of our readers I suspect an executive or directorial role in data quality or data governance is on their future career ladder. It is the ultimate career progression after all, the ability to report to the executive board and make a real difference across the organisation.

The reality is that senior roles are still relatively uncommon and as a result some organisations may view those roles as a holding bin in which to stuff a myriad of unrealistic requirements.

Below is an extract of a job description I recently found online which appears to typify this situation. I've added comments in bold, why not add your comments in the form at the end of this post?

Data Quality Director Required

Main Responsibilities:

1. Drive the definition and implementation of data governance initiatives - if you are defining and driving data governance then surely the role is one of data quality/data governance director?
2. Recommend and implement an approach for collection, storage, protection and delivery of application usage data, ensuring it meets the needs of the organization. - good, clear requirement but will there be a team to support this? No clear mention of this.
3. Ensure data quality through the strategic management of information throughout the product life cycle. - what does this really mean?
4. Prepare, disseminate, and monitor a data management plan. - This would seem a good fit for this role.
5. Ensure adherence to data governance policies and procedures - would a director take on an auditing role? Where is the hand-off to stewards and local "law" enforcers?
6. Inform business users and executives of data collection and maintenance plans and their progress; ensure all stakeholders understand the impact of any process or content changes. - would you expect the "Director of Corporate Sanitation" to sit with users to discuss their progress on waste collection?
7. Maintain the quality of product usage information in terms of accuracy, integrity, standardization and completeness, according to corporate standards. - Is this really the role of a Director? This would appear a full-time role in its own right.
8. In conjunction with the Data Warehouse Architect and the BI team, ensure application tracking data file formats and content comply with data warehouse standards and meet business needs for reporting. - Does "checking file formats" really fit a directorial role? Admittedly there is finally some talk of utilising other resources.
9. Ensure metadata is maintained and is compliant with corporate standards. - Should data stewards not be responsible for maintenance and validation of standards, reporting to the director?
10. Ensure faulty data is brought to the attention of those who will repair it as well as those impacted by its incorrectness. - Low-level data quality analyst/information chain steward role? Do you agree/disagree?
11. Track data bugs and their fixes; drive timely resolution of data quality issues. - Same as 10?
12. Maintain and publish a data change history, including impact of each change. - Same as 10?
13. Participate in database and data warehouse schema design discussions with the Database Engineering and Business Intelligence teams to ensure a full understanding of the design and agreement it will meet reporting requirements. - Have you ever participated in these? Do you know how long they can drag on for and how detailed this can be? Is this really a directors responsibility?
14. Understand our businesses and appropriately prioritize data quality activities to best serve business needs. - Yes, good requirement for a director.
15. Ensure data format consistency across implementations. - No, bad requirement for a director, surely too low-level and vague.
16. Ensure that data quality is routinely assessed against ongoing business needs. - Where is the hand-off? Surely should read - "Ensure that data quality analysts (or other) are routinely assessing data against business needs and reporting appropriate KPI's"
17. Manage data quality investigations and report findings to business and technical stakeholders. - Are they delivering or delegating here? Not clear.

Major Functional Areas:

1. Data Analysis: Document application usage data inventory and requirements. - Senior Data Analyst role?
2. Data Planning: In conjunction with the Enterprise Data Architect and the Database and Data Warehouse teams, plan and develop an information architecture to fulfill the application usage data requirements of the organization. - Data Architecture role?
3. Data Control: Set data standards and procedures to ensure enterprise-wide quality and integration. Monitor and enforce data policies, procedures and standards. - Senior Data Steward role?
4. Education and Training: Promote the use and appreciation of data standards throughout the company. - Should they really be expected to deliver education and training? Where is the hand-off?

Thoughts from the blogosphere

Long-time industry blogger Vincent McBurney (@vmcburney) recently wrote an interesting article titled "Data Governance is Suicide" which also discusses the issue of organisations expecting too much of their data governance leaders. When we see roles advertised as far-reaching and all-encompassing as the one above, you can certainly see how this situation arises.

Dan Power (@dan_power) responded with an excellent post imploring companies to create a "SWAT" team to help these leaders hit the ground running and deliver with the appropriate resources, see Data Governance: The People Make It Real. Some great tips if you are currently putting a job description together for your data governance leader.

Variety is not always the spice of (corporate) life

The job description above cites 17 areas of responsibility across 4 functional areas and not one mention of the support team that will help this energetic leader complete these tasks. If someone is looking for a job with variety, this could be the one for them. One day they will be checking file formats and tracking data bugs, the next they may find themselves defining corporate data governance initiatives, never a dull moment for sure.

Show commitment in equal measures

If you want a skilled, experienced leader to deliver serious data quality improvement in your organisation you have to equip them with resources that are equally skilled and experienced. They cannot do this alone.

If you want your leader to deliver data governance processes and procedures across the organisation, equip them with a task force to see this through on the ground.

If you want your leader to ensure data quality management is practiced as standard across all information systems, give them a team with the right training and technology to make this a reality.

If you want a leader to quit, disillusioned and demoralised after sailing solo into a political headwind for month after month, please pass the above job description to your HR department and let the search commence.

What are your views? Do we expect too much from data quality and data governance directors or should they be able to deliver on the ground and in the boardroom? Please add your comments below.

Recommended resources:

• Guerilla Data Governance - with Jill Wanless - Live Webcast on Data Quality Pro, 8th July
• Expert Interview: Succeeding in data quality by Kathy Hunter of Kynetika
• Introduction to Guerilla Data Governance: An interview with Mike Meier
• DEBATE: How should data governance and data quality work together?
• How to create a data quality competency center: Expert interview with John Schmidt
• Creating An Internal Data Quality Community: Introduction (Part 1 of 4)
• Scaling the heights: Data quality professionals give their insight into what it takes to climb the data quality career ladder to the very top (#1)
• Insights from data quality professionals who have attained senior roles - Andrew Brooks (#2 in series)
• Data Governance is Suicide by @vmcburney
• Data Governance: The People Make It Real by @dan_power

Rise of the Data Quality Practitioner 2.0

My data quality career effectively started in the early nineties. At this time the internet really was in its infancy in terms of job services. Like most graduates, I had to send off my CV for jobs listed in trade magazines and wait for a phone call which mostly never came.

The evolution of data quality job hunting

The internet accelerated our career aspirations. Online job boards and alert engines made it far easier to search and react to jobs as they came online. Speaking to a headhunter specialising in senior data quality roles recently it is becoming clear that job boards are now making way for social networking because it provides social proof. The recruiter confided in me that she no longer uses job boards because the quality of response is far too variable. She instead prefers to network within the online data quality community, searching for referrals to the best resources.

TIP: We're about to launch a data quality career bootcamp in the coming weeks which will cover how to boost your online presence, come back to the Data Quality Pro home page for full details next week.

Data quality professionals spend far more of their time online which is evident by the high traffic rates we see on sites like Data Quality Pro and the other popular data quality blogs that are launching practically every week.

I spoke with an end-organisation last month who had spent 6 months searching for a data quality practitioner. The delay had a major impact on their roadmap and is a common issue in the industry. There are a limited supply of experienced practitioners but the demand can often be sporadic as companies are still relatively slow to mature their data quality programs.

Going mobile

On a personal level, I last worked in a full time data quality consulting role about 5 years ago. In recent years I've been fortunate enough to carry out out various consulting roles before switching full time to running Data Quality Pro. What I think is encouraging for the profession, particularly in the UK, is that all of the organisations I've consulted with have embraced a more flexible, remote working arrangement.

My last consulting project was carried out more than 90% remotely with the occasional client visit. My equipment consisted solely of GotoMeeting.com, some data quality and data discovery licenses and a powerful laptop. By networking and finding skilled resources I was able to scale up and down as the work dictated.

TIP: I have found that the cost of data quality and data discovery software is not prohibitive. By developing relationships with vendors you can often obtain licenses either for free or at very low cost as they are keen for their solutions to be demonstrated in front of future prospects.

I think this kind of mobile working represents a fantastic opportunity, particularly for data quality professionals who may not wish to live in the more expensive urban areas. The costs of working remotely are not prohibitive and make sense for both organisations and professionals alike, particularly when we explore the recent trends in the data quality technology market.

Data Quality software takes to the cloud

Speaking to a number of data quality software vendors it is clear that managing data quality in the cloud is featuring heavily in their roadmap. Some vendors have already staked out their position in the online market and I know of at least three more that are about to follow.

This is excellent news for mobile workers and organisations.

They will now have the option to manage the technology elements of data quality management, largely remotely. Good news from a physical security aspect too as we no longer need to carry encrypted data sets around to perform analysis away from client sites. Obviously there will be concerns regarding online data security but elsewhere these issues have been addressed so I cannot perceive any major barriers in the future.

This also opens up the opportunity to hire data quality specialists based on their expertise, not just their location. In particular, the technology focused aspects of a data quality role such as data profiling, data discovery, data quality rule design, data quality monitoring - call all be undertaken remotely.

Pros and cons of the Data Quality Practitioner 2.0

The benefits of employing a new breed of mobile data quality practitioner are numerous:

• Cost - you could hire a resource on a time and materials basis, even sharing the resource across different areas of the organisation, equipment costs will be lower and cloud based data quality management will certainly be cheaper
• Flexibility - the ability to flex your service needs, scaling up and down through the use of other remote workers is a clear advantage
• Time - finding skilled specialists should become easier for end-organisations and recruiters, particularly as most practitioners are online in some capacity
• Experience - no longer constrained to searching for skills in a limited geographical pool you can select resources with more varied experience

As ever, there are potential drawbacks with this new way of working:

• Security - operating remotely or in the cloud will always present some risk
• In-person engagement - one of the biggest drawbacks is that the modern data quality practitioner must spend a fair degree of time consulting with the various business, technical and management communities to help mature the approach to data quality, obviously a challenge remotely but getting easier with online collaboration tools
• Certification - there needs to be a formal method to certify the skills of a data quality practitioner so that faster recruitment of mobile workers can take place, some progress in this area but not widespread so far

The Data Quality Practitioner 2.0 Toolkit

Finally, if this has whetted your appetite to go mobile here are some tools that you may want to explore:

• Basecamp / Huddle - online project collaboration tools
• LinkedIn - social networking and finding data quality practitioners online
• Data Quality Pro Practitioner search - coming soon
• GotoMeeting / DimDim - both offer flexible online meeting solution, DimDim has a neat online whiteboard (and is a lot cheaper)
• DabbleDB - for creating project specific data stores (*** SEE UPDATE BELOW FOR ALTERNATIVES, SOLD TO TWITTER***)
• CIMP from eLearningCurve - online education site, now has a certified data quality track
• CIQP Certification from the IAIDQ - still appears to be in the planning phase however
• See also: 7 Productivity Tools for the Innovative Data Quality Leader

What do you think about the prospect of mobile working in the data quality profession? Perhaps you work remotely, what are your experiences?

Please share your views in the comments section below.

Recommended reading:

• 7 Productivity Tools for the Innovative Data Quality Leader
• CIMP Certification Explored: Interview with Dave Wells of eLearningCurve
• WANTED: Data Quality Entrepeneurs
• 5 Resources For Finding Data Quality Jobs
• 15 Tips for transforming knowledge-workers into a data quality task force

To help our readers learn more about Solvency II, we spoke with resident expert panelist Stephen Mills, Business Analytics Lead for Insurance, UK at IBM Global Business Services.

Solvency II - Expert Interview with Stephen Mills of IBM Global Business Services

Please listen to the interview using the video player below.

If you have questions following this webcast please enter them in the comments section below or contact Stephen directly via his expert profile: Stephen Mills.

Recommended reading:

Solvency II Wikipedia entry

FSA (UK financial services body) Description of Solvency II

European Commission Solvency II overview