Data Quality Edge

A gold DQ team!

noreply@blogger.com (Daniel Gent) — Wed, 03 Mar 2010 03:31:00 +0000

With the Olympic Games come and gone, and the NHL hockey season fast approaching the playoffs I can not help but take a moment, although brief, to state that a golden data quality team should resemble a golden hockey team.

Now I’m not going to say who‘s perfect in hockey, I’m not here to wave flags. It’s the skills that count. In data quality you want people who have all these skills:

The offense:
-- Put the pressure on the other team (the delivery team, the source team) to get what they want – a winning situation. They proactively look for the bad data, and get rid of it and create plans for cleaning the data or preventing the situation from arising again.

--You need people with technical skills, people who can read and analyse data models with ease.

-- The offense will come back and support the defense to help them get that bad data out, based on their knowledge and skills.

-- They have the ability to set the stage for an opportunity for improvement.

The defense:
-- The defenders on a data quality team, see the big picture.
-- They know what’s coming down the pipe before the others do.
-- They drive bad data to the corners to prevent it from getting in.

-- They keep the bad data out by supporting the offense.

The netminder:
-- If the defenders see the big picture, the netminder sees the wall it’s hanging on.
-- The netminder’s job is solely to ensure bad data does not get in. If it does, then you will be basing some bad decisions based on their mistakes.

-- Has quick reflexes, and understands exactly how to prevent the situation from occurring.

Each and every member on your team will work together to support each other, they will have skills from each category. As a team of data quality analysts, they must ensure they keep your reports, databases clean so your decision makers can make sound quality choices for your organization. Now that's golden.

Coaching Data Quality to Skate on Ice

noreply@blogger.com (Daniel Gent) — Fri, 29 Jan 2010 15:29:00 +0000

Learning to skate is much like learning data quality.

It holds great rewards when done right, and on the other hand it can be cold and unforgiving.

When you take a new skater you give them a helmet and take them by the hand, lead them onto the ice.

-- A new analyst, you take them in, show them around, show them the work and the data, the data model and the business. You are there to help. You tell them if they have any questions to stop by.

You teach them how to bend at the knees, and push off with their legs, first one side and then the next.

-- You show them wher to go in the data architecture to find problems and correct them. You show them that even if a user is saying it's wrong, it may very well be that the user does not understand the data at all and it is correct. it's all about positioning.

You coach them how to stop, else they will keep on going into the boards.

-- You coach them that they must stop. They must stop and look, and always assess what is the bets return to the business.

You teach them balance and composure. You give self-confidence so that they feel the cool air on their face and have the ability to take a step forward and skate on their own, and yes they will fall.

--They will succeed and yes they will fail. However, with the proper coaching they will not give up and they will push on for the betterment of your organization.

Some TLC for Your Data

noreply@blogger.com (Daniel Gent) — Fri, 04 Dec 2009 19:17:00 +0000

Did you ever wonder why the data error was entered into the system, database, or report that’s right there in front of you?

You can look at it a hundred different ways.

-- The customer entered the data wrong on the website.
-- The call centre rep, entered the data wrong on the website.
-- The sales rep, forgot to enter his sales for the month of September and keyed it in for October.
-- The programmer entered the wrong statement in the data integration script.
-- The programmer put ‘greater than’ instead of ‘less than’ statement in the summarization script.
-- The business analyst did not provide the correct data retention requirements, and that’s why you have 6 months of summarized data vs. 16 months.
-- No one could come to a consensus on the definition of the value, that’s why we have 187 values for that field.

These are just a few reasons bad data is where it is.

What I’d like to know is what are your reasons, or the reasons you’ve heard. Feel free to let me know, send me a list of reasons why bad data existed in your system, application, database or report. I don’t want high level reasons, let’s have the granular reasons. When I get to 101 I’ll publish the list for all to see (no names or course).

However, here’s another reason to think about it. It’s apathy.

Really, really.

To have good, quality, accurate data all you need is a little TLC. For data to be accurate people, need to care just a little more about what they are doing. In the above examples, if people gave a little TLC there would be no bad data.

We live in a rushed, hurried world where everything is needed yesterday so a little TLC is hard to come by.

Book Review: Viral Data in SOA

noreply@blogger.com (Daniel Gent) — Thu, 12 Nov 2009 14:42:00 +0000

“Virus: A microorganism smaller than a bacteria, which cannot grow or reproduce apart from a living cell. A virus invades living cells and uses their chemical machinery to keep itself alive and to replicate itself. It may reproduce with fidelity or with errors (mutations)-this ability to mutate is responsible for the ability of some viruses to change slightly in each infected person, making treatment more difficult.” Medicine.net

In the early stages of the flu season, it is only appropriate we take a quick look at viral infections.

With discussions about service oriented architecture, concerns about data quality, and data management will become highlighted to any organization. As bad data infects one portion it will easily flow through to other modules, databases, process flows, reports and decision points of your company. One must be vigilante in monitoring data, and managing it.

What did I like about this book, everything. What didn’t I like about I, not much.

“Service –Oriented architectures are intended to encourage solution builders to create offerings that can readily transcend point-in-time solutions.”

The author, Neal A. Fishman, talks about data governance and the critical role communication plays. He identifies that data governance can also be handled in a proactive and reactive manner. He identifies what needs to be done to enforce data governance in a SOA environment, and how the control points can govern data quality. Those points being:

-- 1. Ensure: Controls for operating
-- 2. Assure : Controls for performing
-- 3. Insure: Controls for sustaining
-- 4. Reassure: Controls for continuity

He describes data quality and data governance in great detail within the SOA environment and the author states:

“The effectiveness of data governance depends on how the governance body reacts and adapts to the cultural environment.”

With that the author describes the dialing system to tweak operations. ED-SODA provides the dimensions needed to adjust the data governance process. It can be used for virtually any culture if not all.

And if you are having issues with building a data governance model, step into the reference model, for this is where you will get your basics for developing controls in data governance.

Even though the book is about data in a SOA environment, this is a book for every data analyst particularly the sections on data quality, data governance, and a myriad of thought provoking points throughout the book. and how bad data can become viral. These points and examples of what others have done will provide insight into your own issues and processes.

September Festival del IDQ Bloggers

noreply@blogger.com (Daniel Gent) — Fri, 02 Oct 2009 16:01:00 +0000

With the month of September come and gone, the changing colour of the leaves starting, hockey season starting and the wind getting colder by the day, we found another month filled with interesting posts about data quality. This month I am happy to say, I'm hosting September's "Festival del IDQ Bloggers".

An annual data quality blogging carnival held by the International Association for Information and Data Quality, an international not-for-profit association dedicated to the development of the data and information and data quality profession.

The following is a quick list, I'll say quick but it actually took an excruciating long time to compile, split coffee on my keyboard, hit my head on the light and I stubbed my toe in the process ;-)

On with the data quality blog round-up...

From the DoBlog (http://obriend.info/) the personal blog of Daragh O Brien, IAIDQ Director and Information Quality consultant and writer. Since 2006 Daragh has been writing about Information Quality related topics (amongst other things) on this blog and has even won an Obsessive Blogger award for his writing on Information Quality topics.

We find two posts of interest one about the Law and the other about Market Research.

Blog Post: http://obriend.info/2009/09/25/finding-red-herrings-or-missing-a-trick/

Market Research often falls foul of poor quality information about the target sample population. In this post Colin Boylan (a freelance Market Researcher) discusses some of the issues that can lead to you chasing Red Herrings or just Missing a Trick.

Colin Boylan is a freelance market researcher living and working in Ireland. He has worked with many of the leading market research firms in Ireland and the UK, with particular experience in Pharmaceutical studies (where good quality data is essential).

Also in the same blogging journal we have an interesting tale about the law looking at data quality...

Blog Post: http://obriend.info/2009/09/29/a-game-changer-ferguson-v-british-gas/

For about 4 years, Daragh has been hammering on about how poor quality information can and could get an organization sued. In January it happened, with a very clear and explicit ruling in the Court of Appeal for England and Wales that sets a very interesting legal precedent (binding in England and Wales and persuasive in all other Common Law jurisdictions such as Ireland, Canada, Australia, India, USA, Pakistan....). This post (based on an article Daragh wrote for the IAIDQ in April) looks at that case and the implications for Information Quality professionals.

Daragh O Brien is a Director of IAIDQ, a Fellow of the Irish Computer Society and, after escaping from indentured servitude in a leading Irish Telco after 12 years is in the process of establishing a specialist Information Quality Management and training practice. He is also writing a book on legal issues in Information Quality with Fergal Crehan, a prominent Irish barrister (lawyer).

No Blog Carnival is complete without a post from the Obsessed Jim Harris in this short but sweet post Jim talks about knowledge and the fact that we know what we know, and we don’t know the rest. Something to think about as you read Jim’s post.

Blog Post: http://www.ocdqblog.com/home/the-fragility-of-knowledge.html

Jim’s OCDQ blog is an independent blog offering a vendor neutral perspective on data quality. A place where he offers a diversity of viewpoints in a collaborative style environment. Jim himself is an independent consultant, speaker, writer and blogger with over 15 years of professional services and application development experience in data quality (DQ), data integration, data warehousing (DW), business intelligence (BI), customer data integration (CDI), and master data management (MDM). Jim has worked with Global 500 companies in finance, brokerage, banking, insurance, healthcare, pharmaceuticals, manufacturing, retail, telecommunications, and utilities.

Jumping across the pond and over to Sweden, where I’ll take a moment and say hi to the Ericsson’s, Brit, Mikael, Max, Guztav and Hanah, I hope all is well. Then a quick move to Denmark where, we have DQ blogger Henrik Liliendahl Sørenson. A man of many talents, who has worked over 20 years in applications, databases and data in general. Henrik has demonstrated his expertise in business directory matching and international aspects of data quality improvement and master data management.
Henrik’s blog, Liliendahl on Data Quality, is a collection of his personal opinions, experiences and observations around data quality. Accumulated over decades and I do mean decades of experience.

Henrik discusses the multi-use potentials of data quality...could it be...can data quality be used for increasing revenues, and for marketing...read on and find out.

Blog post: http://liliendahl.wordpress.com/2009/09/24/multi-purpose-data-quality/

The post has a follow up post sparked by the comments: http://liliendahl.wordpress.com/2009/09/27/process-of-consolidating-master-data/

Going across the globe and going to that lovely local known as Australia we have Vincent McBurney a manager with Deloitte Consulting in Australia and has 15 years as an application programmer, database programmer, ERP implementer and information management consultant. The blog is dedicated to a tool based approach to data integration with news and tips on IBM InfoSphere, Informatica, Oracle, Microsoft and any breaking data integration news.

This particular post is interesting because it talks about something we all like - fudge. But not this fudge, no way, this fudge is actually fudging moments and then having to apply some kludge techniques or go in and kludge the situation to fix it. Fudge, Kludge all around a great read.

Blog Post: http://it.toolbox.com/blogs/infosphere/the-data-quality-and-how-to-fudge-it-34289

From the Data Quality Pro…Dylan Jones provides us an excellent interview with Ken O’Connor who discuss the a means in creating a data issue assessment process, something everything DQ Team should have in place, and that’s why it’s here. Coming from the trenches this is something any data quality analyst can use over and over again.

Blog Post: http://www.dataqualitypro.com/data-quality-home/how-to-create-a-data-issue-assessment-process-expert-intervi.html

DataQualityPro is an online community resource that is solely dedicated to the needs and development of data quality professionals everywhere. Dylan Jones, is the founder and editor of Data Quality Pro and Data Migration Pro, leading online knowledge centre and community sites for their respective professions.With a 15 year background in data quality and data migration Dylan now supports a global community of several thousand professionals who actively collaborate and contribute to help increase the collective knowledge in these fields.

Here’s an interesting read about data governance from Gwen Thomas. Gwen Thomas is Founder and President of The Data Governance Institute, which is the premier provider of in-depth, vendor-neutral information about, and assistance with, tools, techniques, models, and best practices for the governance/stewardship of data and information. This is Gwen’s personal blog from the Data Governance Institute.

This post is here because data quality is a big piece of data governance. Data governance provides guidance in defining quality. There is a symbiotic relationship between the two.
With this post we get a high level view of the net-centric governance and the potential issues of control one may have with it. It gets the wheels turning when you begin to think of the implications that may and could very well follow.

Blog Post: http://datagovernancematters.com/2009/09/14/net-centric-data-governance-not-for-sissies/

Finally…
After hearing about the actions of an old friend of mine working hard in a data quality team, I was surprised to learn she is still having to justify the existence of the data quality team. It’s always good to know about what you’re doing and the value you bring to the table, but this being the 3rd time within a year, I believe enough is enough…so here it is. Who am I, well I am this guy, business analyst by day, data quality analyst by night. My blog, Data Quality Edge, is really a place to voice my opinions and what I hope will provide a grassroots look at data quality, something really for the data quality analyst in the trench. Because they are the ones that get the job done.

Blog Post: http://dataqualityedge.blogspot.com/2009/09/stop-justifying-data-quality-programs.html

Wishing you all the best in the cooler months ahead! Good reading.

Dan

Stop Justifying Data Quality Programs and Do the DQ Work Already!

noreply@blogger.com (Daniel Gent) — Wed, 16 Sep 2009 19:59:00 +0000

In a recent discussion with a good friend, I learned that they are in the middle of justifying their work in a data quality team. This being said, a few months ago they were doing it as well, and at the beginning of the year they had just wrapped up another justification project, in the beginning of the economic downturn, it was being done as well. I also know that a few years ago when I was with the team, we also had to do it.

It's a shame. A terrible shame! Some organizations understand the importance of data quality, sometimes that understanding has come at a cost:

• Lost thousands to millions;
• Faced national embarrassment;
• Or made significantly big policy screw-ups.

While other organizations, are more pro-active and have established a data quality team and program to prevent such events from happening. An activity that is considered a best practice and essential to any information technology/business intelligence structure.

However, in either case, you may have someone, traditionally a senior manager, who sees data quality as a cost, a black hole. Yes there is a cost, however the benefits outweigh the costs in a variety of ways.

• Reduction in re-work due to good data quality;
• Improved incoming data quality and data processing due to pro-active initiatives with incoming data migration and integration projects;
• Proactively preventing data quality issues from occurring;
• Improved decision making, using quality data, and more.

To my old team and senior management:

Stop with the justification exercises and begin looking at the benefits and what this dedicated group of data quality analysts have accomplished year after year.

• Recognized Finalist Best Practice by TDWI in DQ;
• Hundreds of data modelling, metadata, data processing and data corrections to incoming projects per year;
• Proactively seeks data processing improvements to improve data loads - ultimately reducing costs;
• Client support to decision makers who really don't understand the technology aspects of the data and its routines;
• Dozens of change management practices each year to improve data quality and data processing which collectively prevents lost revenues, increases sales and manages maintenance costs by reducing reruns and supporting programs such as customer profitability, and other CRM initiatives.
• The estimated benefits weigh in at an average of $1-1.5 million a year if not more.

Another justification exercise only takes the team away from doing what needs to be done, data quality.

So to the senior management in this organization and any other, yes there is a cost to any data quality program. Just remember a data quality team is your vanguard to any organization that deals heavily in data. They bring in benefit. They enable your decision makers. They protect your greatest asset - data!

A good DQ team = Great Value!

New to Data Quality Analysis Try These “9+1 Things To Do”!

noreply@blogger.com (Daniel Gent) — Fri, 28 Aug 2009 13:01:00 +0000

Did you just get moved over from one data warehouse support group to another? Do you know nothing or very little about the data in your new data warehouse? Or are you new to data quality analysis and want to get started on some solid footing?

The following post by Sylvia Moestl Vasilik “9 things to do when you inherit a database” at SQLServerCentral.com is an excellent article for anyone jumping into a new database environment, regardless of the environment\vendor or type of database relational\columnar, Sylvia’s 9 things to do can be applied anywhere.

Building on those “9 things”, if you are less technical and more into data quality analysis or into a data steward role, I recommend adding a 10th thing to do … begin and complete a data profile.

A solid data profile will provide you with a wealth of information and more. A solid data profile will provide you with some interesting insight into the data. Here are a few items that you should be able to capture with a good data profile project.

-- You will gain an understanding of the completeness of the data, you’ll see what’s missing and you can begin to ask the questions to the business users why are we missing this component of the data set(s).

-- How accurate is the data, does it meet the initial requirements or not. How often does a job fail because of bad data; have you lost customers, revenues or received fines due to bad data? You’ll discover soon enough how inaccurate data affects your organization.

-- How timely is the data? Do you have real-time, near real-time or less timely data. Is your data arriving late, on time or not at all? How long is the data relevant for, this will be important for you, your users and maintaining the environment.

Just remember focus yourself first on the most important data, the highly used data, then you can spread out and tackle the rest of the datawarehouse. Make sure you have senior management approval, and are able to prioritize the other 9 things to do along with this one.

Other items you can gather while running a data profile project can be identified from the following post, 5 Non-Quality Items to Consider in Data Profiling.

The Art of Pickling Data

noreply@blogger.com (Daniel Gent) — Mon, 24 Aug 2009 13:27:00 +0000

I have never pickled before, and probably won’t but I do enjoy eating them. The best tasting pickles one can imagine were pulled out of our 69 year-old backpacking mountaineer\pickling savant companion’s backpack last week. Yes, he brought a jar of pickles into the mountains, which we all enjoyed and devoured. So to the man known as ‘Uncle Dave’, I salute you and here’s a little analogy of pickling data. Besides who doesn’t like a good crunchy pickle.

-- 1) In pickling we need to sterilize the equipment. Otherwise you may get contaminants that can ruin your pickles.

In datawarehousing, we need a computer or server to store the data electronically. You want to start with a clean server to maximize the amount of data you can store and to ensure no ‘cross-contamination from old tables’. I haven’t heard of this happening other then in mainframe environments; where the back-end data from ‘shadow tables’ can still come back and repopulate the ‘main’ front-end tables. If the bad data was not removed from both the back-end and front-end tables simultaneously contamination will happen.

-- 2) Prepare the brine with salt, vinegar, garlic and other spices/ingredients to create your pickling solution, bring to a boil.

Prepare your scripts, data loading jobs, data models,tables, attributes, your data quality routines and more. I included data quality routines because you want to study the trends determine when they break from the norm. Data quality is the spice that will make it all better.

-- 3) Boil vegetables place in jar with pickling solution, and seal.

Prepare your files and run the jobs to load the data in your repository.

-- 4) After a few weeks, enjoy the pickles of your labour, the crunchier the better.

Unlike pickling, you can begin to enjoy the crunchy bits of your data and what they are telling you immediately after the data is stored. It might not be tasty but it may very well be interesting. After all, the interpretation of data is information, and information is power.

3 Easy Checks to Ensure Quality Analytics Reporting

noreply@blogger.com (Daniel Gent) — Wed, 12 Aug 2009 12:27:00 +0000

Are you new to business intelligence, predictive analytics or analysis in general. With all the talk about data visualization and how effective it is in telling a story; well it’s grand!

Here’s another way of looking at it – really, data visualization allows your VP to review the report over a cappuccino and reading emails on their Blackberry, while talking on the phone, and waving in the admin assistant (I believe that’s the politically correct term).

So how often are decisions made based on bad reports. Or bad decisions based on good reports.

Let’s just say the data is 100% accurate, is it still possible to have less than stellar reports (I was going to use a word that rhymes with ‘happy’, “but les than stellar” – works as well).

Here are three simple things you can do to make sure your reports are good quality reports.

-- 1) Pie charts: they are suppose to equal 100%, so if you have a chart that totals 103%, ask yourself what happened and fix the calculations.

-- 2) Rounding Issues: Some old applications and databases drop the full value of the numbers so that they may round the value, and some modern day applications as well (depending on your settings). If this is the case, remember to sum your columns to ensure that the value you are reporting as the total is the actual total. The other problem with rounding is lost data, and increased values. Your 6.25 or 5.87 value can each be rounded to 6. So be careful. Choose wisely!

-- 3) Decimal point: If you’re doing a report on the amount of sales you perform on your products, or services, just make sure the decimal place is where it should be. Having the decimal place in the incorrect position will ruin everything! You don't want to say your selling Widget ABC for $1.10, when it's actually $0.11.

Decimal Point Facts:

· Remember for doctors, pharmacists, nurses entering the decimal point in the wrong place can actually kill someone (i.e., incorrect dosage).

· We all know that Popeye’s spinach fetish had more to do with the decimal point then with actual iron content (How often did you hear, eat your spinach it'll make you strong like Popeye, when you were young, when all you wanted was a nice juicy steak).

· How often do banks withdraw 5500.00$ instead of 55.00$?

Something else to think about.

If you’ve checked the above three points in a report that has been produced for eons, and these errors are happening, and no one has said anything, not a word, nothing! Then there’s two things going on…no one is using that report (imagine that for a moment, no one is using your report, what a waste of resources). Or your organization will eventually feel the effects of bad reporting in the financials or other areas of the organization.

Pooh, Data Analyst

noreply@blogger.com (Daniel Gent) — Mon, 10 Aug 2009 12:00:00 +0000

Obviously Data Quality is an important issue. But so is good communication and having a good data analyst around!

After story time with my daughter, taking some elements from a classic that everyone loves…saying no more…let’s stop in and take a look at what’s happening in the “Hundred Acres Woods Data Centre” with our friend Pooh, Data Analyst!

In ran Piglet on his short stubby feet, “Rabbits coming, and he is doesn’t have happy feelings with him”.

“Pooh bear, you’re not thinking again!” bellowed Rabbit as he marched into the Hundred Acres Woods data centre.

“Well, thinking is often a hard thing to do, without a smackerel of honey.” commented Pooh Bear.

“POOH BEAR!”

“Oh I think I have a headache!” said Pooh.

“How is it bear that you have it wrong, the price of honey bottomed out, it’s less valuable then the pebbles on the floor. You said honey was the commodity of choice. That it is was going up. Eeyore invested everything, and now he doesn’t have anything.”

“Oh that is so sad! What is it that he lost everything of?”, asked Pooh.

“EVERYTHING!”

“We need to hoosh, Eeyore!” interrupts Piglet.

“Yes”, agrees Pooh bear, “but maybe a little less hooshing this time then last time, would you not agree Piglet?”

“No”, said Piglet, “Yes”, replied Piglet quickly afterwards.

“What, NO!”, cried Rabbit, “ You can’t hoosh Eeyore, he’s suing us for more sticks so he can build a new house.”

“Suing?”, interrupted Pooh and Piglet together.

“Yes suing, Eeyore’s going to sue us and we’ll have to work for him, looking for sticks and other stuff. Now tell me why did you say honey. It’s an important thing to know.”

“Yes that’s what I saw on the computer screen.”

“Show ME!”

Pooh, waltzed over to the terminal and showed Rabbit the honey dripping off the computer terminal.

“Poor Bear, you spilt honey on your screen, the server, the computer. It’s everywhere! That is not good analysis of the data. Now print up the report and bring it to me. And I’ll analyze it!” said Rabbit, as he stormed out of the data centre.

A few minutes later, “Here’s the report Pooh”, stated Piglet as he hands the report to Pooh Bear.

“Mmmm!” said Pooh

“What’s mmmm?” asks Piglet.

“I think Rabbit is wrong, I have honey on this paper report too. I think we should stay with honey.”

“Pooh, you have been eating honey, it’s on your paws. A smackerel smudged your report.”

“Yes, Piglet! I believe that’s a good thing.”

DQ Alert: Easy Savings by Removing Dups

noreply@blogger.com (Daniel Gent) — Wed, 05 Aug 2009 13:32:00 +0000

Poor data quality costs over $600 billion annually as per TDWI surveys and studies.

Whether you are the VP looking over your latest sales, the secretary compiling a mailing list, the data quality analyst rummaging through data sets, the business analyst working on a data integration project, or an accountant going over the projected budget.

Whether you are in a large Fortune 500 company, in the government, or a small community association you will benefit.

Duplicate records

Duplicate records of customers cause discontented customers and multiple mail-outs. How is it possible to have duplicate records?

-- Records are manually entered twice;
-- Processes create record twice;
-- Participants registered under multiple names;
-- Participants registered at multiple locals.

So before you send out a mailing list to your community members, or potential program participants for marketing campaigns or program sign-ups for this year’s sports, arts, sales, and/or membership drive seasons:

-- 1. Sort that list by name. Why because you may have the child on the list twice, or three times; after sorting by name;

-- 2. Sort by address, you may have the same family household multiple times because one the parents registered under their name, and/or siblings are registered with your organization as well.

Don’t know how to sort…here’s a good way to start if you’re using Microsoft Excel…highlight your records and click on this little icon , it’s easy as breaking eggs.

Or

New SQL type programming here’s a simple query that will identify duplicate records.

SELECT attribute1, attribute2, attribute3, attributen… count(*)
FROM dbo.tablex
GROUP BY attribute1, attribute2, attribute3, attributen…
HAVING (COUNT(*) > 1)

I hope this helps anyone working with customer lists..

Sun Tzu and the Art of Data Quality (Part 3)

noreply@blogger.com (Daniel Gent) — Mon, 20 Jul 2009 12:25:00 +0000

Concluding a three-part series of several Sun Tzu teachings in the Art of War and how they may apply to the Art of Data Quality.

Sometimes it’s all about the troops in your camp.

“To begin by bluster, but afterwards to take fright at the enemy’s numbers, shows a supreme lack of intelligence.”

Are you the type of data quality analyst who wants everything. Do you take all challenges on so you look good, or don’t trust others to do the job right; well perhaps you should think before you take it on. Because if you ever ask for it, and can’t deliver the goods you will look like a fool, and be considered unreliable, a definite career shaker.

“He who exercises no forethought but makes light of his opponents is sure to be captured by them.”

Closely linked to the previously mentioned teaching. In this one we learn that if you do not plan (use forethought), you may find yourself in dire situation. Remember the general rule of thumb. Eighty percent planning-twenty percent execution.

“Hence the experienced soldier, once in motion is to set up one standard of courage which all must reach.”

It always pays well to have experienced staff in your data quality competency center. They are active, they know the data, they know the business, they’ve helped developed the standards required to make your data a high quality asset. Another aspect of an experienced analyst, they are the ones to establish the governance that can be applied to almost any situation. In their absence, bad data can be targeted and the result will be a corrected situation with quality data at your disposal; all based on their governance input. A lot of people won’t do this, for fear of job security, it’s those rare individuals that do it, that will move your organization forward. So to you I say, keep the courageous.

“Success in warfare is gained by carefully accommodating ourselves to the enemy’s purpose.”

Data in general always has a purpose. Bad data has no purpose, it is created because someone, or something, somewhere ‘screwed up’. You cannot accommodate yourself to thepurpose of bad data, but you can accommodate yourself to ‘high-traffic’ areas, areas that are most likely to see bad data come-in. To sum it up, be prepared and recognize those areas in your model where bad data will rear it’s ugly head and be ready to chop it off.

In conclusion, I’d like to quote Jim Harris’s comment from the first Sun Tzu article. A paraphrase from Sun Tzu.

“Hence it is only the enlightened executive and the wise leader who will use the highest intelligence of the enterprise for purposes of data quality, and thereby they achieve great results. Quality is an important element in data, because upon it depends an enterprise’s ability to succeed.”

Sun Tzu and the Art of Data Quality (Part 2)

noreply@blogger.com (Daniel Gent) — Mon, 06 Jul 2009 12:14:00 +0000

Continuing with the ancient teachings of Sun Tzu and the Art of War, we can gain further insight in the actions needed for building a solid foundation for the Art of Data Quality.

Sometimes it's about timing.

"The quality of decision is like the well-timed swoop of a falcon which enables it to strike and destroy its victim."

Place the data quality analyst in the position of a falcon and your enemy is that lovely data error that needs to be utterly removed from the system. Ultimately, the data will ony be removed by a decision made by the data quality analyst or promoted to higher-ups to decide upon, again identified and promoted by the data quality analyst. The quality of your decision and your analysis has significant impact on how your data repository is viewed by others.

"Rouse him, and learn the principle of his activity or inactivity, Force him to reveal himself, so as to find out his vulnerable spots."

Taking this teaching into account, we can state that Sun Tzu may have been telling his military students to be proactive. A proactive approach to seeking out and removing bad error to improve your data quality is essential.

Taking into account the definition of proactive, "Taking the initiative by acting rather than reacting to events."

Jim Harris, talks about a hyperactive data quality and mentions a proactive approach to handling problems from coming in. That approach looks at prevention. Prevention is a significant key in proactive data quality. Incorporate Sun Tzu's teachings in proactive data quality and you will be vigorously looking for errors, you'll be data profiling like you've never profiled before. Nothing wrong with that either, it's may lead to unwanted discoveries or great opportunities for data cleansing.

"Do not repeat the tactics which have gained you one victory, but let your methods be regulated by the infinite variety of circumstances."

I'm all for standards, governance, policies and processes to get the job down. The way the job should be down. BUT, you also need to be flexible and adaptable. You may have just saved the company a great deal of money with some fancy DQ manoeuvre. However, just remember the next problem that comes along does not necessarily follow the same trail in. So, you must know your systems, processes and be prepared to make adjustments to your tactics to clean that data quickly and effectively.

"We cannot enter into alliances until we are acquainted with the designs of our neighbours."

Not so much about timing, but an important message nevertheless. When developing a service agreement with internal client groups, external customers, vendors, reporting teams, or anyone that needs your data. Be prepared to get 'acquainted' with them. Wine and dine them, understand their needs, desires and limitations. What is it that they want and why they want it? If they are providing provisioning statistics to regulators why do they need marketing campaign statistics. Why do they want the data on a weekly basis when they only analyze it monthly. Get to know them, so you can better serve them. Remember data quality is not just about bad data or good data, it's also about relevant data.

In conclusion for this post, I'd like to say,

"Ponder and deliberate before you make a move."

Remember "haste makes waste", plan and time yourself carefully and know your facts.

Sun Tzu and the Art of Data Quality

noreply@blogger.com (Daniel Gent) — Mon, 29 Jun 2009 13:10:00 +0000

Having read Sun Tzu’s Art of War I’d like to say its genius in its simplicity. This is a book that is referred to by many people to many things. Politicians, business leaders, professionals and even educators refer to the man’s ancient wisdom. This time I’d like to refer to it for - the Art of Data Quality.

Sun Tzu’s book has many teachings one can examine, we’ll only draw out a few and how they can be related to data quality.

“The art of war is of vital importance to the State.”

Liken State to Enterprise, and we come to understand that data quality is a critical and vital importance to any organization that handles data. Pointing out the following three examples and we see just how bad data can impact the organization and/or it’s customers, the life blood of any organization.

1. Millions of dollars giving away to a pair of customers who flee the country.

2. A database that holds 7 versions of one customer.
3. A unique health number assigned to two children living 90 miles apart.

“It is a matter of life and death, a road either to safety or to ruin. Hence it is a subject of inquiry which can on no account be neglected.”

It can not be neglected! You know that, the people that read this post know that. When things are a matter of life and death, it must become viral, it must be communicated. Bring it to the forefront of this week’s status meeting, bring it to the board room. If you are IT talk to the business people about it, if you are business talk to your IT team. It’s everyone’s job.

People like Jim Harris, Dylan Jones, Daragh O’brien and so many more. They all get it, they talk about it, they live and breathe it. After reading their posts do YOU say great post and leave it at that, or take it and tell others in your enterprise about it, spread the word, tweet it, present it, talk about it. Remember, Data Quality cannot be neglected. Billions of dollars are wasted each year because of poor data.

“In war, then, let your great object be victory, not lengthy campaigns.”

Sun Tzu dedicated a section to the length of military campaigns, and the importance of having shortened campaigns vs. long ones. This can be applied to data quality as well. The importance here is every organization has limited resources, (people, money and time). You must use your resources effectively so that your teams’ moral does not become demoralized because the bad data is still occurring or re-occurring after a lengthy cleansing project. You must work efficiently and effectively so that the bad data can be isolated and removed, whether it is a small billing adjustment that went wrong or a terrible boondoggle that was created, and let’s face it we’ve all seen boondoggles.

“If you know neither the enemy nor yourself, you will succumb in every battle.”

In every case where you have bad data sitting in a repository you will want to clean it. In preparing for battle you need to know two things. Your own level of skills, can you clean the data yourself or do you need expert technical help. The other you must know the data. You must understand the data warehouse and it’s structure, else you will succumb to the situation. You may even be the cause of future situations of poor data. So plan and be prepared.

In conclusion I’d like to remind you all that,

“The art of data quality is of vital importance to the Enterprise.”

Entry Point: Architecture or Crumbling Foundation

noreply@blogger.com (Daniel Gent) — Mon, 22 Jun 2009 18:58:00 +0000

Let us talk for a moment about architecture.

Good architecture is built to last, to withstand the elements and the test of time. Good data architecture will allow you to extract data quickly, will help prevent data errors from occurring and promote easy integration of future data assets.

Bad architecture will see the following persist like vermin in your basement:

1) Increases data retrieval times;
2) Data retrieval becomes more difficult;
3) Integration and migration projects become cumbersome;
4) Fosters the spread and creation of bad data.

Soon the walls around you will begin to crumble as more and more data becomes questionable. Your users will question the data and eventually your system will become synonymous with the term “poor data quality”.

When building your data warehouse remember the following:

1) Ensure you size it properly and measure future capacity for continuous growth;
2) Bad data does occur, allow it to be cleansed by your data analysts, don’t build overly complicated data models, remember the KISS principle;
3) Improves speed to delivery and reaction time;
4) Improves query and data retrieval times.

When defining your architecture and/or database system remember the following steps to help prevent bad architecture from occurring:

1) Define the objective of the data warehouse;
2) Research the data and datasets (understand the business and it’s processes);
3) Design the data model;
4) Define the database relationships;
5) Define rules, triggers and constraints;
6) Create views and/or reports;
7) Implement it.

Entry Point: Change is a Constant

noreply@blogger.com (Daniel Gent) — Tue, 02 Jun 2009 01:46:00 +0000

How many times have you received bad data from an upstream ‘stable’ database environment for no reason what so ever?

1…
2…
10…
13…
76…

Never…?

How’s this for a reply…no environment is stable! PERIOD.

Each and every data warehouse environment is subject to change, subject to growth, subject to budget constraints, and other external conditions (i.e., political changes). Their will always be change, THAT you cannot control.

Remember SOx. One recent example in Ontario is the Harmonized Sales Tax move. This means for those organizations tracking tax in their internal systems, they must make changes to their databases and systems to incorporate the HST and alter their PST and GST taxation collection and tracking in Ontario. This is a nice example of an externally forced upon change. This particular impact will impact both Operational and Decision support systems.

Personal Experience:

The data files were coming in just fine from a source system that was considered stable (i.e., no data issues from them since project delivery).

Then one day, the volume dropped by more then 70% on file feeds received daily.

After some investigation the source system (System B) identified that their volumes had changed as well. They did not even know their data volume had decreased. The investigation was escalated to their source (system A).Who identified that all the records where being sent to system B. There were no change to System A, “more on that shortly”. Back to System B, they do not have the data. Open the source files and their the data was, the records that were missing were in the source files with blanks in the identifying fields.

A scheduled software (note scheduled) upgrade on System A, could process French characters and inserted blanks in the initial fields and subsequent ID fields. So when System B arrived to pick up the record IDs it found nothing to insert.

A simple software upgrade that resulted in wasted time, money and missing data.

Ignore the potential for change and you will be left holding an empty bag. Never get comfortable.

Remember to inform all your upstreamers of how their changes may potentially become a critical impact to your environment.

Some of the most common forms of changes in systems are the result of the following items:

Data integration
Mergers and acquisitions
Politics, laws and regulations,
Software changes
Web interfaces (change of portals)
Hardware changes

I’m certain you may be able to add to this short list, and I welcome you to.

If you are someone making change, remember to practice proper Change Management technics, a topic for future discussions.

DQ is 1/3 Process Knowledge + 1/3 Business Knowledge + 1/3 Intuition

noreply@blogger.com (Daniel Gent) — Tue, 26 May 2009 00:49:00 +0000

Sitting in any data quality competency centre, or working on any data warehousing support team, there are times when you come across data that just screams at you, and not in a good way. (Kind of like the ghost from the Grudge.) The data's gone through all data processing without a hitch, business says it's good data, but something tells you it's just not right. That my friends is called intuition.

My recommendation would be to use it.

Intuition does not come naturally, it comes to you over time. After you've gained an understanding of the processes and learned about the business, intuition will set you above the other data analysts.

Business Knowledge
Know the business, I've mentioned this before in my post about Data Quality Analyst Attributes.

What do you do when you have 23,000 service cancellations on one order. This would be red flagged immediately in most service oriented companies. It might even cause a job to stop processing.

The process tells you there's a problem. The business would rightfully tell you to investigate, it can't be right. However, someone in the business knows what the truth is. Check the data, you just may find the indicator to identify the type of customer it is, and easily learn that the business is justified in cancelling 23,000 service items. Perhaps the business lost a call centre customer.

Process Knowledge
Processes will establish checks and validation points that tell you when the data is good and bad. It will tell you where the data is and what happens to it at specific system touch points and more. You must know this in order to understand the process and the data that goes through the processes. Having a good understanding of the processes will allow you to identify where and when errors could occur with your data sets. Process knowledge will guide you to where you need to correct the data or processes.

Intuition
Your knowledge, your understanding of the data will guide you and your intuitive feeling to determine what is right. There is no substitute for it. Intuition will give you that sixth sense and you will be able to differentiate true data issues from false data issues when the tools and processes set in place cannot make that differentiation. Use it when the tools just don't cut it.

Example:
The process says the CRM application must match the records, the business believes the records should be matched and even want them to match. You know that even if John Smith, and John Smith living in the same city aren't necessarily the same person, two positives matches does not necessarily make a single profile.

Entry Point: The Call Centre or the Death Star

noreply@blogger.com (Daniel Gent) — Mon, 11 May 2009 19:10:00 +0000

Depending on the type of industry you are in, you may call it the service desk, the help desk, the call centre, or even the support desk.

When it comes down to it, it is a consolidated (or virtually connected) team of employees working for one or multiple organizations to provide support services or general sales and services to the organizations’ customer base and potential customer base.

The call centre, as many refer to it as, is a major data entry point for many organizations and large corporations. Fortune 500 companies may handle thousands of calls a day. Each call provides new data for the company. All that data will be used in some way by someone so that they (the company) can understand their customers’ needs and wants.

To perform good analytics on call centre data you want and you need good quality data.

For a call centre to function at it’s highest potential and to provide good solid customer service they want and must have good quality data at their disposal.

"Personal Experience:

After calling our television provider and explaining to them that the PVR is not functioning normally and spending an hour on phone until they finally agreed upon handing the headset over to a manager and then sending a new PVR to our home.

After receiving it, we had to call back initialize the PVR code. Of course it’s easier then it sounds, because they had no record of us receiving the PVR? What’s up with that! So we went through the explanation about getting the new PVR and had it initialized, they stated it would take 2 hours.

2 hours pass and nothing.

Call back…wait until the morning, sir.

Morning…we have 1 viewing package from the 4 we pay for. Apparently somewhere in their systems a call centre operative indicated we cancelled our subscriptions. I need my TSN, so no I was not happy.

And they didn’t fix this when we tried to initialize it the first time?"

I once read years ago that the number one cause of data quality issues are caused by employees. Case in point. The above personal experience is just an example of how untrained employees in a call centre environment can affect your data quality, your reporting and your decision making.

Looking at the above, some reporting systems will show the knowledge worker: 1 lost customer, one gained customers, or even a winback situation, it may not even show the cause of the change, faulty equipment? All because someone didn’t record the information properly.

It’s not good enough to train call centre staff to enter data and read from scripts they must be intuitive and be able to make decisions faster and better, they must know the business. They must understand what it means to enter quality data into their front-line applications and how it will impact the company. They must understand that their data entry does affect your data, your data warehouse, your customer service and ultimately your bottom line.

Entering the term ‘Death Star’ for the sales channel, might be funny for a young CSR, but let’s face it…it could blow up in the company’s face!

You talk to anyone who has dealt with call centres and you will know that the majority of people end up frustrated to no end. The majority of call centres are stepping stones for many employees who want to move on. Therefore, keeping a motivated staff, who cares about their work is fundamentally a difficult task for any call centre manager.

For many data quality analysts, I would imagine looking at the data from a call centre is like being sentenced to the 9th layer of Hell, it’s just not a fun place to be. Why? Because lets face it, trying to correct bad data from the front-line can be a cumbersome task, you have multiple systems to work through, lineage to deal with, and when you want data corrected or to set up preventative safeguards, there’s no one to call.

Do you have a ‘Death Star’ situation in your company? Think of the following to help you, the data quality analyst, out.

1) Contact: Move up the lineage line, at every upstream system, make contact with someone. Tell them your story and your issues. When they have problems they’ll let you know. And better yet, they may let you in the sandbox, and introduce you to their sources, or call-centre managers.

2) Prevention is the Key: One of those contacts will be useful and they may be able to correct the data or put in place the corrective measures needed to stop further death star situations.

3) Squeal Like a Stuck Pig: Pass the cost of making data corrections up the line, straight to the VP if you are able to. The eye-opening costs will force the VP to take action and talk to the VP of the offending data sources.

4) Educate the Young: Start a data quality education initiative targeting call centre employees and new call centre employees. Ensure that new employees understand what it means to provide good data. Talk to the call centre managers and have them incorporate data quality into their training packages. They may not be young age-wise, but they are young in terms of being employed with your company.

5) Mind Meld: Find someone with like minds in the offending department to become your champion of data quality. Use them to help preach the benefits of data quality, and consistent quality data entry. Remember, two minds are better than one.

6) Recognize the Best: As mentioned in a previous blog, implement a data quality recognition program.

Tribute to the Piano Man

noreply@blogger.com (Daniel Gent) — Mon, 11 May 2009 13:10:00 +0000

In light of Billy Joel's birthday this past weekend, I'd like to add this attempt at an Information Management and Data Quality Song. Sung to the beat of "We didn't Start the Fire".

Now I'm no song writer so if there's a beat off...just grin and bear it.

Phone Calls, Punch Cards, Magnetic Tape,
Pay Checks, Health Care, Data Management.

SAP, IBM, Microsoft, ATMs
PMP, ITIL, Change Management.

Objects, Data Points, Elements, Analysis
Reports, Entities, Attributes, Screen Freeze.

Data Warehousing, Let’s put it to the test
Data Quality, Bad Data, Do not cry.

We didn't make the data
It was always shocking
Since the server’s been running
We didn't make the data
No we didn't mine it
But we tried to hide it.

Wireline, Wireless, Voice over the Internet
Sextant, Compass, Satellite GPS.

DOS, Windows, Linux, iMAC OSX
95, 98, XP, Vista, Integrate.

Standards, Governance, Best Practice,
Service Desk, Third Line Support is the Best.

Java, XML, Perl, HTML
Web Portal, Interface, Trouble on the Server.

We didn't make the data
It was always shocking
Since the server’s been running
We didn't make the data
No we didn't mine it
But we tried to hide it.

FoxPro, SQL, Teradata, Oracle
Access, Excel, Crystal, Where to go.

Master Data Management, Business Process Management
DBM, DBA, The Data Warehousing Institute.

Data Architecture, Data Models, Stored Procedure
Parsing, Deduplication, Cleansing, Data Integration.

Business Intelligence, Predictive Analytics,
Microstrategy, Cognos, Infosphere.

We didn't make the data
It was always shocking
Since the server’s been running
We didn't make the data
No we didn't mine it
But we tried to hide it.

Agile, Iterative Model, IDEF, Spiral Model
Design, Test, Deploy, Nomalization.

RSS, PHP, ASP, Datamania
eMail, gifs, docs, Unstructured Data.

GIS, ERD, SOA, ETL
DOT COM blown away, what else do I have to say.

We didn't make the data
It was always shocking
Since the server’s been running
We didn't make the data
No we didn't mine it
But we tried to hide it.

Kick-off, JAD Session, Use Case, Swim Lanes
PM, DM, Business Analyst.

Thresholds, Trending, Anomalies,
Outliers in the chart, Data Quality is an art.

Knowledge Management, Collaboration,
Blogs, WIKI, Sharepoint, Web 2.0.

Corporations lying, Sarbanes-Oxley is the law
Market Crash, I can't take it anymore.

We didn't make the data
It was always shocking, Since the server’s been running
We didn't make the data
But when we are done
It will still be wrong, and wrong, and wrong, and wrong...

We didn't make the data
It was always shocking
Since the server’s been running
We didn't make the data
No we didn't mine it
But we tried to hide it.

We didn't make the data
It was always shocking
Since the server’s been running
We didn't make the data
No we didn't mine it
But we tried to hide it.

We didn't make the data
It was always shocking
Since the server’s been running
We didn't make the data.

DQ Certification a Noble Cause

noreply@blogger.com (Daniel Gent) — Tue, 05 May 2009 02:56:00 +0000

The International Association of Information and Data Quality has been raising funds for a noble cause. That cause to create and implement a true data quality certification designation.

For all you data management, data analysts, data modellers, data architects and other data types, this certification will provide a professional designation for your efforts in data quality.

Interested in knowing more...visit the IAIDQ website.

Or visit the IAIDQ Certification blog and CHIPIN!

You can also CHIPIN! by visiting my blog!

CRTC Fails to Grasp Simple Data Protection Concepts

noreply@blogger.com (Daniel Gent) — Fri, 01 May 2009 13:23:00 +0000

After getting a phone call for a carpet cleaning request, I gave the following some thought! In Canada this is old news, but may be new for some of you.

The United States has one, Canada has one, but do they work?

No, I’m not talking about national leaders.

I’m talking about our respective national do not call registries. Canada’s is less than a year old and holds over six million records. While the United States has a 6 year-old fat repository holding over 70 million numbers.

Since the inception of the Canadian Do-Not-Call-List (DNCL) there have been numerous complaints about telemarketers calling people. It has raised national criticism as outlined in these two articles about the registry:

Toronto Star, Tough Action can reverse do not call registry

Needless to say the American repository has guidelines and processes set in place about who can have the data in the list. One simple security feature is that only businesses with an Employer Identification Number (EIN) issued by the Internal Revenue Service can be a purchaser of the list.

In Canada, as seen in the listed link below, anyone with bogus information can obtain the list, including problematic offshore telemarketers.

Global News Uncovers Serious Loophole in Do-Not-Call List

What the politicians and the Canadian Radio-Television and Telecommunications Commission (CRTC) created is truly an effective tool not for the people of Canada but for telemarketers. This is entirely due to a lack of foresight, a lack of data management understanding and lack of basic data security practices. The fact that you can enter bogus data as identified in the Global clip, means they don't even have a grasp of proactive data quality concepts, as identified by Jim Harris in his Hyperactive Data Quality blog post.

This is an example of exceptionally poor data protection. The CRTC must rectify this as soon as possible, else they must be held accountable for their own inability to manage data.

Have they corrected their little data security fiasco? Visit the CRTC site and find out, I’m not interested in downloading a list of 6 million plus phone numbers, but maybe you are?

The Retail Data Nightmare: Coming to a Store Near You!

noreply@blogger.com (Daniel Gent) — Mon, 27 Apr 2009 21:26:00 +0000

Not to long ago in the not to yesterdays of our calendar. I visited a retail store to purchase numerous items. Now this is no small local store, it has a large presence. It’s located across Canada and across the United States.

You would think they would be in the 21st Century with their data warehouse methodologies and CRM apps, but they aren’t.

What I discovered as the clerk punched in our telephone number was that there where not 1, not 2, not even 3 records with the same unique telephone number, but 7 records. This is what they had:

1) One past owner of the phone from over 15 years ago;
2) Three versions of my name;
3) Two versions of my wife’s name; and,
4) One record that was a joint name account.

This is master data at it’s worse. Being curious I started asking questions! Smart right! You ask data warehousing, MDM and DQ questions to a sales clerk and see what responses you get, and then you’ll realize just how smart of a move that is!

What I found out was that all my records had the same address content, phone numbers and postal code information, excluding the older record. I told the sales rep they should delete the other records and keep just one, it would be better for their record and data management. The clerk said they cannot delete the duplicate records. So I gave my elevator speech about data quality and uniqueness of the record and that she should, or her manager, contact their IT department and fix those records.

I’m sure she’ll hop right on that request. Deer in headlights…mean anything to you!

Purchase complete. My extended warranty is somewhere under one of those records. The way I see it. In the future if I do call in for service, they may not be able to find my warranty, or they’ll ask me which record it’s under. In either case it will equal one frustrated customer and a new blog all about it. Next time I’ll make sure to mention the store name.

And to sum it all up I now know why I get 6 calls a year for carpet cleaning service and another 6 for air-duct cleaning.

By the way I never mentioned the other 2 records at my old address, different phone number!!!

Let's face it, this is an excellent example of a company operating with a lack of process, and poor, poor governance.

When Bad Data Becomes Acceptable Data

noreply@blogger.com (Daniel Gent) — Mon, 20 Apr 2009 13:50:00 +0000

Sometimes you need to weigh the options and determine if all your effort to fix bad data is actually worth it.

Yes, bad data costs the company money, added expenses and so much more.

Yes bad data may critically impact decision making at your organization.

Yes, it will take effort to get rid of bad data.

However, when you start your efforts to clean the bad data think of the following decisions you must make when you take the mountainous task to scrub the data clean.

1) Is the ROI worth it?

a. If you have two bad records that are wrong, do you spent the same amount of time working on a $3 error vs. a $30,000 error. I think it’s obvious, but some of us will burn the midnight oil to get ride of the $3 error just as vehemently as the $30,000 error.

b. BUT remember any error that makes it to a customer’s bill will become a big topic for the customer. Your evil over-billing practices may become the twitter topic of the day. And nobody wants that now do they!

2) How long before the error disappears?

a. With archiving and deletion jobs, that record may be gone before you know it. Saving you some valuable time to concentrate on more pressing matters.

3) What’s the impact of keeping it?

a. Yes, what is the impact of keeping that bad, bad data. BUT ask yourself who is using the data and what type of decision is being made with it?

4) Can you communicate the bad data to the users?

a. This may sound strange to you, but if you have to get the data out and you know the exact problem, duplicate records, over-billing, etc., etc. then make the knowledge workers aware of what they are looking at and reporting.

b. Sometimes just letting the knowledge workers in on the problem will prevent a disastrous decision from being made. They can identify the situation in the foot notes, or remove it all together from their reports and inform the decision maker that the issue exists.

On a personal note, I say clean it, scrub it, polish it, make the data shine, like it has never shined before. BUT, we don’t live in an ideal world and you may have to keep the rotten data there. It may be primarily for budgetary reasons or the size of the impact may be very minimal. So when this is the case ask yourself what’s the impact, what’s the ROI, how long will it be there and can you communicate the situation to the users. When this happens the data is still bad data but becomes acceptable data.

Leaving bad data in there may be viewed as the lazy-man's solution, but when you're swamped with 13 other data quality issues to tackle you need some methodology to identify the key situations and those that can be pushed aside.

DQ Problems? Start a Data Quality Recognition Program!

noreply@blogger.com (Daniel Gent) — Tue, 07 Apr 2009 13:05:00 +0000

Having spent some time in an Enterprise Data Warehouse Competency Centre, one item I noticed was that we received a lot of input files, and we’re talking about hundreds of files a day coming from dozens upon dozens of different sources.

Needless to say we had over 80 source systems, legacy systems, web based systems and a few PC files as well. Each of those systems was responsible for sending us anywhere between 1 and 60 files per week. So needless to say there were lots of data related issues that came across my and my colleagues’ desks on a daily basis.

Is your EDW similar in nature? Do you have overworked data quality analysts?
Do you have source systems that just don’t care about the contents of the files they send you?

Remember some of the goals of an Enterprise Data Warehouse is to create one version of the truth, to reduce redundancy, to streamline the decision making process and to reduce redundancy and improve data quality.

Apart from implementing data profiling tools, performing data assessments, monitoring thresholds, correcting code and asking for executive support for data quality in general. What else can you do?

If your organization is large enough and during these trying times, the budget permits it, and the organizational culture accepts it introduce a Data Quality Recognition Program.

What, a data quality recognition program, you said? Yes, I said, a data quality recognition program. The following list contains a few steps you will need to establish such a program:

1. Executive buy-in, (primarily needed for budget approval).
2. Advertise the program to all your source systems.
3. A means to track data quality by source system.
4. Identify qualifying systems (those with an owner).
5. Track the data issues and identify the source systems.
6. Compile the results annually.
7. Hand out the award or awards.
8. Advertise the winner(s) and their results.

You can also establish multiple awards, such as most improved data quality, best data quality, most timely data, most accurate data entry clerk, call centre with the best data quality or even a DQ troll award (for the worse data, if they have a sense of humor) and more. Really the type of awards and the number of awards is entirely up to you and of course the budget.

Make sure your award winners have a great day. Provide the team responsible or owner of the source system with a tangible award, or a plaque. It makes for an interesting conversation piece and it gives them bragging rights, especially on the resume.

Once your program is established and awards handed out, watch the data quality of your EDW improve over time, as everyone does their best to be named in the next award ceremony.

Data Quality: A Cause and Effect Story

noreply@blogger.com (Daniel Gent) — Mon, 30 Mar 2009 14:29:00 +0000

While reading a classic story to my kids, "Because a bug went ka-choo!" I thought to myself that some CEO's who just don't get data quality might learn a little cause and effect along the same lines as this great story. So here's a lesson about the importance of bugs in your code and the data quality issues it may cause and the effects that pour forth...enjoy!

You may not believe it,
But here’s how it happened.
One long business day,
A little bug was scripted.

And because of that script,
A character was dropped.
Because that character dropped
A record got lost.
Because the record was lost,
Provisioning was not informed.
Because they weren’t informed
No service was issued.
Because of no service,
that customer got mad
because he got mad.
He talked to his neighbour,
Because he talked to his neighbour,

the neighbour named Paul chatted to his coworker.

Because of that chat, the coworker made an anti-company website,
Because of that website people shared similar bad service stories.
Because of those stories, the competition smiled.
Because the competition smiled, they decided to entice.
Because they were enticed,
The customers started to leave.
Because they started to leave, sales came down, and it hit CEO Brown.

And that drop got stuck in his head. Because it got stuck, CEO Brown phoned for help.
Because of his call, the CFO began counting.
Because the CFO was counting, they noticed the shares.
And so the execs sold those shares all alone.
Because they sold those shares, the share price came down.

Because it came down, employees were let go.
Because they were let go, they decided to sue.
Because of that suit the shares price started to sink,
Because it started to sink the executives were investigated for insider trading.

Because they were investigated, they headed to court,
And you may not believe it, it’s true I’m afraid they ran right into executive protest parade.

And that started something they’ll never forget, and as far as I know it is going on yet.
And that’s how it happened, believe me it’s true, because, just because a little bug got scripted.