The previous article Are Fingerprints Unique? discussed the case of Brandon Mayfield, a Muslim American attorney from the Portland, Oregon area who was wrongly identified as one of the Madrid train bombers in 2004 by the FBI based on an erroneous fingerprint identification.

The Mayfield case is probably the most famous case of an incorrect fingerprint identification. The Mayfield case is an example of a “cold hit” in which a huge biometric database was searched for a possible match to an unknown fingerprint taken from a crime scene. Unlike suspects with plausible links to the crime, there was nothing specific to connect Mayfield to the crime other than the database search match.

There are subtle and serious mathematical and statistical problems with cold hits, which occur with both DNA profiling and fingerprint identification. This article explores in detail the mathematics and statistics of the cold hit problem.

The cold hit problem is closely related to a well-known problem in probability and statistics known as the birthday problem. Imagine a room full of people: Bob, Frank, Mary, Estelle, and others. Each person has a birthday: May 1, December 13, March 11, July 17, and so on.

Not knowing the birthdays of the people in the room, what is the probability that at least two people in the room have the same birthday? How many people need to be in the room for there to be an even (50/50) chance that at least two people in the room have the same birthday?

A naive and incorrect answer would be to reason as follows. There are three-hundred and sixty-five (365) days in the year. The probability that two people have the same birthday is 1/365. Therefore, the probability that at least one pair of people in a room with N people have the same birthday is about N/365. Thus the room needs about 183 people for an even chance of a match. The actual answer is twenty-three (23) people, much smaller than 183!

Let us consider the problem in detail. First, what is the probability that Bob and Frank have the same birthday? There is a 1/365 chance that Bob was born on January 1. There is a 1/365 chance that Frank was born on January 1. Thus, there is a 1/(365*365) chance that both Bob and Frank were born on January 1. There are, however, three hundred and sixty-five days in the year, so the probability that Bob and Frank were born on the same day is 365/(365*365) or 1/365.

We need to find the probability that at least one pair of people in the room (Bob and Frank, Bob and Mary, Bob and Estelle, Frank and Mary, Frank and Estelle, Mary and Estelle, and all other possible distinct pairs) have the same birthday. If there are N people in the room, there will be distinct possible pairs of people. Each pair will have a probability of 1/365 of having the same birthday.

The probability that at least one pair of people have the same birthday is:

P = 1.0 - (Probability that the Pair Does Not Have the Same Birthday)^(Number of Distinct Pairs of People)

which is

P = 1.0 - (Number of Distinct Pairs of People)(Probility that the Pair Does Not Have the Same Birthday)

or 

P = 1.0 - (1.0 - 1/365)^(N(N-1)/2)

It turns out that P is 0.50048, almost exactly even, for N = 23. The number of distinct pairs of people in the room is proportional to the square of the number of people in the room , not the number of people in the room (N). Hence, it takes far fewer people in the room than one would naively expect for there to be an even chance that at least two people in the room have the same birthday.

Probability At Least Two People in Room Have Same Birthday

The plot of the probability of at least two people in a room having the same birthday was generated using the two Octave scripts below: birthday.m and plot_bday.m.

Octave is a free open-source numerical programming environment that is mostly compatible with MATLAB.

birthday.m

function [p] = birthday(n, m, bTrace)
% p = birthday(n [, m, bTrace])
% probability that at least one pair of members of set of N have same birthday (M days in year)
% n  number of people
% m  number of "days" in year (default value = 365)
% bTrace flag to trace operation of function (default value = false)
%
% (C) 2011 John F. McGowan
% E-Mail: jmcgowan11@earthlink.net
% 

if nargin < 2
	m = 365;
	bTrace = false;
end

if nargin < 3
	bTrace = false;
end

p = 0.0;

p_no_pair = 1.0; % probability no pair of people in the sample have the same birthday

% loop over pairs of people in the sample (room full of people)
% brute force
% for i = 1:n
	% for j = i+1:n
	% p_pair = m*(1/m)*(1/m); % probability i and j have same birthday
	% p_no_pair = p_no_pair*(1.0 - p_pair);
	% end
% end

% fast
number_pairs = n * (n-1)/2;
p_pair = m*(1/m)*(1/m);
p_no = 1.0 - p_pair;
if bTrace
	printf("number_pairs: %d  p_pair: %f p_no: %f\n", number_pairs, p_pair, p_no);
	fflush(stdout);
end % if

p_no_pair = p_no_pair*power( p_no, number_pairs);

p = 1.0 - p_no_pair;

end % function

plot_bday.m

% plot probability of at least two people having the same birthday
% in a room full of N people
%
% (C) 2011 John F. McGowan, Ph.D.
% E-Mail: jmcgowan11@earthlink.net
%

p = zeros(1,100);

for i=1:100
	if mod(i, 10) == 0
		printf("processing %d people in the room\n", i);
	end
	p(i) = birthday(i);
end

printf("displaying graph");
fflush(stdout);

figure(1);
plot(p);
title('Probability At Least Two People Have Same Birthday');
ylabel('P');
xlabel('Number of People in Room');

printf("writing plot to file prob_bday.jpg");
fflush(stdout);

print('prob_bday.jpg');

What does the birthday problem have to do with fingerprint identification, DNA profiling, or other forms of biometric identification? Replace the people in the room with fingerprints or other biometric identifiers (DNA profiles, iris images, faces,...) in a database.

Replace the three-hundred and sixty-five distinct birthdays with thousands, millions or more distinct biometric identification codes derived from the fingerprint, DNA profile, iris, or other form of identification. The pairs of people with the same birthday become pairs of people with the same fingerprint or other biometric identifier: the actual criminal who commits a crime and at least one other innocent person.

What happens if a fingerprint database has 100 million people and the chance of two people having the same fingerprint (we are referring to the same partial prints such as a thumb print lifted from a crime scene) is only one in a trillion ().

Astonishingly, the probability of at least two people in the database having the same fingerprint is almost one (1.0). This is because there are (100,000,000)(99,999,999)/2 possible pairs of people in the database — about five quadrillion (1,000 trillion) possible pairs. Even though the probability of any two people having the same fingerprint is extremely low, at least one misidentification occurring somewhere in the system is almost certain (probability 1.0).

The FBI fingerprint database contains about 200 million people, accumulated since the 1920s, and the probability of two people having identical or indistinguishable partial fingerprints (or even all ten fingerprints) is unknown.

DNA profiles are currently claimed to have a probability of two people having the same profile of about one in ten trillion. With cold hits, with a search of a large database of DNA profiles such as are currently being collected, it is actually likely that there will be incorrect matches somewhere in the system.

Brandon Mayfield probably fell victim, in part, to the counter-intuitive statistics of the birthday problem. As the size of biometric databases collected by governments, law enforcement agencies, intelligence agencies, and private companies grows, the cold hit problem will grow — as the square of the number of entries in the databases.

If everyone, all of the nearly seven billion people on Earth, was in the databases, one could produce a list of all possible suspects based on fingerprint or other biometric identification alone. This could easily be hundreds or thousands or even more people.

How does one handle possible suspects who lack an adequate alibi and could have flown to a crime? How many of those possible suspects will have some tenuous seven degrees of separation connection to the crime? Brandon Mayfield was a Muslim American who had represented an alleged Islamic terrorist in a child custody case: a tenuous but possible connection to the terrorists responsible for the Madrid train bombings. This is the crux of the cold hit problem.

© 2011 John F. McGowan

About the Author

John F. McGowan, Ph.D. solves problems using mathematics and mathematical software, including developing video compression and speech recognition technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Possibly related articles:

1. Bad Mathematics: A Trillion Dollar Problem

singingbanana released an interesting video about the odds of flipping 10 heads in a row. It is basic probability and the video is entertaining enough to warrant sharing it with your friends, regardless of their mathematical background.

Possibly related articles:

1. The Cost of Not Understanding Probability Theory

The recent New Yorker article The Truth Wears Off: Is there something wrong with the scientific method? by Jonah Lehrer (December 13, 2010) discusses several cases where a new scientific result was initially confirmed by several seemingly independent scientific studies and then subsequently faded away, sometimes to nothing. To quote briefly from the article:

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis [professor of psychiatry at the University of Illinois at Chicago] has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.

The article cites a number of possible explanations for these cases ranging from confirmation bias to regression to the mean. None seem entirely satisfactory, either separately or together.

As a graduate student, the author attended a lecture by a senior particle physicist who expressed distinct skepticism of the validity of standard statistics in particle physics, referring parenthetically to several cases of reported results at high levels of statistical significance, several standard deviations, that subsequently proved invalid. A recent example, similar to the cases described in the New Yorker article, is the saga of the pentaquark. Not one, but several research groups, reported evidence of the pentaquark, which then seems to have faded away without a full explanation of the multiple observations. See, for example, the online article “The rise and fall of the pentaquark” in Symmetry Magazine. Indeed, from many years of experience, particle physicists tend to view even claims of new effects or new particles at a five standard deviation level cautiously.

Particle physics is often considered a very “hard” science without some of the problems presumably present in “softer” sciences such as medicine, biology, psychology, or parapsychology from which most of the examples in the New Yorker article are drawn. The New Yorker article notes striking parallels between the cases in mainstream science and the work of J.B. Rhine in parapsychology where a so-called “decline effect” has repeatedly been noted.

The New Yorker article is well written and well worth reading. Nonetheless, a few comments seem in order. The article avoids discussing the possibility of fraud. One possible explanation for cases of this type is organized scientific fraud where multiple research groups collude to produce confirmatory results: for example, to ensure the approval and adoption of a new drug developed and promoted by a large pharmaceutical company. Scientific fraud is extremely difficult to prove. In most cases, all a scientist can legitimately say is that he or she was unable to replicate the results of a another researcher. A suggestion of fraud would be unsupported speculation and quite possibly constitute legally actionable defamation or libel. Most proven cases of scientific fraud involve an insider, a colleague in the same laboratory or office, who blows the whistle. In many cases, the whistleblowers have suffered personally and professionally even if they were eventually vindicated.

The notion of replication seems straightforward and is heavily touted in popular science books and textbooks. Replication will probably weed out statistical flukes and gross errors that a competent researcher should have avoided anyway. However, what if the error is more subtle? The independent scientific study may simply replicate the same subtle error. For example, in particle physics, there are a number of extremely complex simulation programs such as the Lund Monte Carlo, the MINUIT fitting package, and the GEANT detector simulation package that are used by many different groups to simulate particle interactions, particle detectors, and analyze results. Computer programs, of course, have bugs. These bugs can be quite arcane and difficult to detect. Consequently, independent research groups may replicate the same spurious results due to a bug in a widely used software package.

Modern scientific research is often technically quite sophisticated. It often takes years of study and practice to master the theoretical or laboratory techniques of a field. Frontier research where ostensibly important new results such as the pentaquark are likely to be encountered often involves sophisticated new techniques. Consequently if a researcher or research group is unable to replicate a reported new result, they must always ask themselves: am I doing something wrong? The researcher who cannot replicate a result may be accused of lack of skill or even incompetence. This is particularly a concern where the new result is reported by a high status researcher or group, or embraced as the hot “new new thing” of the field. Hence, researchers concerned about their career may, like Thomas More in A Man For All Seasons, adopt a policy of prudent silence.

Even so, cases like the pentaquark or the several cases in “The Truth Wears Out” continue to raise questions about the validity of standard statistical methods in the real world. The New Yorker article touches repeatedly on this concern, without reaching any firm conclusions. “The Truth Wears Out” indirectly alludes to the enormous power of modern mathematical methods in concert with powerful computers and software to slice and dice data to produce, consciously or unconsciously, desired results or to construct elaborate models that will fit the data as discussed in the author’s previous posts Frankenstein Functions and Gold Fever. In conclusion, there is both empirical evidence and theoretical reason to entertain doubts about the validity of seemingly solid, well-established statistical methods in the complex world of modern scientific research.

© 2011 John F. McGowan

About the Author

John F. McGowan, Ph.D. is a software developer, research scientist, and consultant. He works primarily in the area of complex algorithms that embody advanced mathematical and logical concepts, including speech recognition and video compression technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Sponsor’s message: Check out Math Better Explained, an elegant and insightful ebook that will help you see math in a new light and experience more of those awesome “aha!” moments when ideas suddenly click.

No related posts.

Misconceptions about probability theory and statistics have major repercussions on society. From seemingly minor things like the excessive sensationalism of some headlines, all the way to the jailing of innocent people based on “statistical evidence”. One of the most common misconceptions is the so called Gambler’s fallacy. Wikipedia defines it as follows:

The gambler’s fallacy, also known as the Monte Carlo fallacy or the fallacy of the maturity of chances, is the belief that if deviations from expected behavior are observed in repeated independent trials of some random process then these deviations are likely to be evened out by opposite deviations in the future.

This definition may seem a bit abstract, so let’s clarify it through a practical example. What’s the probability of flipping a fair coin 10 times in a row and obtaining heads consecutively each time? The answer is:

.

This would be very unlikely. How unlikely? One in 1,024 to be exact. So if we’ve just observed the coin appear as heads 9 times in a row, what are the odds that the same coin will land on heads on the 10th toss?

Many people would argue that the chance of this happening is less than one in a thousand, as we just calculated. However, that answer is blatantly wrong. The probability that the 10th fair coin toss is going to come up as heads is still 0.5, because each trial (toss) is statistically independent from those that preceded it. Tossing 9 heads in a row is very unlikely, however once it has happened, it doesn’t influence the outcome of the 10th toss in any way.

People who fall for this fallacy, do so because of a fundamental misunderstanding of how probability works. They combine the probability of past events (irrelevant for independent trials), with that of future events. With the example above, some people would also erroneously conclude that “tails is long due to come up” and as such would think that it’s more likely to occur.

It’s not a difficult theory to understand, but a lot of people make the mistake of confusing probability with sheer luck. Every instance of an event relies on the same probability regardless, whether you’re rolling dice, waiting for a grade during your online education, or even waiting for buses. If the odds were 5000:1, 4999 events later you’ve still only got a 0.02% chance of the odds going in your favour, the same as the first time the event occurred.

This informal fallacy has contributed to the ruin of many gamblers over the years. A tragic example of what happens when you uphold this way of looking at odds occurs with many who play the game of “Lotto” in Italy, a very popular lottery game played amongst the general population.

The idea behind this game is very simple. Five distinct numbers between 1 and 90 are randomly selected in 10 different Italian cities, three times a week. Gamblers can place several types of bets, but the one we’re interested in, for the sake of this article, is called the “estratto semplice” (simple draw). This type of game requires gamblers to correctly predict that a specific number will be drawn in a particular city.

The probability of placing a winning bet is 1 in 18 (i.e., 5/90), while the payout is 11.232 times the amount that you put down (so if you bet 1 Euro and won, you’d walk away with 11.23 Euros before taxes). The odds are clearly stacked in favor of the house, of course. Incidentally, Lotto is run by the state and as is also known as “a tax on the stupid” for rather obvious reasons.

There are many “systems” and theories used by a large pool of gamblers who want to “beat the system”. More often then not such systems are based on some flawed understanding of how probability really works. A very popular theory is that of the “numeri ritardatari” (“late numbers”, as we will refer to them throughout this article). The basic principle behind late numbers is this: since it’s extremely unlikely that a given number will fail to appear at least once out of 150, 180 or 200 draws in a row, in a given city, you can identify what numbers are “due” to appear and thus bet on them. For example, if a number hasn’t been drawn in the past 140 trials, the number of bets on it will start to grow very quickly.

Of course, despite the fact that a number hasn’t come up in a given city 140 times in a row, its probability of occurring on the next draw is still just 1 in 18. So betting any of the other 89 numbers would yield the same probability of winning.

The application of this fallacy becomes extremely dangerous when coupled with Martingale betting systems, which are often adopted by “late number theorists”. The theory they use is very simple. Since they assume these late numbers are “due” very soon, they think they are going to be able to afford to put down double their previous wager on every bet until the number eventually appears. So when it does happen, the last sum they bet is multiplied 11 times (for the payout) and they will recoup all the money they’ve spent up until then, and end up netting a large additional payout, which is the (last wager x 9.232 + 1) Euros.

Martingale betting systems are guaranteed to work provided that the gambler has an infinite amount of capital and no limits are imposed on the maximum bet that’s allowed to be placed. In the real world, both of these requirements cannot be realistically met. The amount bet grows exponentially, so the Martingale system ends up being a surefire way to bankrupt those who employs it.

In the case of the Italian Lotto, both the fallacy that late numbers are “due” and the choice of betting systems (Martingale) are responsible for the ruin of many. The gambler’s fallacy plays an important role in this case because most people realize that they can’t sustain a Martingale type system for 200 consecutive draws. It’s their faith in the idea that late numbers are very likely to pop up soon, that tempts them into toying with this risky system.

If we assume these people are convinced that a very late number (say, one that hadn’t been drawn in the past 180 lottery draws) will be selected at some point during the next 5 weeks or so (15 trials), and that they’re starting with a bet of one Euro, we can see that the maximum amount they’d need to invest (according to their theory) would be 32,768 Euros, with a max bet of 16,384 Euros by the 15th draw. This is a sizable sum of money, but something that some people would still be able to put down, especially because they knew they payout would be 184,025.088 Euros (before taxes). A tempting prize indeed.

But what are the real odds that the number in question, the one that’s been eluding the gamblers, will not end up occurring at least once in the next 15 draws?

So there is a 42.43% risk that the punter will lose their 32,768 Euros, because they won’t have sufficient funds to double their wager at the next turn (assuming 32,768 Euros was the maximum amount they can afford to bet).

Bear in mind that with an exponential growth of the bet, a huge amount of capital will only afford our late number gamblers a few extra draws, thereby only slightly increasing their probability of making a profit. (With a payout of 11.232 times the wager, they could afford a smaller increase in the amount of money they put down draw by draw, but the overall principle remains the same.)

What has an adoption of this faulty theory led to in Italy? What kind of impact has it really had on those who adhere to it? The honest truth is that it’s gone so far as to contribute directly to things like suicides, people swindling their friends and employers, divorces, people betting their life savings and their homes, families being destroyed, and so on. Do such dire consequences occur to everyone who plays this game? No, of course not, but the fact that it’s happened to some people, and that these flawed theories are still employed today, is indicative of the misunderstanding about probability (and the risks of gambling) that occurs in the general population.

One could – and should – argue that such peoples’ demise is due to their gambling habits and to good old fashioned greed, yet I can’t help but feel that a solid understanding of probability theory would go a tremendous way in helping to cut down on the number of people who fall prey to these types of widespread theories.

An increased awareness of probability and statistics can only improve society and its ability to assess situations and make rational decisions. How do we begin to remedy this situation, not only in Italy, but around the world? We can start by devoting far more time in grade, middle and high school math classes, in order to teach students about this important subject and the implications that it can have on their everyday lives, understanding of society, and ability to make wise financial decisions.

No related posts.