Understanding Uncertainty

Welcome to Understanding Uncertainty

david — Thu, 23 Sep 2010 15:50:00 +0000

Welcome to the site that tries to make sense of chance, risk, luck, uncertainty and probability. Mathematics won't tell us what to do, but we think that understanding the numbers can help us deal with our own uncertainty and allow us to look critically at stories in the media.

Winter Wipeout
Microlives
Doubtful adoption league
Puzzling Chance
A Maserati for £1
Three-fold variation in UK bowel cancer death rates
Lottery league tables
Visualising uncertainty about the future
Cochrane visualisation
Spotting a hoax

Do you have a coincidence story? David Spiegelhalter is collecting them over at http://cambridgecoincidences.org.
We're supporting an exciting schools programme led by Nadia Baker of the Millennium Mathematics Project called What are the Odds? - the Hands on Risk and Probability Show. Follow the link for more information and to find out how to get your school involved.
We guest edited an issue of NRICH recently - a collection of school level problems looking at surprising outcomes.

There will be a lot more things appearing - see About Us.

Levels:

level 1

Free tags:

about

Wiped Out

david — Sat, 17 Dec 2011 11:10:43 +0000

Appearing on Winter Wipeout today. Enough said.

Looking deranged at the prospect of the Big Balls

Wrote an article for the Times, which appeared as this.

BBC website headline wrong shock horror

kevin — Thu, 08 Dec 2011 12:53:49 +0000

Bowel cancer screening 'does cut deaths', said the BBC News website today, in a report on a study using data from the NHS Bowel Cancer Screening Programme in England, published in the magnificently named journal Gut. Wow, I thought, that was quick, the programme has been going only since 2006 and didn't cover the whole country till 2010. Have they really found clear evidence of an effect on death rates already?

Well, no, they haven't. The story is a bit more complicated and subtle than the headline indicates.

There are several ways one can try to measure the success (or otherwise) of a screening programme for a life-threatening disease, but many of these can be misleading, and it's generally accepted that the only real way to measure the effectiveness of such a programme is to see what it does to the rate of mortality from the disease.

But it can take a long time to collect enough data to detect and measure changes in mortality rates. (Hence my surprise about the headline.) In deciding to introduce the screening programme, the NHS took account of the results of randomized trials of the screening which followed people up for long periods (in some cases well over 10 years) and did measure mortality. But these were conducted under specific conditions in a limited number of centres, and one cannot be sure that things will work the same way in a national programme.

On the basis of the trial results and various assumptions about how the national programme would work, it had been originally predicted that the NHS screening programme would reduce bowel cancer mortality in England by 16%. (In 2008, about 26 people in every 100,000 in England died of bowel cancer, so around 13,000 people in total - more data here.)

But it's important to understand that this new study did not measure mortality. It measured things like the numbers of people who took part in the screening programme as a percentage of those invited to take part, the results of the screening tests for those people, and the results of further investigations in people who were investigated further as a result of their screening results. (Overall, 2.5% of the men screened and 1.5% of the women had an abnormal screening test result.) These figures led the authors to conclude that the programme is still "on track to cut bowel cancer deaths by its target of 16%." That is, there was nothing to persuade them that the initial assumptions were wrong - but nothing at all to confirm the 16% figure directly.

All this is made perfectly clear in the study report itself. It's arguably a bit less clear in the press release, but that doesn't say that the screening "does cut deaths" and neither, I'd say, does the body of the BBC story.

So why does the headline say that the programme "does cut deaths", and in particular, why does it put that statement in quotation marks, as if someone other than the BBC had said it? The words in question are not in the original study report, not in the press release, and not in the quotations from various experts that are given in the body of the BBC report. So who actually said it? It matters, because it changes the importance of the story quite a lot.

Of course this might be just another example of statistical pedantry. But I think it's important - do you agree?

While I'm on, the story made a bit of a mess of some of the numbers it quoted, too. It says that the people screened were "aged 69 and over" - actually they were aged between 60 and 69 (though the scheme has recently been extended to older people). And it says that "Among the men with abnormal results, 43% turned out to have either cancer or pre-cancerous growths in their colon. The figure for women was 11.6%." No it wasn't. The figure for women was 29% - the 11.6% figure does appear in the press release, referring to something different. Maybe, by the time you read this, the BBC will have noticed these errors and fixed them.

Free tags:

Levels:

Why it’s important to be pedantic about sigmas and commas

david — Sat, 03 Dec 2011 18:59:48 +0000

The BBC reported last week that evidence for the Higgs Boson is “around the two-sigma level of certainty” and provides further explanation:

Particle physics has an accepted definition for a "discovery": a five-sigma level of certainty. The number of standard deviations, or sigmas, is a measure of how unlikely it is that an experimental result is simply down to chance rather than a real effect”

This is nice and clear, but it is also wrong, as we have pointed out before in a previous blog by Kevin McConway.

The number of sigmas does not say 'how unlikely the result is due to chance': it measures 'how unlikely the result is, due to chance'.

The additional comma may seem staggeringly pedantic (and indeed statisticians have been accused of being even more pedantic about language than lawyers). So what is the problem?

The first, incorrect, 'how unlikely the result is due to chance' applies the term ‘unlikely’ to the whole phrase ‘the result is due to chance’, ie it says that the hypothesis that the Higgs Boson does not exist is unlikely, or equivalently it is likely the Higgs Boson exists.

The second, correct, 'how unlikely the result is, due to chance' applies the term 'unlikely' to the data, and just says that the data is surprising, if the Higgs Boson does not exist. It does not imply that it is necessarily likely that the Higgs Boson exists.

Take Paul the Octopus, who correctly predicted 8 football results in a row, which is unlikely (probability 1/256), due to chance. Is it reasonable to say that these results are unlikely to be due to chance (in other words that Paul is psychic)? Of course not, and nobody said this at the time, even after this 2.5 sigma event. So why do they say it about the Higgs Boson?

This is important - people have been wrongly condemned for murder because this comma was left out. The comma needs to be in there.

Are the Brits really fatter than other Europeans?

kevin — Sun, 27 Nov 2011 18:31:20 +0000

Lots of press reports in the last couple of days on how UK women are the fattest in Europe, for example in the Daily Mail and on the BBC News website. I'm still in Berlin, and it was in the papers here too. The tabloid-style Berliner Kurier went with the headline "Man, they are fat, man", while the N24 news service went with "British and Maltese are the fattest Europeans". But is it another dodgy league table?

Well, yes, though for different reasons from those we've looked at here or here. And probably we shouldn't be blaming the reporters, because they made a tolerably good job of reproducing data in the press release from Eurostat, the EU statistics office, that they were reporting on. It does really say that 23.9% of UK women are obese (body mass index (BMI) over 30), and this is indeed the highest percentage out of all 19 countries they looked at. The figure for men is not much lower, 22.1%, and this is indeed the second highest (a bit less than Malta)...

...except there's some quite important small print in the press release. First, it points out that the data aren't for the whole of the UK at all, but just for England. The BBC article mentioned this, correctly pointing out that things aren't any better in terms of obesity in the other countries of the UK either. Then, Eurostat says in the same footnote that for England, "adult" means 16 and over, whereas it's 18 or over in all the other countries. Actually that's not going to make much difference to the percentages either, but it made me wonder why it should be different. The press release says that the data are from the European Health Interview Survey (EHIS), published by Eurostat, and that the EHIS aims to measure various things across the EU "on a harmonised basis". Using different age groupings in different countries doesn't sound very harmonised to me. What's going on?

There's another footnote in the press release, about the EHIS. All that does is provide a link to another article on the Eurostat site - but this one helpfully explains that the English data don't come from the EHIS survey at all, they come from the Health Survey for England. (The Italian data aren't from EHIS either.)

Now the Health Survey for England isn't like many routine health surveys. Participants aren't just asked questions about their health and so on. If they agree, the interviewer actually measures their height and weight, and indeed a nurse comes and takes several other physical measurements of various kinds. It's these measurements that are used to work out the body mass index, and hence to provide data on how many people are overweight or obese.

In contrast, the EHIS survey (and indeed the survey that was used to provided data for Italy) did not take actual measurements, but simply asked people their height and weight.

This matters. The scales used in the English survey don't lie - they might not be utterly accurate in every way, but compared to asking people their weight and height, there are unlikely to be major biases. People do not always know exactly how much they weigh, and even if they do, they might tell the interviewer a rather smaller weight, or perhaps add a centimetre or two to their height. That would bring down the person's body mass index, and if it happens systematically, it will introduce a bias into the obesity figures. Furthermore, it's likely that the amount of bias will be different for men and women, and for different age groups, and possibly for different countries too. And, without having data from surveys using physical measurements in all the other countries, we don't even know how big these biases are likely to be.

So the figures for the other 18 countries aren't really comparable with the English figures. There are biases, but we don't know how big. The Eurostat data say, for instance, that a very alarming 16.6% of young English women (aged 16-24) are obese, far greater than the figures (actually for ages 18-24) for all the other countries listed. (Malta is next with 10.7%, and most of the others are well below 5%.) Is this because English young women are really so much fatter, on average, than women elsewhere in Europe, or are young women more likely than, say, old men to knock a kilo or two off when an interviewer asks their weight?

This isn't really very comforting to us Brits, though. The percentages that are overweight are still scarily high, and the knowledge that, possibly, we're not top of this league after all, doesn't bring our percentages down. It's the other countries' figures that are likely to be subject to bias, not ours. Maybe the Maltese and the Latvians really are heavier than us, on average - but if so, that's because they are heavier than the survey said they were, not because we are any lighter. (And I really shouldn't conclude anything at all from my observation that the people I see on the street here in Berlin do seem less likely to be really large than the people I see on the street back in the UK. That observation, is likely to be even more biased!)

Moral: it's worth reading the small print in press releases, even if they come from a respectable statistical agency. Things might not be as simple as they seem.

Free tags:

Levels:

Microlives

david — Tue, 22 Nov 2011 10:46:30 +0000

Acute risks, such as riding a motorbike or going skydiving, may result in an accident - a natural unit for comparing such risks is the Micromort, which is a 1-in-a-million chance of sudden death, for some defined activity.

However many risks we take don’t kill you straight away: think of all the lifestyle frailties we get warned about, such as smoking, drinking, eating badly, not exercising and so on. The microlife aims to make all these chronic risks comparable by showing how much life we lose on average when we’re exposed to them:

a microlife is 30 minutes off your life expectancy

Life expectancy for a man aged 22 in the UK is currently about 79 years, which is an extra 57 years, or 20,800 days, or 500,000 hours, or 1 million half hours. So, a young man of 22 typically has 1,000,000 half-hours (57 years) ahead of him, the same as a 26 year-old woman. We define a microlife as a chronic risk that shortens life on average by just one of the million half hours that they have left.

Here are some things that would, on average, cost a 30-year-old man 1 microlife:

Smoking 2 cigarettes
Drinking 7 units of alcohol (eg 2 pints of strong beer)
Each day of being 5 Kg overweight

A chest X-ray will set a middle-aged person back around 2 microlives, while a whole body CT-scan would weigh in at around 180 microlives.

Details of the derivations of these quantities are given below.

Of course this is not a precise consequence of each cigarette smoked or drink drunk: it comes from accumulating the effects of a lifetime of behaviour and averaging it over a whole population. And these are very rough figures, based on many assumptions. If you just smoke 2 cigarettes and then stop it is impossible to say what long-term effect it will have on you, for a number of reasons. First, microlives are usually based on comparisons between different people rather than people who have changed behaviour (although below we consider evidence from people who stop smoking). Second, we can never know what would have happened had an individual done something different, the counterfactual.

There is a simple relationship between change in life expectancy and microlives per day. Consider a person aged around 30 with a life expectancy of 50 years, or 18,000 days. Then a daily behaviour or status that leads them to lose a year of life expectancy (17,500 microlives) means they are using up around one microlife every day of their lives.

The speed of living your life

Microlives encourage the metaphor that people go through their lives at different speeds according to their lifestyle. For example, someone who smokes 20 a day is using up around 10 microlives, which could be loosely interpreted as their rushing towards their death at around 29 hours a day instead of 24. This idea of premature ageing has been found to be an effective metaphor in encouraging behaviour change, such as the effect of telling people their 'lung age' - the age of a healthy person who has the same lung function as you (1). 'Heart age' is also becoming a popular concept.

An interesting issue is that life expectancy has been increasing by around 3 months every year for the last 25 years. This could be interpreted as follows: although just by living we are using up 48 microlives a day, our death is also moving away from us at 12 microlives a day. Essentially our health care system and healthier life-styles are giving us a bonus pay-back of 12 microlives per day. However, all this assumes lifetime behaviours and is not necessarily relevant for older people : in fact, as we get older, our life expectancy does not decrease by exactly 24 hours for each day that we survive, and so we don't even use up a full 48 microlives from just living through the day - see the technical appendix below.

So overall we can think of using up around 48 microlives just by living, getting 12 back thanks to healthcare improvements, and then losing or gaining a few due to our lifestyle, genes or environment. Of course the numbers are all very approximate - they should not be extrapolated to extreme behaviours and behaviours may interact.

What is the relation between a micromort and a microlife?

If we expose ourselves to a micromort, we take a 1-in-a-million chance that our future life will be 0, and hence our life expectancy is reduced by a millionth. Hence a young adult taking a micromort’s acute risk is almost exactly exposing themselves to a microlife. An older person taking the same risk, while still reducing their life-expectancy by a millionth, is only perhaps losing 15 minutes life-expectancy. However, acute risks from dangerous activities are not well expressed as changes in life expectancy, and so different units appear appropriate.

There is one big difference between micromorts and microlives. If you survive your motorbike ride, then your micromort slate is wiped clean and you start the next day with an empty account. But if you smoke all day and live on pork pies, then your microlives accumulate. It’s like a lottery where the tickets you buy each day remain valid for ever - and so your chances of winning increase every day. Except that, in this case, you really don’t want to.

What is the official value of a microlife and a micromort?

Governments also put a value on microlives. The UK National Institute for Health and Clinical Excellence (NICE) has guidelines that suggest the National Health Service will pay up to £30,000 if the treatment is expected to prolong life by one healthy year. That’s around 17,500 microlives. This means that NICE prices a microlife at around £1.70. The UK Department of Transport prices a ‘value of a statistical life’ at £1,600,000, which means they are willing to pay £1.60 to avoid a 1-in-a-million chance of death, or a micromort. So two government agencies put similar value on a microlife and a micromort.

Derivation of values

Smoking

A very basic analysis was carried out by Shaw et al 2000 BMJ (2), who estimated a 6.5 years difference in life-expectancy between smokers and non-smokers, which is 3,418,560 minutes. They considered median consumption of 16 cigarettes a day from ages 17 to 71, which comes to 311,688 cigarettes. Making the simplifying assumption that each cigarette contributes equally to the risk, this comes to 11 minutes a cigarette or around 3 cigarettes for a microlife.

Smokers, however, may have lower life-expectancy anyway because of type of person and other life-style factors. So a more sophisticated analysis would look at the effect of stopping smoking. Doll and Peto (2004) (3) analyse data on 40,000 UK doctors, many of whom gave up smoking during the period of the study from 1951 to 2001. They estimate a 40-year-old man who stopped smoking gained 9 years in life-expectancy, or 4,700,000 minutes. If he carried on smoking he only expected to live another 30 years or 11,000 days, during which he smoked 325,000 cigarettes (assuming the higher consumption of 30 a day in 1950s and 1960s). This works out at 15 minutes lost per cigarette smoked, or around 2 cigarettes for a microlife.

Alcohol

White et al (2002) (4) provide spreadsheets for the annual risk of all-cause mortality for different ages and quantities of alcohol drunk per week, in UK units. Consider a 20 year-old man who drinks 28 units a week (which is officially considered ‘hazardous’), rather than 7 units a week. His life expectancy is decreased by 6 months to 55 years, which works out at a loss of 4,400 hours or 8,800 microlives. During this period he will drink 3 units extra a day, which is 60,000 extra units of alcohol. Thence he loses 1 microlife per 7 units of alcohol consumed, which is about 2 pints of strong beer.

Weight

A recent paper (5) estimates a relative annual risk of 1.29 for all-cause mortality per 5 kg/sqm increase in body mass index (BMI) over the optimum of 22.5 to 25 kg/sqm. For a man of average height (1.75m), this corresponds to a relative risk of 1.09 per 5 kg overweight, where the optimal weight is taken as between 69 and 77 Kg. Using the formula derived from relative risks (below), this corresponds to a loss of around 0.95 years life-expectancy for a man aged 35 who is 5Kg overweight. He is expected to live around another 40 years or 15,000 days, This means he is losing around 350 days life-expectancy, 16,500 microlives, for every day he is overweight. So roughly 1 microlife lost for each day being 5Kg overweight.

Radiation

Chest radiogram: courtesy Wikipedia commons

It is difficult to establish the risks from small doses of ionising radiation, as any effects on, say, cancer, have a long delay and so are masked by the substantial lifetime risk of developing cancer anyway. For larger doses the experience of survivors of the Japanese atomic bomb attacks can be used to build models, which then need to be extrapolated down to lower doses. Assuming a smooth extrapolation with no threshold under which ionising radiation poses zero risk, leads to estimates of around an average of 1 year of life lost per Sievert effective exposure when aggregated over a population (6) which, at an individual level, is perhaps best thought of as 1/1000 of a year [9 hours or 18 microlives] per milliSievert (mSv). This assumes an age distribution corresponding to the general population.

For a middle-aged person receiving say an effective dose of 0.1 mSv from a chest X-ray this corresponds to around 2 microlives. A whole-body CT scan delivers around 10 mSv, corresponding to 180 microlives of risk.

CT scanner: courtesy Wikipedia commons

One needs to be careful extrapolating from these figures. A US National Cancer Institute study (7) estimated that the 70,000,000 CT scans carried out in the US in 2007 would eventually cause 29000 cancers. Assuming an average of 10mSv per CT, our very simple model would suggest a loss of 180 microlives per CT scan which would imply 700,000 years of life lost due to the CT scans in 2007. This is more than would be expected from 29,000 cancers and so is an over-estimate: as Berrington et al explain (7), such crude estimates do not take into account the older age of many people receiving CT scans.

Technical appendix: A simple way of obtaining change in life-expectancy from epidemiological studies

John Haybittle (1998) (8) (who was a very fine but modest statistician) describes very nicely how to approximate changes in life-expectancy from standardised mortality ratios (SMR) [the ratio of observed to expected deaths over a specified period] or hazard ratios [the ratio of the daily risk of death to the 'normal' risk] . This is based on the Gompertz approximation to human survival risk, where the annual risk of dying at age $t$ is approximated by a function $h_t = B e^{kt}$ - this says that the log(hazard) is linear, which is very accurate between ages 30 and 80. Assuming this Gompertz model, the life-expectancy $e_t$ at age $t$ is given by
$$ e_t = \frac{\log (1 + \frac{k}{h_t}) - \frac{1}{2}/(1+ \frac{h_t}{k})^2 }{k}.$$
For current survival, $k \approx 0.09$ for both men and women.

An SMR or hazard ratio $\lambda$ throughout the whole life leads to a change in life-expectancy at age $t$. This can be obtained by using the above formula but substituting $\lambda h_t$ for $h_t$.

When $h_t$ is small compared with $k$, which holds for ages below 40, the second term in the life expectancy becomes negligible, and the life expectancy can be approximated by $e_t \approx [\log (\frac{k}{h_t}) - \frac{1}{2}] / k$, and so the change in life expectancy is simply $- \log(\lambda)/k$. Since $k \approx 0.09$, the change in life expectancy from a lifetime excess risk $\lambda$ is roughly $-11 \log(\lambda)$ which for small excess risks is roughly $-11 (\lambda-1)$, so that something that raises your daily mortality risk by 1% ($\lambda=1.01$) will on average take around .11 years or 40 days off your life.

Very roughly, a person in their 30's facing, from some cause, a daily relative risk of death of 1.09 throughout their life is losing around 1 microlife per day. This follows as they expect to lose around 0.95 years in life-expectancy, or 17000 microlives, and yet expect to live another 50 years or so which is around 18000 days.

Technical appendix: Why don't older people use up 48 microlives a day just by living?

Suppose you have lived to $x$ days, and your probability of dying on each successive day is given by $p_{x+1}, p_{x+2}..$ and so on. The your life expectancy (in days) is given by
$$e_x = 1 p_{x+1} + 2 p_{x+2} + 3 p_{x+3} ... ,$$
where $p_{x+1} + p_{x+2} + p_{x+3} .. = 1$. If you then manage to live through the next day and so the event (your death) with probability $p_{x+1}$ did not occur, then your probability of dying on successive days is now changed to
$$ p_{x+2}/( 1- p_{x+1}), p_{x+3}/( 1- p_{x+1}) ..$$
and so on, which will again sum to 1. Your life expectancy now will be
$$e_{x+1} = 1 p_{x+2}/( 1- p_{x+1}) + 2 p_{x+3}/( 1- p_{x+1}) + 3 p_{x+4}/( 1- p_{x+1}) ... = (p_{x+2} + 2 p_{x+3} + 3 p_{x+4}..) /( 1- p_{x+1}) $$
which can be seen to be equal to $(e_x - p_{x+1} - 1)/ ( 1- p_{x+1}) $. This means that the reduction in life expectancy from living through day $x+1$ is
$$ e_x - e_{x+1} = 1 - \frac{p_{x+1} e_x}{1-p_{x+1}} \approx 1 - p_{x+1}e_x,$$
assuming $p_{x+1}$ is fairly small.

If $p_{x+1}e_x$ is very low, then $ e_x - e_{x+1} \approx 1$ and so surviving a day means that your life expectancy is reduced by 24 hours or 48 microlives. But suppose you are old, with a life expectancy of only another 5 years (1800 days) and a 1 in 5000 chance of dying tomorrow. Then if you survive tomorrow, your life expectancy is only reduced by
1- 1800/5000 = 0.64, and so you have only lost 0.64 x 48 = 31 microlives. In the extreme case that you have a constant daily risk of dying, then $e_x = 1/ p_{x+1}$ and so living a day does not decrease your life expectancy - each day is started afresh. This is the case with a 'memory-less' exponential survival distribution, which does not hold at any time in normal human life - your daily risk is always expected to be increasing.

References

1. Parkes G, Greenhalgh T, Griffin M, Dent R. Effect on smoking quit rate of telling patients their lung age: the Step2quit randomised controlled trial. BMJ. 2008 Mar 15;336(7644):598–600.
2. Shaw M, Mitchell R, Dorling D. Time for a smoke? One cigarette reduces your life by 11 minutes. BMJ. 2000 Jan 1;320(7226):53.
3. Doll R. Mortality in relation to smoking: 50 years’ observations on male British doctors. BMJ. 2004 Jun 26;328:1519–0.
4. White IR, Altmann DR, Nanchahal K. Alcohol consumption and mortality: modelling risks for men and women at different ages. BMJ. 2002 Jul 27;325(7357):191.
5. Prospective Studies Collaboration. Body-mass index and cause-specific mortality in 900 000 adults: collaborative analyses of 57 prospective studies. The Lancet. 2009 Mar;373:1083–96.
6. Little MP, Hoel DG, Molitor J, Boice JD, Wakeford R, Muirhead CR. New models for evaluation of radiation-induced lifetime cancer risk and its uncertainty employed in the UNSCEAR 2006 report. Radiat. Res. 2008 Jun;169(6):660–76.
7. Berrington de Gonzalez A, Mahesh M, Kim K-P, Bhargavan M, Lewis R, Mettler F, et al. Projected Cancer Risks From Computed Tomographic Scans Performed in the United States in 2007. Arch Intern Med. 2009 Dec 14;169(22):2071–7.
8. Haybittle JL. The use of the Gompertz function to relate changes in life expectancy to the standardized mortality ratio. International Journal of Epidemiology. 1998 Oct 1;27(5):885–9.

Free tags:

chronic risks

The next Piccadilly line train is leaving from ....

david — Sat, 19 Nov 2011 16:24:08 +0000

Kings Cross Station now not only has a platform 9$\frac{3}{4}$, but also a platform 0. And for the numerically challenged, there are repeated announcements that 'customers are advised that Platform 0 is situated next to Platform 1'

I suppose the Underground platforms will now have to be given complex numbers.

[That is an unofficial entry from the Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, for next year's Round Britain Mathematical Joke competition.]

Levels:

level 1

The risk of queuing?

david — Tue, 08 Nov 2011 15:29:47 +0000

Got a short article in the Times today about the UK Border Agency relaxing its checks over the summer.

I wish I had included the following interesting information provided by the excellent Home Office Immigration Statistics April-June 2011. In 2010 there were around 100,000,000 admissions to the UK , and around 19,000 non-asylum individuals were refused entry. That's around 1 in 5000 admissions, about 35 plane-loads.

So someone should be able to estimate how many people were admitted who would otherwise have been refused entry - it may not be very many.

Another doubtful league table?

david — Tue, 01 Nov 2011 12:13:58 +0000

David Cameron has prominently commented on the recent performance tables concerning adoption in local authorities, in particular the proportion of children whose adoption placement occurs within 12 months. But are the local authorities really as different as they have been made out to be?

The league tables are available here but only give percentages. Considerable searching is necessary to find the raw numbers on which these percentages are based, but it is possible to eventually discover details here under ‘local authority indicators at LA level’ Table I2. This spreadsheet only gives numbers rounded to the nearest 5, although the percentages given in the league table use the unrounded numbers. By adding the rounded denominators (and substituting ‘5’ for denominators which are not given at all), a spreadsheet can be constructed for data for 2008-2010 in 143 authorities – this is available from Googledocs.

Just as in an analysis of bowel cancer mortality rates , which was also covered in a recent article by Ben Goldacre, we can use a funnel plot to display the data. The one below has been constructed simply by cutting and pasting the final 3 columns of the spreadsheet into ERPHO’s excellent funnel plot generator.

Most of the local authorities (LAs) lie within the 95% (2SD) funnel and so their variation is essentially indistinguishable from chance. Many of the more extreme authorities handle low numbers (eg York was top of the table but only handles around 10 adoption a year): many of those at the top of the league table are similarly small and have performance that is not significantly different from average.

While York and 5 other Las can be considered as having systematically better performance, only a few LAs can be confidently said to be worse than average.

Although there is clearly room for improvement, it seems inappropriate to construct a league table based on such low numbers. In addition, highlighting this one performance measure obscures the overall picture obtained from the multiple indicators.

A probability paradox?

david — Mon, 31 Oct 2011 08:49:07 +0000

I recently tweeted a link to this problem drawn on a blackboard, which got a lot of retweets.

Multiple Choice: If you choose an answer to this question at random, what is the chance you will be correct? A) 25% B) 50% C) 60% D) 25%

This is a fun question whose paradoxical, self-referential nature quickly reveals itself – A) seems to be fine until one realizes the D) option is also 25%.

A quick search reveals hundreds of discussion contributions of this problem, for example here and here and from a year ago. People often appear very confident that their answer is the only possible solution.

I am no logician and so unqualified to place this within the grand structures of mathematical paradoxes. I have not waded through all the discussions and so there may be something I have missed, but in among all the arguments there seem to be four conclusions that could be considered as 'correct'. These are my personal comments:

1) There can be no solution, since the ambiguity of ‘correct’ makes the question ill-posed.

It's true the question is ambiguous, but this still seems a bit of a cop-out.

2) There is no solution.

This seems to take this interpretation of the question.

Which answer (or set of answers) of “p%”, is such that the statement ‘the probability of picking such an answer is p%’ is true?

Then this appears to be a well-posed question, but there is no solution.

3) 0%.

Consider a different interpretation of the question.

Is there a p%, such that the statement ‘the probability of picking an answer “p%” is p%’ is true?

Then this appears a well-posed question and has the solution p = 0, even though this is not one of the answers. Of course if answer C) were changed to “0%” (as it is in this 2007 version of the question ), then this would also have no solution.

4) We can produce any answer we want by changing the probability distribution for the choice.

Why should ‘random’ mean an equally likely chance of picking the 4 answers? If we, say, assume the probabilities of choosing (A) (B) (C) (D) to be (10%, 20%, 60%, 10%) then the answer to either formulation (2) and (3) is now “60%”. But if we make the distribution (12.5%, 15%, 60%, 12.5%) then we seem to back to square one again, since there is now both a 25% chance of picking “25%”, and a 60% chance of picking “60%”.

I like conclusion 3) best, ie 0%.

Maybe the main lesson is: ambiguity and paradox are often the basis for a good joke.

Levels:

level 1