Risk is not always a numbers game

People are too quick to try and quantify things.  If there are important things to count, then count them.  If you can’t count the important things then DON’T:

• Count unimportant things
• Invent things to count

Risk is important, but it can be very difficult to count.  Working in the fuzzy world of environmental consultancy, I often come across (and indeed use) ways to “quantify” the risk of environmental harm as a result of a construction project.

One recent “quantification” worked like this:

• You scored the likelihood of a negative event occurring from 1 (improbable) to 5 (almost certain)
• You scored the severity of said event from 1 (negligible) to 5 (catastrophe)
• To get the overall risk score you multiplied one figure by the other, giving a range from 1 (an improbable event of negligible significance) to, theoretically, 25 (an almost certain catastrophe)

I can understand the idea behind a semi-quantitative assessment of, individually, the likelihood and severity of an event occurring.  But it all falls down at the multiplication stage – the scheme above would give the same score (5) to an almost certain negligible event and an improbable catastrophe!  Really, all catastrophes should be improbable.  The mind boggles at the idea of a 25 – an almost certain catastrophe.

Perhaps it makes more sense when you can give fairly robust figures to the actual probability and cost of an event.  No doubt many risk assessment consultants make a living doing this.  But even if that works, bastardising it to such an extent as to multiply two made up numbers together is surely not acceptable.

Evidence-based blogging – “Heart attack admissions fall after smoking ban”

The BBC reports that:

There were 1,200 fewer hospital admissions for heart attacks in England in the year after July 2007 – when the smoking ban came in, research suggests.

While the 2.4% drop was less dramatic than that reported in some areas where similar bans have been introduced, the figures suggest it saved the NHS £8.4m.

Researchers said even a small reduction had “important public health benefits”.

The Bath team analysed English hospital admissions between 2002 and 2009, the British Medical Journal reports.

Three of the four authors are based at the University of Bath Tobacco Control Research Group, and the full article is published here which, I’m delighted to report, is open access.

In coming to their conclusion the authors created a model, made predictions, controlled for various factors, and used specific statistical techniques.  All of this is of course good and proper, but the way these things get reported it always sounds like prior to the ban there were x number of admissions per year every year and after the ban the number dropped to x minus 1200.  Of course this isn’t the case, and people far more expert than me can debate the pros and cons of the tests, models and assumptions used.

Lets look at the data.  The figure below is taken directly from the paper and shows the number of admissions for myocardial infarctions in England.  The data are obviously quite variable, and there appears to be a downward trend before the ban came into force on 1st July 2007.

These figures, also taken directly from the paper, show the above data broken down by gender and age group.  Again, just from looking at the data it appears that there is a downward trend before the ban.  This does not of course invalidate the authors findings, but I’d be surprised if these graphs get shown in many of the media reports based on the research.  Consider this blog a public service.

Finally, I charted the data in Table 1 in the article, which gives the numbers for each year, so giving a smoother overview of the data shown above.  Again, note the downward trend prior to the ban.

Epidemiological data are usually really complicated, because every individual is different and there are so many other factors to consider beyond the one you are interested in.  As such, although it is not sufficient to just look at the data and say what you see, it is still necessary.  There are many different statistical tools, and many ways of using them, some less appropriate than others.  So never forget to use the tools you have in your face – your eyes!

Risk – and splitting hairs

As society becomes more and more risk averse and “compliance” (i.e. box-ticking) based, what once was probably a sensible attempt at quantifying risk becomes increasingly silly.  Contaminated land – ground contaminated by former land uses and now presenting a risk to “receptors” (usually people or watercourses) – is one such area.

Old-school industry frequently left a polluting legacy – there were few if any rules on how you needed to control the polluting output from your processes.  It is not that uncommon to find significant pools of oil on or beneath the surface of derelict land that have persisted for decades.  It is undeniable that this can present a very serious health risk, therefore we are obliged to deal with the problem, and rightly so.

A large pool of oil (“free product” in the jargon) is clearly a problem which needs to be addressed.  But what about contamination that isn’t immediately visible?  Soil on an old industrial site can look like normal old soil but can actually be chock-full of nasties, you don’t need chunks of asbestos and pools of oil to present a risk.  In these circumstances, we will have needed to analyse the soil and determine the level of contaminants present.  But what do we compare it to?  What level denotes an unacceptable risk?

We can try and work that out.  We can make some assumptions, based on what we know of the chemical in question – how it behaves, how mobile it is, how quickly it breaks down.  But soil is highly variable, and differences in the properties of the soil can have a very large effect on the behaviour of the contamination.  Despite this, we can at least take measurements and determine how the contaminant ought to behave based on past experience and laboratory data.  Highly imperfect, but reasonably good.

The problems start to arrive when you consider how a human being using the site will be affected.  Will the site be a car park, with a small fringe of green around the outside?  If so, even quite high levels of contamination will probably not cause a risk as people tend not to spend much time on the small grass verges next to car parks.  But what if the site will become family homes with gardens?  This time, we need to be worrying about little children running around their gardens all summer, getting mucky and generally being exposed to contamination.  Clearly, the risk is greater.

But we still need to quantify this.  And once we have made the very sensible decision that we need to differentiate between people using a site as a car park and it being a family garden, we create a whole heap of trouble.  How do you QUANTIFY the difference in risk?  By making quantitative assumptions about behaviour.  You need to decide on some sensible assumptions for how often the typical person will be on the site, how long each visit will be, how much soil-derived dust they are likely to be inhaling (say the site is a sports field – the heavy breathing caused by exertion will increase this – but precisely how much?), how much dust they will “trackback” to their homes, and so on.

But making all these assumptions makes the model you have used highly specific.  Take the sports field for example.  A teacher regular taking PE lessons there could be at risk from contamination.  But how much is acceptable?  Part of answering that questions comes from making an assumption about how long they will be doing it for (as in years of their life).  To be on the safe side, most of these types of calculations assume that the adult will work at the site their whole working life.  But how long is that?!  I once had to rerun some calculations because it was decided that a teacher would be taking a one-year PGCE (teaching) course after they went to college, so their lifetime exposure to the playing field of death would be one year shorter.  When you get to that level of detailed assumption it becomes slightly absurd.  Unfortunately, this is the rod we make for our own backs when we take the perfectly sensible decision to distinguish between different types of risk.

So next time you read in the local paper that a site is “contaminated”, it may not be that simple.  Someone has made a very long list of assumptions that may or may not be an accurate reflection of what goes on.  And I haven’t even started on sampling error yet.