Learned in First Semester, Quickly Forgotten

Via Pyjamas in Bananas, an article about suicides in Wales, which cites a government report:
Bridgend has higher overall rates of suicide among males; however the rates achieve statistical significance in Neath Port Talbot, Denbighshire and Carmarthenshire.
Three LHBs have rates of suicide among males aged 15-24 that exceed the Welsh average to a level considered statistically significant.
You see this kind of thing all of the time. Look, people, a test of significance answers the question how likely it is that a difference you find in a sample drawn from a universe is actually present in the universe. This will usually be a difference between two groups or a difference from zero. If you have numbers on all of the universe, tests of significance make no sense whatsoever despite your statistics program happily churning out the numbers. It's simple, really.


pj said...

I seem to remember having a very long argument with someone about this - probably on badscience - where I defended statistical testing of this nature (I think it was to do with comparing different years) by framing it in terms of each population (e.g. health board) being randomly sampled (instantiated) from a hypothetical archetypal population described by a set of descriptive statistics with the statistical tests being interpreted as asking whether the latest population/instantiation can still be credibly described as being 'drawn' from that archetypal population i.e. asking if the population structure is really changing over time.

But people got pissed of with me, so perhaps I shouldn't pursue it ;-)

LemmusLemmus said...

Hmm...but your scenario doesn't seem to apply here - that is, if I'm reading you correctly, which I'm not sure about: That is one complex sentence you've delivered.

pj said...

I guess the question you'd be asking here is whether the Bridgend population could be considered (statistically) a random sampling from the overall Welsh population.