This is the UK Labour Force Survey.  When you read that Unemployment has gone up or down the original data largely comes from this survey document.   Most people have never seen it or participated in it, because you have to be drawn at random to be part of the sample.  To keep the sample size small, and the costs down, LFS work on a rolling weighted 3 month average rather than a proper monthly data set.

If you were surprised by recent announcements that the UK has the lowest unemployment since the early 70s then LFS is the source of this data.  If, like me, you felt that the government’s claims on unemployment were a bit far fetched this is the place to start.

The Labour Force Survey was introduced to the UK in the early 70s when we joined the EU, in an era when work was very different to now – men worked, women stayed at home, hours of work were stable, people had one job and self employment/small businesses were very much rarer than now.  This is no longer the case – since the credit crunch 45% of new jobs created have been self employed, since Brexit more than half.

You may have spotted this is an old LFS.  The current one is apparently a secret…..

LFS became the main way the Government uses to measure unemployment in the UK in 1997 when Labour came back into power. Previously we had used a narrower measure, the claimant count, which had fallen into disrepute due to political manipulation in the 1980s and 1990s to massage the unemployment rate.

This is what the Office of National Statistics has to say about the Claimant Count data at the moment:

Looks like DWP are back to their old tricks.

When I was a Civil Servant I was familiar with the ways that data about employment and unemployment could be presented to paint a rosy picture – for example presenting a rise in the number of people leaving the labour market as a fall in unemployment.  If I am honest I took for granted that all Governments present data like this to make a case, that well educated people knew that, and could form their own judgements.   

While I was setting up the Distillery I was picked to be part of the Labour Force Survey sample cohort.  The experience isn’t great, particularly if, like me, you don’t like paperwork and bureaucracy.  I was part of the phone cohort, which meant a series of lengthy phone calls with a researcher, most of which started “thinking back over the last week did you….”. 

The version of LFS I did was 220 pages long, and it me took the best part of an hour to complete. This was repeated 5 or 6 times. 

What it made me realise is how hard it is to capture the distinctions between work, unemployment, self-employment, and under employment using a survey methodology which was created most of my lifetime ago.   

If your hours of work vary week by week you have to wait until page 70 to explain this, and the act of explaining adds time onto the survey.   If you have a second job with variable hours this is somewhere past page 80.

At the time I was setting up my business, and my hours were irregular. Some weeks I was hectic, backwards and forwards to visit Distilleries in Scotland, and viewing commercial properties. Other weeks I would be stuck waiting for a letter from HMRC.    On those weeks I helped build a Lego model of Durham Cathedral, and compiled a fantastic post punk playlist in iTunes.

I quickly tired of answering questions about variable working hours. Frankly after the first couple of times I just said “the same as last week” whether it was true or not.  Apologies to LFS for this, but I was bored, and I’m not great with paperwork.  I once told the Manufacturing output survey that all of my staff were goats for a similar reason.

For people who are working multiple jobs, on zero hours contracts, or self-employed with highly variable hours this is a massive chore, and the survey structure gives an incentive to under report variations in working patterns.

You can add to that the complexities of actually making sure that people in marginal and irregular employment, who may well be in marginal and irregular accommodation, are represented in the survey cohort in the first place. 

I spoke to LFS about this – they confirmed they use the postal address book for the Royal Mail, but were unwilling to enter into further discussion about how this effected their sample size.   Postal Address is one of the more accurate ways of finding people – but as I found out when I worked for CMEC – over the last 20 odd years large numbers of people have dropped off official lists, including the electoral roll, in very large numbers.  The government spends millions every year tracking these people, none of which goes on LFS.

Instinctively the people who aren’t on the post code list are more likely to be working odd hours, and in marginal employment.

I have searched to see if there are academics looking at the problem of gathering labour force statistics.  There isn’t much, but what there is seems to match my conclusions:

The system for recording information about the UK labour force hasn’t kept pace with changes in how we work

As a consequence there are  significant inaccuracies in the basic data about employment and the UK economy

Underemployment is much worse than the headline numbers would lead us to believe, however this is made up of a mass of bad data, over supplies of labour in some areas, under supplies in others, over stretched employees in one place, underemployed people elsewhere.  Lots of people have jobs, but not enough hours to live on.

When the economy is growing strongly the inaccuracy gets less, in fact in a strongly growing economy the statistics might understate employment. In a slow growing economy the statistics over state employment. 

The gap between government and civil service statistics and reality is one of the reasons why people are instinctively reluctant to believe politicians and statisticians (experts), the world they are describing is so far divorced from their real life

I always knew that the way we reported unemployment was spun. Even the Guardian, who are usually sceptical of wild government claims were happy to print this headline this week:

My guess is that we have been overstating employment in the UK for 20 years. At first the inaccuracy was small, when self employment and zero hours contracts were unusual, and the economy was growing strongly.  Since the credit crunch self employment and zero hours contracts have risen and the problem has got larger.    The coalition created a loop hole in the unemployment benefit system which encouraged older wealthier claimants to become self employed – effectively self unemployed. This has made the problem much worse. 

When we leave the EU there will be a chance to rebase the LFS to try and create something which is better able to capture the new categories of work.  Sadly we live in an era where given the choice of good data that is hard to spin, and crap data that is easy to spin we can all guess what we will end up with.

I am not even sure that the traditional categories of employed, unemployed and economically inactive are actually valid, given the number of workers on zero hours contracts who move from full time to part time employment day by day, people in the “gig economy” who might do several jobs, and people who are unwillingly self employed in the industries where they used to be employees.

To prove my hypothesis right would take lots of statisticians, economists and civil servants, and would be politically explosive.  I don’t have those resources, but over the next few blogs I will explore deeper the way government, people and official statistics interact based on my own in and out of central government.

Alternatively if you work on an industrial estate in the North of England you could test the accuracy of the unemployment statistics by just walking around – the slack capacity in the UK economy; people, buildings, machines, is obvious to the naked eye. In fact we have become so used to seeing it we have almost zoned it out – we expect to see empty shops on the high street, factory units for let.

Lots of people were shocked during the EU referendum at the willingness of large numbers of voters to openly scoff at expert opinion, shocked at Michael Gove’s dismissal of their views.   

The fact that the world that experts have been describing is a totally different world to the one most people live in seems to have passed people by.  People don’t believe official statistics or government spokesmen because they are talking about a world which bears no relation to the one they live in.   Like getting GDP figures from Narnia. 

It’s the same with fake news – when the real news and the real facts describe a world that you don’t recognise any more why not put your faith someone with no facts who tells a better story?

I should really have redacted Cathy Staniforth’s name from the emails above, but I was so surprised when writing a blog with a title taken from a Half Man Half Biscuit song to get an email from someone who themselves appears mysteriously  in an HMHB track.  This is exactly the kind of sign which led Dan Brown to the Holy Grail.