The Census highlights problems with Postal Questionnaires

The nation’s biggest research project is here!! That’s right, the Census! Everyone excited?? No? Come on, the census is great. It’s a piece of history. There was first a census in China over 4000 years ago! We've had one every ten years since 1801! (apart from in 1941 when presumably everyone decided there were more important things going on) , and if it wasn't for a census then the start of the New Testament would be a lot less exciting (boy is born in house, Wise Men find with street map, Herod knocks round but can't get past over eager aunties).

But despite all this lots of people, for a variety of reasons ranging from moralistic to suspicion to inertia, won't bother filling it in this year. And many of those who do will, intentionally or otherwise, make a bit of a mess of it. All these issues are ones which reflect just some of problems with self completion research in general

The first problem with self completion research is that you never get as many responses as you'd like to. Even for the Census which has a £1,000 fine for not completing (which I have looked into bringing in for our research projects, without success as of yet), it was still estimated in 2001 that only 94% of the population filled it in. Inevitable lower response rates than the client envisaged means you normally end up having to employ staff to chase people up, which for the census was a whopping 81,000 people in the UK in 2001, and a marginally lower number for self completion projects we have done in the past. This of course means that the job ends up costing a lot more than it should to finish.

But even if you get a satisfactory return rate you still have the problem that once you hand a questionnaire over to someone to complete (be it on paper or online) you are surrendering control of the document and can't be sure what you get back is what you envisaged.

Firstly questions can be misunderstood and therefore answered incorrectly. On the census questionnaire the ‘number of rooms’ question is certainly up for misinterpretation (and snobbery, I’ll be honest I feel they want me to count my kitchen and living area as one room, but I want future generations to think I had more than 3 rooms in my home!), and will everyone read correctly that they should not include their bathroom? Also the religion question is a minefield. If I am christened but not going to church am I of ‘no religion’? Is this potentially going to count against me at the Pearly Gates? If you are doing a questionnaire with a professional Market Researcher these queries can be explained to you. But with a self completion questionnaire we normally just guess.

A researcher will also ensure all questions are answered properly and not mischievously as can be the case with self completion. Famously 390,000 people in 2001 listed their religion as Jedi Knight. In 1911 one occupant of a household was listed as “Peter Tabby”, nationality “Persian” and occupation “Mouser” (although this respondent may have been getting his own back for being previously asked if he was ‘lunatic, imbecile or feeble minded’....and they say today’s questions are intrusive). Although these are humorous stories they do somewhat reduce the Census’s claims as a reliable historic document.

A researcher will look to keep the respondent interested and engaged and make sure each answer is given reasonable thought. This can’t be done for a self completion questionnaire, although Q17 in the Census “This question is intentionally left blank – Go to 18’ did keep me on my toes. A researcher will also make sure it is the correct person filling it in, and so the correct information is gained. What is to stop someone handing the Census over to their 11 year old child because they’d rather be doing something else? Only trust.

Finally you have to be sure with self completion questionnaires that the database you use to send them is correct and up to date as this will affect your response rate greatly. For the 2001 Census it transpired the Manchester database was two years old (far more up to date that some we have been given) and therefore anyone with a newer property was missed out. In particular it seems to be young males who seem to be missed. On this occasion they have presumably decided it is better to be safe than sorry...and sent a questionnaire to a car parking ticketing machine in Hampshire.

Self completion is still seen as the least intrusive and most anonymous way of being surveyed. And it allows respondents to complete at their own leisure. However because of the problems listed, UK research companies often persuade clients away from using self completion questionnaires as a research method. Which is possibly why the government has employed a US arms manufacturer to collect and process the 2011 Census data...

JG