The census is one of the most important sources of data and statistics, informing decisions about almost every aspect of life within the UK. As well as enabling government, local authorities, academics, and commercial businesses to make decisions, the census tells us about the communities we live in, where we are from and how we live our lives.
It is essential that users can have confidence in census statistics so that they can have confidence in the decisions that they inform. A census is unique in its objective of collecting a single response from every member of the population, in the right place. This brings some unique challenges to help users understand, interpret and use the statistics appropriately – not least because every census will ultimately fail in its objective of achieving one hundred percent response.
Our challenge
For Scotland’s Census 2022, at the National Records of Scotland (NRS) we faced some unexpected challenges given that the overall response rate was lower than had been anticipated (89.8% compared to target of 94%). There was concern in the media and amongst users that statistics from this £140 million programme would not be fit for purpose because of the response rate.
Since 2001, census statistics in the UK have been estimates of the total population rather than counts of just those who completed a questionnaire. A Census Coverage Survey (CCS) is conducted immediately after the census itself and responses to the CCS are used in combination with census responses to estimate and adjust for under-coverage.
We undertook a number of unprecedented steps in collaboration with international census experts to change how we calculated final census estimates. This involved groundbreaking use of administrative data alongside the CCS and census responses in our estimation process. The outcome was to reduce the level of inherent uncertainty in the statistics.
Our approach
When the first outputs from Scotland’s Census 2022 were published in September 2023, we gave careful thought to how to communicate uncertainty. Published OSR guidance was helpful to shape our thinking as we wanted to do more than simply provide generic statements about ‘treating estimates with caution’ and links to confidence intervals. There were two key themes we reflected on from the OSR guidance.
Firstly, we recognised the importance of intelligent transparency. We wanted users to understand that the statistics were adjusted for under-coverage so represented the whole population, that this had been the approach taken since 2001, and that additional steps had been taken given Scotland’s response rate. Transparency was important but we didn’t want to undermine the statistics or lose readers who were interested in what the estimates said.
Secondly, we wanted to provide practical guidance about what the uncertainty meant. OSR case studies demonstrated that quality information could be presented in a way that helped users think about what uncertainty meant for their interpretation of the statistics. We reflected on how earlier census publications had presented quality information as either generic statements or with reference to tables of uncertainty measures.
Our approach is illustrated in the first estimates of the population published from Scotland’s Census 2022. We considered our use of language and communication style to explain uncertainty in a way that supported a broad range of users, with varying degrees of expertise, to use the statistics appropriately. For example:
- The title of the output itself refers to ‘rounded population estimates’.
- Information about uncertainty is provided upfront, in the second section of the publication titled ‘Working with census statistics’. The title of this section has been chosen carefully – it’s not simply stating that there are caveats for users to be aware of, it is clearly indicating that this information will help users interpret and use census statistics.
- Within the ‘Working with census statistics’ section, we state that statistical modelling has been used to produce total population estimates across the UK for previous censuses, that there is a level of uncertainty with all estimates that users should consider, and that changes in estimates smaller than +/-1% should be interpreted as minimal change. We considered user understanding of uncertainty and developed the guidance on +/-1% based on conversations with users to ensure that information was provided in a straightforward, easy to understand way.
- There is a further, more-detailed section about uncertainty towards the end of the publication titled ‘More information about data quality’. This section includes information about the CCS and the use of administrative data, plus confidence interval targets as well as what was achieved. As well as actual confidence intervals, there is also an explanation of what these mean in order to help users interpret them e.g. ‘we can be very confident that the true population is between 5,408,700 and 5,464,500’.
- Throughout the publication, there are also proportionate references to terms such as ‘estimated’ and ‘around’ as a simple way to indicate that statistical estimates are not exact.
Our conclusions
We had positive reaction from users about the clear messaging of uncertainty with a greater appreciation that the statistics were estimates. By taking a staged approach to uncertainty we were able to guide users through quality information in a proportionate way alongside the key messaging in the statistics.
In summary, we took a careful and thoughtful approach to communicating uncertainty, that culminated in practical information to help users understand, interpret and use census estimates, ultimately to aid more informed decision-making.
Ed Humpherson’s blog about the constant challenge of communicating uncertainty clearly distinguishes between specific uncertainty (the technical information, such as confidence intervals) and contextual uncertainty (information about inferences that users may draw). Our approach demonstrates how the two can be balanced and come together to provide helpful, valuable information for users to support the use of census statistics in practice.