The findings are also inconsistent with other data from YouGov, the polling firm that collected the data for the Bible Society. A decade ago, the British Election Study (BES) commissioned YouGov to create an online panel. This panel, which includes more people than the Bible Society surveys, was asked about religious affiliation and church attendance in 2015, 2022 and 2024.
According to YouGov’s data for the BES internet panel, the share of Christian churchgoers in England and Wales declined from 8.0% to 6.6% between 2015 and 2024, whereas YouGov’s surveys for the Bible Society apparently show an increase from 8% to 12% between 2018 and 2024.
Reporting suggested that the same questions were asked in the two surveys, which is technically true in the sense that the questions driving these results were worded in the same way. But in one of the surveys a number of other questions were asked before getting to the crucial ones on church attendance.
In the 2024 survey, before asking how often the individuals attended church in the last month, an additional series of questions was asked. Participants were asked whether they think ‘It’s important to me to try to make a difference in the world’ and whether they agree with statement like ‘My life feels meaningful right now’. In total eight additional questions were asked of this nature.
There is a very well known psychological phenomenon called priming, whereby you can incluence thoughts and behaviours through the use of an earlier stimulus or prompt. One example is that you can get a different answer to the same question depending on whether that person has been primed to think in a particular way beforehand. In one famous study, people were asked to guess the date the Mongul ruler Genghis Khan died (1227). Half the group were asked to write down the last three digits of their phone number before making their guess and half were not. The group who were primed to think of an unrelated three digit number were more likely to guess a date before the turn of the first millennium - in other words they were primed to guess a three digit number.
So Mouse is left to speculate whether being asked if you feel connected to your community and have a meaningful life immediately before being asked if you attended church recently has a similar priming effect. In some respects, it would be a surprise if it did not.
The 2018 survey fieldwork was done between 11 October and 13 November, while the 2024 survey was just a couple of weeks later from 4 November to 2 December.
Ordinarily a two week difference would not be considered significant, but looking at the dates makes Mouse speculate whether it had an impact in this case. It certainly isn't speculation that in December you are running into a lot more Christmas trees and hearing a lot more of Wizzard wishing it could be Christmas every day than you are in October. Halloween took place during the 2018 fieldwork, but had already passed by before the 2024 fieldwork period and the Christmas build up was in full swing. It is possible this timing difference had an effect.
The 2018 survey had just over 19,000 participants while the 2024 survey had just over 13,000 participants. Presumably, since they asked more questions in the second survey they saved some money by cutting back on the sample size.
That begs the question how the demographics of the two different sized samples were balanced to ensure comparability. No doubt YouGov will have made every effort to strike a demographic balance in these two samples, but that is no easy task. There is no gold standard for ensuring a group of people are equally likely to go to church or not, so there is a possibility the samples had an effect. The sample picked people and weighted the sample to achieve an equal mix of age, gender and ethnic diversity. That would help, but if there were other differences in the demographics, such as a mix of social classes or a bias between urban and rural, these would have a significant impact on the likelihood of church attendance but would be very invisible in the YouGov methodology.
Gold standard social surveys are based on random (probability) samples of the population: everyone has a chance to be included. The British Social Attitudes survey is one such example – and found that churchgoing fell by nearly a quarter from 2018-23.
By contrast, people opt in to YouGov’s survey panel and are rewarded after completing a certain number of surveys. The risk of low-quality or even bogus responses is considerable.
Passing the sniff test
There are some anomalies in the Bible Society data which indicate a problem with the samples which should make us stop and think. Take the number of people reporting that they go to church 'Daily / Almost Daily'. In 2018 the result for 18-24 year olds was 0%, as we would expect, but it jumped to 2% in 2024. The equivalent figure claiming attendance 'a few times per week' also jumped from 1% to 7%. At face value, an incredible increase. That single data point has contributed enormously to the narrative that it is the young who are behind a jump in attendance, almost itself explaining the total jump in monthly attendance reported. But if we are honest it is simply not credible to believe by extrapolation from the 28 people who ticked the 'daily / almost daily' box (that is 2% of the 1,400 sample in that group) and the 98 people who ticked the 'a few times a week' box that there are now almost half a million 18-24 year-olds attending church almost every single day. If we are to believe these numbers, the average demographic in church for daily prayers through the week would have more 18-24 year-olds than any other age group. Is it a coincidence that these answers were the top two in the list of answers? It seems to Mouse a more credible explanation that the people being paid to fill in the survey just ticked a box high up the list without reading it properly or caring much what they were ticking.
On Radio 4's More or Less programme, 'The Undercover Economist' Tim Harford posited several possibilities to explain the anomalous data. The 2018 survey could have been an ‘outlier’ poll with attendance coming out too low. The 2024 poll could have been an ‘outlier’ poll in the other direction. It is possible that a sudden upswing genuinely has happened late in 2024 after the denominational counts had taken place, and other data is yet to catch up with the sudden and dramatic turnaround. The conclusion was that more data is required.
But Mouse wonders what the response to these two surveys would have been if they had shown an unexpectedly large decline in church attendance. Presumably, Church commentators would have analysed the methodology, compared the results with other authoritative sources and concluded that it was a rogue poll. What a shame similar rigour has not been applied in this case.
As for Mouse, sadly he feels that the most likely explanation for the unexpected report of rising church attendance is that there is a bit of bias in the surveys for the reasons outlined above, and a few self-selecting survey responders didn't bother to answer very honestly.
Mouse clings on to some hope that this instinct is wrong and there genuinely has been a sudden and unexpected cultural shift, but none of the explanations he has yet heard have convinced him, and the evidence to the contrary is strong.
If you'd like to read the data yourself, the YouGov data tables are available here and here. You can also read the Bible Society's response to criticisms of their survey FAQ's here.