Why I Can’t Tell You How Quakers (Or Unitarians) Voted In 2024
(ANALYSIS) I’ve got to admit that I am writing this post while gritting my teeth just a bit. Which is probably not a good state of mind, but I just have to get all these thoughts down in a single post. Let me tell you what is driving my consternation.
I will post a graph such as this:
It always gets a nice bit of traffic and lots of shares and retweets and all that stuff. But there is something inevitable that happens in the comments. People are very upset with me that I did not include their own specific denomination.
People want to know where the Cooperative Baptists are, or the Nazarenes, or the Wesleyans, or the Unitarian Universalists. I’ve done this enough to know that everyone wants to see their own tradition included in analysis. We all want to feel like “we matter.”
However, those kinds of comments just make me so incredibly frustrated with the state of data literacy in the United States. The average American has absolutely no idea how the sausage is made when it comes to the type of surveys that I use to produce all the charts, graphs and maps you see on this newsletter and in various other channels.
Let me pull back the curtain just a bit on why my favorite response has become, “I’m data limited.”
I started graduate school way back in August of 2005. The fact that this was almost 20 years ago caused me to have an existential crisis the other day, but time doesn’t stop.
When I began taking courses in American politics at Southern Illinois University, I quickly realized that the field was dominated by statistical analysis. Every paper seemed to have a nested bivariate probit regression.
So, of course, I wanted to use all those fancy techniques in my own analysis of religion and politics in the United States. But what I ran into was a pretty severe (and hard to overcome) problem. There just wasn’t that much data to be had out there.
The backbone of longitudinal statistical analysis of American politics is the American National Election Study (ANES). It’s been fielded with consistency since 1948. (By the way, there’s almost no publicly available survey data before that date). You can see the total sample size of each year that the survey was collected since its founding over 75 years ago.
Full disclosure — I hate the ANES. It’s changed the way it classifies the religion variables at least three times since the 1960s. It also has an incredibly stupid way to name its variables. For instance, if an analyst wanted to do some longitudinal analysis in most surveys they would use a command like: group_by(year).
You want to guess what it is in the ANES? It’s group_by(VCF0004), because that makes perfectly logical sense. It’s just a nightmare to wrangle, and I only touch it if I have literally no other options when it comes to answering a question.
But the other problem is that there just isn’t that big of sample to use for any real purpose. In the 1960s, the average number of respondents was about 1,300. If you assume that about 5% of Americans were nonreligious during that time period (which is impossible to know for sure), then you’ve got 65 folks you can analyze.
I don’t think you need to take a stats course to understand why that’s incredibly problematic. Now, the sample size has gotten bigger in the last 20 years or so, which is good. But even with those larger surveys, an average wave of the ANES is less than 2000 respondents.
To read the rest of Ryan Burge’s column, please visit his Substack page.
Ryan Burge is an assistant professor of political science at Eastern Illinois University, a pastor in the American Baptist Church and the co-founder and frequent contributor to Religion in Public, a forum for scholars of religion and politics to make their work accessible to a more general audience. His research focuses on the intersection of religiosity and political behavior, especially in the U.S. Follow him on X at @ryanburge.