(ANALYSIS) I teach a graduate course in research methods every Fall semester at EIU. It’s always on Wednesday nights and it lasts for two and a half hours. It’s an absolute slog. I tell the students on the first day of class that this may be the most difficult course that they have ever taken. But, they will probably learn more skills in research methods than all their other classes.

We spend a lot of class time talking about research design — which is the framework that social scientists use to help them answer questions in an empirically rigorous way. I dedicate an entire class session to the challenges of measurement.

Students will come to me with a research idea like, “Do more experienced bureaucrats lead to a more effective government?” I then ask them to define, “effective” and they inevitably stammer and stutter, not entirely sure how to proceed.

One exercise that I have them talk through is just a very simple question that is asked on almost every survey: Thinking back over the last year, what was your family’s income? Seems pretty straightforward, right? Well, there are dozens of ways that this single question can induce error into a research design.

A person could make money doing something illegal and they don’t want to report that. Someone may not know how much money their spouse makes and has to guess. A respondent may only report their personal income, not the household’s income. They could just check the wrong box on the web browser when they are taking the survey. There are so many ways that this simple question can get messed up.

Now, imagine if that question was a little less straightforward. Like this one, “Would you describe yourself as a ‘born-again’ or evangelical Christian, or not?” Pew Research first devised that question and it appears on lots of surveys now.

But here’s what is really interesting about that question — no matter how one responded to the prior question about religious affiliation, they are still asked if they are evangelical or not. That means that atheists are asked about their born-again status and so are Muslims.

To read the rest of Ryan Burge’s post, click here.

Ryan Burge is an assistant professor of political science at Eastern Illinois University, a pastor in the American Baptist Church and the co-founder and frequent contributor to Religion in Public, a forum for scholars of religion and politics to make their work accessible to a more general audience. His research focuses on the intersection of religiosity and political behavior, especially in the U.S. Follow him on X at @ryanburge.