— surveys of religious opinion, stupid questions, and hidden agendas

Sometimes you hear of attempts to measure religious opinion by way of methods such as a survey.

Suppose someone prepares a survey. Suppose they prepare a form with a set of statements, and people are asked to respond to each statement by choosing between “true,” “false” or “don’t know”.

Consider for example the statement:

(A) “In the beginning God created the universe.”   …. True / False / Don’t Know

One will get a range of opinion. But the trouble with it is that this is a very ill-defined question. One hardly knows what the replies mean.

Some respondents will think they are being asked,

“Is it true that the universe was created by an invention of human fantasy and delusion?”

Others will think the question is,

“Does physical reality have as its support a beautiful, rich and challenging reality that is gradually becoming more fully expressed in the universe, and which can be met with in moments of utter honesty?”

Some will think the question being asked them is,

“Is it true that the universe was created by the powerful super-being whom certain mobs of middle-Eastern murderers say they are serving?”

Others will think that the question is,

“Do you mind if I quote without explanation a word which large numbers of people invent their own meanings for, and pluck a phrase out of the context where it was written, in order to present it in an inappropriate way?”

Finally, a few will consider that the question being posed is, in fact,

“Is this survey so badly prepared that really it is an exercise in time-wasting or the promotion of bad science and worse religion?”

 

Now let’s try to imagine a clearer question.

A somewhat more carefully posed question might be to ask for opinions on the following statement.

(B) “At the root of things, that which Jesus called ‘God’ creates all that is.”

This statement has at least shown awareness of the need for careful speech in an area that is notoriously replete with muddle and disorganized thinking. I don’t say it is a perfect statement, or that better ones couldn’t be produced, but this certainly is a more useful statement than statement (A), because statement (A) has no clear interpretation. Statement (A) amounts to an exercise in thoughtlessly bandying around a word that cannot meaningfully be treated that way.

Now suppose we do a survey among, say, professional scientists, using a statement like (B), and we get the results back. For the sake of argument, let’s suppose the results are equally divided among “true”, “false” and “don’t know”. One third of respondents say “true”, one third say “false” and one third say “don’t know”.

What have we learned?

We have certainly learned something about opinions among a group of humans. That much is obvious. But have we learned anything else?

The answer to this depends on what are the expected outcomes under various scenarios. For example, suppose we would like to know whether the statement is in fact true. We won’t be able to get certainty, but we might be able to get some evidence.

One could argue as follows. Suppose we ask one thousand people. Let the numbers of respondents giving the replies “true”, “false” and “don’t know” be x, y and z. So we have

x + y + z = 1000

Suppose that someone has some sort of model of human capabilities, and on the basis of this model, they predict that if the statement were true then one would expect to find x = 500, y = 200, z = 300, and if the statement were false then one would expect to find x = 200, y = 500, z = 300. In this case, the situation is quite interesting, because the survey has some chance of giving some evidence that points one way or the other concerning the truth or falsehood of B. It is not guaranteed to happen, however, because some observed outcomes give the same degree of match to both the predictions. For example, if the observed outcome is x = 333, y = 333, z = 334, then it has the same degree of match and mismatch from both predictions. Therefore it does not support either over the other.

Unfortunately, however, the situation is worse than the one I just considered. Our ability to discern human capability here is so small that it amounts to almost none at all. It could easily be the case that, with a good model of human capabilities in hand, one would find that the expected responses in the two cases (B true or B false) were the same. For example, one might find that on B being true, one would expect x = 333, y = 333, z = 334, and on B being false, one would also expect again x = 333, y = 333, z = 334. In this case it will not be possible to discover anything whatsoever about the truth or falsehood of B using such a survey.

The actual situation is more complicated than this, because humans do not act in isolation. Humans are not a set of independent decision-makers and view-formers. They are strongly linked together by all sorts of cultural ties, fashions, assumptions, public discussion, history and so on. The expected outcomes, for a question anything like B, will be dominated by the influence of all these cultural pressures. This is one reason why I feel that the idea of submitting a statement like B to the methods of survey and box-ticking is itself misconceived.

I have written this piece because there have been a few surveys that attempt to gauge opinion on some religious questions amongst the scientific community. The hidden agenda of such surveys is often some sort of hint or assumption that if the scientific community votes in favour of something, then that thing is more likely to be right. Or if the higher-calibre scientists tend towards one view, then that view gains some kudos and extra credibility. However, what if the question being asked was, “should salaries for scientists be reduced?” or, “was Keats a better poet than Yeats?” or, “does scientific expertise correlate strongly with wisdom?” In this case there is no particular reason to trust the opinions of scientists more than other groups, and in fact they may be worse at assessing these issues than some other groups of people.

Another reason why this whole survey idea is misconceived in the case of religious questions is because it is like someone who decides whether to befriend a man called Mark by first taking a survey of how many friends Mark has already. What are they going to do with information? Befriend Mark if he is popular? Befriend him if he is unpopular?

The survey is also like someone who plays loud music in the quiet carriage on a railway train. They are imposing their preferred behaviour in a situation which calls for different behaviour.

They are also like someone who has never read a poem by W.  B. Yeats.

Tread softly, for you tread on my dreams.