The Usefulness And Validity Of Surveys And Data

I recently had a “discussion” with a friend of my wife’s who argued, in essence, that data from surveys are invalid unless the data are perfect. By perfect, she appeared to mean that no “statistically valid” survey exists because it cannot represent every potential view or experience of every person in a target population.

Specifically, I was talking about surveys about Gen Ys and the conclusions being drawn, by us and by others, about them – generalizations really, not conclusions, e.g., that Gen Ys are closer to their parents than other generations; more altruistic/less altruistic; more resistant to authority; more interested in finding their own way; and so on. My wife’s friend countered that the survey could not accurately represent some people who probably are not surveyable, e.g., the Gen Y children of illegal migrant workers in Missouri, so no conclusions about Gen Y or even generalizations about them are valid because they are not likely to reflect the views of or apply to this subset of Gen Ys who are likely not going to be reached by a survey.

My understanding of her position is that unless 100% of the people you are targeting with a survey are included, the survey data have no meaning. Okay, I admit to losing it here in that discussion and just withdrawing. There was no argument I could muster against “perfect or nothing.”

I won’t argue that surveys reveal “The Truth” (if that exists), but data from statistically valid and well constructed surveys do reveal a reality that may or may not apply to everyone but that does exist.

I also admit that it was hard for me to argue my position because I am already suspicious of how accurately Gen Y surveys actually reflect the Gen Y population. I tend to rely on anecdote – which has little validity – as a touchstone to what the surveys reveal. I have two Gen Y children so I tend to ask myself whether survey conclusions about Gen Ys work for them. Sometimes yes, sometimes no. My children are more open by far than I was about their lives, but they do not reveal everything, nor do I want them to.

All this is really a lead-in to saying that survey data – about Gen Ys or anything else – at the very least provide something to think about and consider, even if the surveys and the data are always imperfect. To simply declare that we can know nothing unless we talk to everyone is too easy and lets everyone of us off the hook when it comes to digging into the why of everything. That said, I tend to validate survey data by the organization that does the work; like everything else, some are just much better than others at this. We at nGenera will continue to survey because we are curious.