The biggest problem wasn't the self-selecting. It's that it was self-selecting presumably from a crowd that was already overwhelmingly dominated by 4e fans. That he notes self-selecting without noting that the pool itself was tainted, is telling. If you do phone surveys then you are only surveying people that use phones. If you do a survey on a site that is support for 4e games, then you are largely only going to get input from 4e players.Shazbot79 wrote:erik wrote: Have they not heard of sampling bias? The people visiting the site right now are the people playing/preferring 4e almost overwhelmingly. Of course that sample set is going to use the most positive value statement for their preferred selection. If 4e wasn't the most "just right" for any question, then there's probably something fishy going on.Mearls wrote: I thought it would be useful to pause this week and talk a little bit about the polls that have been appearing in this column. Some folks think of them as poorly disguised marketing research. In all honesty, they’re simply an attempt to engage in a dialogue. We already have an entire department here at Wizards of the Coast dedicated to collecting data, running official surveys, and so on. Plus, I also took enough statistics in college to understand that a self-selecting audience is by no means a sound foundation for the sort of polling we’ve been running in Legends & Lore.
There are useful bits of information still to be gleaned from these polls as others have wisely noted, but mostly only as indirect things like showing how many players presumably started 4e without any DnD background, how many players still visit their forums despite not playing 4e. It is not a totally useless survey, but I still doubt their competence to make accurate use of such information.
If they had more questions at once then they could have used these questions as a kind of filter to tell them about their audience so they could figure out what future responses meant. They can still loosely of do that, but it is a lot less solid than if they could tie these answers directly to other answers from the same respondents.