Dear GrandPa:
After consulting 2 statisticians, of which one is a Ph.d., here is some scientific opinion of the Grand Terrace Community survey.
The Problem(S) with Probolsky Survey Design:
The Grand Terrace Community Survey was an opportunity for the city to gauge the opinions of its residents, using statistics as a basis for assessment. Although some portions of the evaluation were correct, other parts of the study reflected poor statistical design, leaving the entire survey open to scientific scrutiny. Listed below are the major comparisons.
Probolsky Design
· Random sample size of 300 based upon names from registered voters only
Better Design
· Random sample size of 300 based upon names from water or electricity bill
Probolsky Design
· Responses generated only by home phone
Better Design
· Responses generated by phone, mail, and web
Probolsky Design
· Poor statistical analysis, with no statistical tests to show differences between subgroups
Better Design
· Stronger statistical analysis, including weighted percent corrections for biased (skewed) samples
Probolsky Design
· Proper framing of questions about the library; bias shown to those about eminent domain, the power plant, and the senior center; use of leading questions
Better Design
· Proper framing of questions, with no bias to issues; use of “Do you agree or disagree…”
Probolsky Design
· Open-ended questions followed by closed-ended question, with exclusions to respondents that weren’t informed on the issue in few questions
Better Design
· Open-ended questions followed by closed-ended question, with exclusions to respondents that weren’t informed on the issue in all questions
Probolsky Design
· Language of eminent domain question was complex
Better Design
· All questions simple
Probolsky Design
· One survey, that is generalized to reflect the opinions of all residents
Better Design
Pilot study or focus group to eliminate bias and complex questions, followed by a properly executed survey
First, although the sample size of 300 was statistically correct, the population of those surveyed was limited only to registered voters, restricting the number of randomly determined surveyees to only those on the voter roles. A truly random population could have better been derived from names from the water or electricity bill. Recall, that “random” means that each resident has the equal probability of being chosen by chance. Yet, if the sample only included registered voters, then immediately non-registered voters were eliminated from the population, creating a bias.
Second, the population of those surveyed was further restricted to only those who had a home phone, and answered that phone between a narrow span of only 4 days. A better design would have allowed random responses from phone, mail, and the web; as much as 20% of residents are estimated to use only their cell phone, rather than home phone.
Third, the survey lacked strong statistical analysis to truly show differences between subgroups. For example, in the raw data tables, the demographics of age groups 35-54 and income group of 100,000+ represented the majority of those surveyed. These groups skewed the data. In addition, proper statistical corrections, such as weighting or log values were not done. Rather, only percents were totaled and grouped, further weakening the data. A better method of analysis would have used stronger statistical tests for true comparisons of differences.
Fourth, only some of the questions were properly framed with an open-ended structure, followed by a closed-ended formation, allowing only respondents who were knowledgeable about a particular issue to respond. This structure properly created a statistical climate in which only those who used the library, for instance, could respond. In addition, the question about the library was one of the few that were not biased or misleading. On the contrary, questions about eminent domain, the power plant, and the senior center were set up to be leading, or the wording was too complex to be understood. A better design would have used questions that were not biased and that included “Do you agree or disagree…”
Finally, the results of this survey should be taken with a grain of salt. Rather than making the survey out to be a generalization of the opinions of all residents of Grand Terrace, the survey should be regarded (upon further completion of statistical analysis) as a pilot study. A better method would have included a focus group first, followed by the actual survey, for that focus group would have eliminated bias and complex questions. Next time, use the science of statistics to get more value for your $16,000.
