To establish a dialogue with the respondent, interviewers were expected to gauge the types of probes that would elicit the fullest responses. Interviewers were also instructed to use their "natural curiosity" when trying to elicit answers from the respondents. Even after the survey was in the field, the training process continued in the use of these probing techniques. Opportunities were created before and after each interviewing shift for interviewers to share their experiences and they received continual feedback from the survey managers and the investigators. Post-shift meetings were held to discuss effective probes and the problems interviewers were facing. After monitoring interviewers, supervisors held one-on-one discussions to discuss the pros and cons of probes used and other probes that could be used in similar situations. The responses were regularly examined by the project manager to distinguish the types of probes were worked best. The interviewing process was far more collaborative between the interviewing staff, the study managers, and the investigators than a standardized survey. It was also much more labor intensive.

How did these techniques work? The most appropriate measure of success is the value of the data to the researchers. The historians who developed the idea for the project expressed a great deal of satisfaction with the data and the richness of the responses to the survey questions. Historians and other humanities researchers who are aware of the project have expressed an eagerness to hear about the results. Thus, there has been a strong positive reaction to the quality of research data resulting from the use of these nonstandardized techniques.

As part of the CSR’s internal evaluation of the techniques, we sought feedback from the interviewers on how they felt the survey worked. Their responses were overwhelmingly positive. Thirty-five interviewers were asked to self-administer a brief survey on their experience with the History Study. Over one-half (54.3%) said the History Survey was much more rewarding and another 34.3% said somewhat more rewarding than other surveys they worked on. They felt this way despite the fact that 72.2% of the interviewers found the survey much more or somewhat more difficult than other surveys. Many interviewers commented on how much they learned while doing the survey.

The positive reactions of the interviewers was based on more than their own level of enjoyment and what they themselves learned. Many of the interviewers commented on how they thought the data they were gathering was improved substantially by the nonstandardized techniques employed in the study.

Both the interviewers and the investigators of the study believed the techniques were a successful in terms of gathering quality data. While we would not argue that survey research is ideal for all types of research problems, we do feel that the advantages of the method extend beyond instances in which standardization is the most useful interviewing technique. Based on experience, we believe that well-trained interviewers are capable of carrying out surveys that require the use of nonstandardized techniques in a way that results in the collection of high-quality data. Such nonstandardized techniques permit the use of survey research beyond the method’s reach when only standardized interviewing techniques are used.

A second goal of the survey was to examine the differences in the perceptions of the past among various racial and ethnic groups. Funding limitations mandated a cost-effective method for sampling minority groups. This section of the paper describes the targeting of African-American and Mexican-American respondents. For both groups, telephone numbers were randomly generated using the Genesys sampling system. The demographic compositions of telephone exchanges was used to target the samples. We completed 224 interviews with African-American respondents and 196 with Mexican-American respondents. Spanish-language interviews were conducted with 55 Mexican-American respondents.

For the African-American sample, the exchanges were divided into three strata: exchanges where the African-American population was greater than 75%; exchanges where the African-American population was between 50% and 75%; and exchanges in which the African-American population was between 1% and 49%. Telephone numbers were selected from each strata, but most telephone numbers came from those exchanges with the largest proportions of African Americans — exchanges with a concentration of greater than 50% African Americans. Only a small sample (400 telephone numbers) was selected from the exchanges with concentrations less than 50% because calling these exchanges was not cost-effective (only one completed interview was obtained from the strata).

Table 1 compares the African-American respondents in the History Study with data on the African-American population of the United States. A breakdown of the region of residence of the survey respondents indicates that the sampling methods resulted in a diverse group of respondents that came fairly close to approximating the national population of African Americans. The largest difference between the sample and the population was in the region of residence. The data in Table 1 indicate that midwest African Americans are overrepresented in the sample. This outcome appears to result from the high number of high African-American concentration exchanges in that region. That is, African Americans in the midwest are more likely to be living in exchanges with high proportions of African Americans.

The sampling method produced a group of African-American respondents that was fairly representative in terms of the age of respondents. The youngest and oldest groups were somewhat underrepresented and there were a larger percentage of 45-54 year old respondents than there are 45-54 year olds in the population, but the percentages are very similar to their percentages in the population. While females comprise 54.5% of the African-American population over age 18, they accounted for 62.8% of the African-American respondents in the History Study. These data are relatively similar to other studies that find that women are more likely to be included and older and younger persons are less likely to be included in telephone surveys. These outcomes appear to be more likely in surveys of minority populations (Throneberry and Massey, 1988).

An examination of marital status distribution in our sample indicates that the proportions are quite similar to those in the national population. The survey respondents were a little less likely to report being married and a little more likely to indicate they were divorced than the African-American population that responded to the Census.

A smaller percentage of the survey respondents had less than a high school education and a higher percentage had a bachelor’s degree or higher than is found among the African-American population as a whole. This may be the result of differential cooperation rates. A common reason for refusing to participate in the interview was the respondent’s claim that s/he did not know much about history. This occurred despite attempts to convince respondents that the study was not just about the history of the nation or the world but also but people’s family and group histories. Another possible reason for the higher levels of education among respondents is telephone access. Thornberry and Massey (1988) found that, 15.6% of African-Americans were without a phone and that the likelihood of having phone increased with higher levels of educational attainment.

According to the 1990 census Mexican Americans account for 61.2% of the Hispanic Origin population in the United States. However, the population of Mexican Americans is largely concentrated in a relatively small number of states. This concentration determined our strategy of focusing on a small number of states to sample Mexican-American respondents. Table 2 displays a comparison of our sample with decennial census statistics.

The sampling method we used resulted in a large proportion of interviews from Mexican Americans in Texas. There were more high-concentration Mexican-Americans exchanges in Texas than in the other states and there was a somewhat better response rate among the Mexican-American respondents in Texas than in California. We attribute the lower response rate in California to two factors: (1) we had more difficulty reaching respondents in California, and (2) we were interviewing during the Prop 187 campaign, so there was likely more mistrust of surveys among Mexican Americans. Despite being one of the targeted states, calls to Colorado produced no completed interviews. A comparison of the respondents’ demographics to statistics on Mexican Americans in the US population gives some indication of the impact of this disproportionate sampling of Mexican Americans in Texas.

The respondents resemble fairly well the age distribution of the US Mexican-American population. We interviewed a disproportionate number of females. In our sample, 61.3% of the respondents were female. Census data shows that females comprise 47.7% of the population over age 18 who identified as Mexican American. For place of birth, the census found that 54.4% of those who identified as Mexican American were born in the United States and 45.6% were born elsewhere. Our respondents were more likely to report that they were born in the US (64.2%) than the general population. Some interviewers, especially those who conducted the Spanish interviews, believed that a number of respondents became very uneasy when asked the questions about residency and the birth place of themselves and their parents, so there are some doubts about the validity of the question.

The available figures for marital status for Mexican Americans included the population 15 years and over. Since we interviewed only those over 18, a more appropriate comparison is with the marital status of all Hispanic Americans over age 18. Table 2 shows the marital status distribution of respondents in the sample were similar to the national population of Hispanic Americans. Some of the differences might be attributed to differences in the available response categories. Similar to the African-American sample (and in telephone surveys, in general), there is an underrepresentation of those with less than a high school education and an overrepresentation in the categories with higher levels of educational attainment.

The tables indicate that the procedures for drawing the minority group sampling produced fairly representative samples. There were some demographic features where the respondents differed somewhat from the national populations of these groups but, in general, the proportions were relatively close given the small sample size . We might have reduced the portion of the Mexican-American sample from Texas since the percentage of respondents from that state was so much greater than the percent of the Mexican-American population that lives in Texas but the costs of interviewing in California would have reduced the overall sample size. We might also have reduced the proportion of African Americans in the midwest, but since the other demographic characteristics track the census distributions fairly well, we would have gained little for relatively greater interviewing costs.

Yet despite these limitations we were able to interview a diverse group of respondents. Given the cost restraints that were present this is a significant accomplishment. This strategy enabled us to provide previously ungathered data on Americans’ perceptions of the past. It enabled us to reach a national sample and ask questions that were not typical of telephone interviews. These data could not be obtained by more traditional ethnographic interviewing without tremendous cost. While a broader sampling strategy is preferable, resource limitations often require compromises in these areas. The History Study gives us some sense of what we are able to accomplish given such resource limitations.

A major concern in survey research is how best to achieve standardization. The goal is to insure that "procedures are consistent each time an observation is made and data are collected" (Fowler 1991, p. 260). Interviewers are trained to ask questions exactly as they appear in questionnaires and, when appropriate, to probe using standardized statements. While training interviewers in standardized techniques may reduce interviewer-related error, it has some limitations. Suchman and Jordan (1990) provide examples that show how standardization can result in situations in which interviewers record responses that are clearly based on a respondent interpreting a question in a way that was not intended by the question designer. The interaction that results from standardization may lead a respondent to become bored or impatient so that they answer questions in an acceptable manner, but with the goal of getting it over with rather than providing thoughtful answers.

Problems in question interpretation may be minimized through careful questionnaire construction but there are instances in which standardized interviewing may not be the most appropriate technique for generating the answers to certain types of questions. In these cases, the choice then becomes either using a method other than survey research or adapting survey research methods to the research problem. While we would not argue that survey research is ideal for all types of research problems, we do feel that the advantages of the method extend beyond instances in which standardization is the most useful interviewing technique. Based on our experience, we believe that well-trained interviewers are capable of carrying out surveys that require the use of nonstandardized techniques in a way that results in the collection of high-quality data that are collected on topics that might be beyond the reach of survey research if standardized interviewing techniques are used.

Another factor in survey sampling is the representativeness of the sample. For most surveys, that is a legitimate concern because each respondent is considered as representing others in the population. In the History Survey, there were mixed goals. For the closed questions, standardized interviewing techniques were used to enhance the representativeness of the responses to those questions. For the open questions where the nonstandardized techniques were used, representativeness is less of an issue because we assume that each respondent had a unique story to tell us about the past. Our concern was more focused on helping the respondents tell their stories fully and carefully. We felt that as long as there was diversity in the respondents’ demographic characteristics that the data would tell us a lot about how Americans interpret the past and its effect on their lives.

Sources for Statistics on African-American Population of the US

Region from Table 1, p. 1 in Characteristics of the Black Population: 1990 Census of Population. Bureau of the Census. 1994.

Age, sex, and marital status from Table 2, p2 in Characteristics of the Black Population: 1990 Census of Population. Bureau of the Census. 1994.

Educational Attainment from Table 20, p. 121 in Characteristics of the Black Population: 1990 Census of Population. Bureau of the Census. 1994.

Income from Table 48 in Social and Economic Characteristics, United States: 1990 Census of Population. Bureau of the Census. 1993.

Sources for the Statistics

Age, gender, state of residence, and place of birth are from Table 1, p.5 in Persons of Hispanic Origin in the United States: 1990 Census of Population. Bureau of the Census. 1993.

Marital Status of the Population, by Sex, Race and Hispanic Origin: 1970-1993. Table 59, in Statistical Abstract of the United States 1994. US Department of Commerce.

Education from Table 3, p. 81 in Persons of Hispanic Origin in the United States: 1990 Census of Population. Bureau of the Census. 1993.

Income from Table 5, p. 157 in Persons of Hispanic Origin in the United States: 1990 Census of Population. Bureau of the Census. 1993.

---

References

 Fowler, Floyd J. Jr. 1991. "Reducing Interviewer-Related Error through Interviewer Training, Supervision, and Other Means." Pp. 259-278 in Measurement Errors in Surveys, edited by P.P. Biember, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz, S. Sudman. New York: John Wiley & Sons.

Suchman, Lucy and Brigitte Jordan. 1992. "Validity and the Collaborative Construction of Meaning in Face-to-Face Surveys." Pp. 241-267 in Questions About Questions: Inquiries into the Cognitive Bases of Surveys, New York: Russell Sage.

Thelen, David and Lois Silverman. 1992. "About People and the Past: A National Survey of the Attitudes and Behaviors of the American Public." Proposal submitted to the Spencer Foundation.

Thornberry, Owen T. and James T. Massey 1988. "An Overview of Telephone Coverage." Pp. 25-49 in Telephone Survey Methodology, edited by R.M. Groves, P.P. Biemer, L.E. Lyberg, J.T. Massey, W.L. Nicholls II, and J. Waksberg. New York: John Wiley & Sons.