Chapter 11 of 13

Appendix B. Survey Methodology

Appendix B. Survey Methodology

Appendix B. Survey Methodology The National Survey of Youth and Religion (NSYR) is a nationally representative telephone survey of 3,290 Englishand Spanish-speaking teenagers between 13 and 17, and of their parents. The NSYR also includes 80 oversampled Jewish households, not nationally representative (described below), bringing the total number of completed NSYR cases to 3,370. The survey was conducted from July

2002 to April 2003 by researchers at the University of North Carolina at Chapel Hill using a random-digit-dial (RDD) method, employing a sample of randomly generated telephone numbers representative of all household telephones in the 50

United States, including Alaska and Hawaii. The national survey sample was arranged in replicates based on the proportion of working household telephone exchanges nationwide. This RDD method ensures equal representation of listed, unlisted, and not-yet-listed household telephone numbers. Eligible households included at least one teenager between 13 and 17 living in the household for at least six months of the year.' To randomize responses within households, and so help attain representativeness of age and gender, interviewers asked to conduct the survey with the teenager in the household who had the most recent birthday.

Parent interviews were conducted with either a mother or father, as available, although the survey asked to speak with mothers first, believing that they may be better qualified to answer questions about their family and teenager. Stepparents, resident grandparents, resident partners of parents, and other resident parent-like figures were also eligible to complete the parent portion of the survey.

An RDD telephone survey sampling method was chosen for this study because of the advantages it offers compared to alternative survey sampling methods.

Unlike school-based sampling, for example, our RDD telephone method was able

APPENDIX B: SURVEY METHODOLOGY

to survey not only school-attending youth, but also school dropouts, homeschooled youth, and students frequently absent from school. Using RDD, we were also able to ask numerous religion questions which many school principals and school boards often disallow on surveys administered in school. Explicit informed consent from parents also proved more feasible using RDD than school-based sampling. And the verbal reading of survey questions by trained interviewers facilitated question-and-answer clarifications that increased the validity of answers, compared to paper-and-pencil questionnaires administered en masse in school classrooms. Also, given the relatively low incidence rate (14 percent) of American households with teenagers 13 to 17 years old, the NSYR's RDD telephone survey method was much more cost-effective than an in-home survey, which would have been cost-prohibitive. The NSYR's RDD telephone method also eliminated potential design effect problems associated with sampling from a limited number of geographic or school clusters. Furthermore, the greater anonymity of an RDD survey interviewer on the telephone, compared to an in-person interviewer in the home, may have also increased the validity of teenagers' answers to sensitive questions and reduced possible biasing effects of in-person interviewers' sex, race, and age.2 No good sampling frames exist with which the NSYR might have conducted a mail survey, which typically garners extremely low cooperation and response rates in any case. Finally, superior Internet-based methods of sampling and surveying were not sufficiently developed and tested by the time of this survey's fielding to have been useful for the NSYR.?

Prior to conducting this survey, the researchers conducted 35 in-depth pilot interviews with teenagers to help inform the construction of the survey instrument. The researchers also conducted additional survey-focused interviews and focus groups with a variety of types of teenagers to improve question wording and comprehension. Prior to the survey, researchers also conducted modest

(N=175) pretests of the survey instrument using both nationally representative and convenience samples. Based on pretest results, the researchers revised questions and answer categories to enhance survey clarity and validity. The final survey instrument is available by Internet download at the project Web site: www.youthandreligion.org/publications/docs/survey.pdf.

The NSYR survey was conducted with members of both Englishand Spanishspeaking households. The English version of the survey was translated into Spanish by a professional translation service. That translation was then closely reviewed, evaluated, and revised by four separate native Spanish-speaking translation consultants and six Spanish-speaking survey interviewers to ensure the best translation for Spanish-speaking respondents. The final Spanish-language version was then programmed into the computer-assisted telephone interview

(CATI) system for calls to Spanish-speaking households. Surveys with Spanishspeaking households were conducted by native Spanish-speaking interviewers who are fluent in both English and Spanish and who had extensive experience conducting the survey in English before conducting the Spanish version. The parent and teen respondents from households could each choose the language with which to complete the survey, so that a parent might use the Spanish version, for example, while the teen used the English version. Spanish-speaking household numbers are included in the calculations of the national sample cooperation, completion, and response rates below.'

All survey interviewers received two days of project-specific training in the

SOUL SEARCHING

significance and purpose of the survey, the meaning of all survey questions and their answer categories, the proper pronunciation of religious terms, and the ethical treatment of human subjects. They also completed an Internet-based Human

Participant Protections Education for Research Teams course offered by the National Institutes of Health, U.S. Department of Health and Human Services

(www.cme.nci.nih.gov). Immediately prior to conducting all surveys, interviewers obtained respondents' verbal informed consent and provided respondents with information about the confidentiality of their answers and their right to refuse to answer questions. Household eligibility was determined through the use of an initial screening question about resident teenagers. Incentives of $20 to parent respondents and $20 to teenage respondents were offered to complete the survey, for a total of $40 to completing households.' Survey respondents were also able to complete the survey at their convenience by calling a toll-free number that linked to their sample record. Throughout the fielding of the survey, interviewers were monitored by project staff (primarily the authors) using remote technology to ensure data quality, and the interviewers, monitors, and researchers were routinely debriefed about survey performance. Upon completing the survey, all respondents were given contact information for the researchers, the research firm, and the university Institutional Review Board to verify the survey's authenticity or ask any questions about the survey or their rights as respondents. This information was also included in written form in thank-you letters accompanying the mailed incentives.

To help protect the privacy of survey respondents, the NSYR obtained a Federal Certificate of Confidentiality from the National Institutes of Health. With this certificate, researchers with the NSYR could not be forced to disclose information that might identify respondents, even by a court subpoena, in any federal, state, or local civil, criminal, administrative, legislative, or other proceedings. The certificate was thus useful for resisting any potential demands for information that would identify respondents (with the following exceptions: a Certificate of Confidentiality does not prevent respondents or members of their families from voluntarily releasing information about themselves or their involvement in the NSYR; if and when an insurer, employer, or other person were to obtain respondent's own voluntary written consent to receive research information, the NSYR could not use the certificate to withhold that information; neither does the Certificate of Confidentiality prevent the researchers from disclosing without respondents' consent information that would identify them as a participant in the research project in stated cases of child abuse or intent to hurt self or others; if teen respondents disclosed evidence of neglect or abuse, the NSYR had an obligation to inform the appropriate authorities).

The NSYR survey was conducted over nine months, between the end of July 2002 and the beginning of April 2003. All randomly generated telephone numbers were dialed a minimum of 20 times over a minimum of five months per number, spread out over varying hours during weekdays, weeknights, and weekends. The calling design included at least two telephone-based attempts to convert refusals.

Households refusing to cooperate with the survey yet established by initial screening to have children age 13 to 17 in residence and with telephone numbers able to be matched to mailing addresses were also sent information by mail about the survey, contact information for researchers, and a request from the principal investigator to cooperate and complete the survey; those records were then called

APPENDIX B: SURVEY METHODOLOGY

back for possible refusal conversions. Most cell phone numbers were screened out of the initial sample through the identification of unique cell phone exchange numbers. All other nonhousehold numbers (business, government, nonprofit, payphones, remaining cell phones, etc.) were screened out of the sample through direct calling dispositions and ascription of contact and noncontact telephone numbers for noncompletes based on proportions of household numbers among working telephone numbers." Of the sample producing only voice-mail or answering machine, 34 percent were dialed more than 99 separate times; 23 percent of those voice-mail numbers were dialed between 50 and 99 times, and 43 percent of those voice-mail numbers were dialed between 25 and 49 times.

The NSYR survey itself took a mean of 30 minutes to complete the parent portion and 52 minutes to complete the teen portion of the instrument, for a mean parent-teen combined survey length of 82 minutes. A total of 3,370 respondent households completed our full survey, 3,290 of which were RDD national sample respondents and 80 of which were Jewish oversample respondents.

The overall cooperation rate of our national sample was 81 percent. Ninety-six percent of parent complete households also achieved teen completes. Using the American Association for Public Opinion Research (AAPOR) RR4 calculator, the final NSYR national sample survey response rate was 57 percent. For descriptive purposes, a weight was created ("weight2") to adjust for number of teenagers in household, number of household phone numbers, census region of residence, and household income.'

Diagnostic analyses demonstrate that the NSYR provides a nationally representative and unbiased sample of U.S. households with resident teenagers age 1317. Comparative tests were run for the national representation of or potential sampling biases in the NSYR sample employing known population characteristics on key variables from 2002 U.S. Census data. Table B.1 shows that the NSYR provides a nearly perfectly representative sample of 13to 17-year-olds living in U.S. households by the comparable variables of gender, age, race/ethnicity, and household type. The region and household income variables also demonstrate a very close representation by NSYR data of the known national population, which is nearly entirely corrected for when weights are applied."°

Comparisons are also made in table B.2 between weighted NSYR data and 1999 National Household Education Survey (NHES) data, 1996 Monitoring the Future survey data, 1994 National Longitudinal Survey of Adolescent Health

(Add Health) data, and 1999 Survey of Adults and Youth data on key comparable variables. Table B.2 reveals only minor differences between the NSYR and other samples on these variables. The percentage in the NSYR sample who never drink alcohol, never used marijuana, and who smoke cigarettes regularly is similar to the percentages found in the Monitoring the Future and Add Health surveys.

Based on these interdataset analyses, we can say with some confidence that findings from the NSYR appear to offer a reasonably unbiased representation of the sampled population and so, particularly when region and income are weighted, might be assumed to accurately describe the population of U.S. teenagers age 1317 and their parents living in residential households.

These between-dataset comparisons were supplemented with analyses in the NSYR dataset comparing key demographic and behavioral traits of respondents who initially cooperated (90 percent of the sample) with respondents who were initial refusals but subsequently cooperated after successful attempts at refusal

SOUL SEARCHING

Table B.1. Demographic Characteristics Comparing Unweighted and Weighted NSYR Samples and the 2002 U.S. Census Population, Households with 13to .

17-Year-Olds in Residence, 2002-3 (Percentages) NSYR

NSYR (Unweighted) Census (Weighted) Census Region Northeast

Midwest

South

By.

West

Gender Male

Female

Age

15) Di

Teen race/ethnicity White/Hispanic

Black

Asian/Pacific Islander/American Indian/

mixed/other Household type Married couple

Income Less than $10K

$10K-$20K

$20K-$30K

$30K-$40K

$40K-$50K

$50K-$60K

$60K-$70K

$70K-$80K

if ie

$80K-$90K

$90K-$100K

More than $100K

Note: Percentages may not add to 100 due to rounding. conversion (10 percent of the final sample).!! Results, displayed in table B.3, reveal that the refusal-conversion respondents in the sample disproportionately are somewhat more Midwestern (and less Southern), involve more male parent and teen respondents, involve more 15and 16-year-old teen respondents (and fewer 13and 14-year-old respondents), are white (and not black), represent more married parent households, represent higher-income families, and represent more regular religious service—attending parents. Most of these differences are quite modest, however, typically representing less than 10 percentage point spreads.

Nevertheless, most of these differences remain statistically significant when considered together in logistic regression analyses predicting refusal-conversion versus

APPENDIX B: SURVEY METHODOLOGY

DO, Table B.2. Comparison of Weighted NSYR Results with Parallel National Survey

Results on Selected Lifestyle Variables, U.S. Adolescents, Ages 13-17 (Percentages, unless otherwise noted) Survey of Monitoring Adults and NSYR the Future Add Health Youth NHES

(N=3,290) (N=45,173) (N=15,084)

(N=874)

(N=6,569)

(2002-3) (1996) (1996) (1999) (1996) Never drinks alcohol

— i Never uses marijuana

_— Smokes cigarettes regularly a ve

—- Nights per week eats dinner

—_ with one parent (mean) School type attending Public

— — _—

Private religious

—- — —

Private secular

_ — — D) Home-schooled

_ — —

Attends religious services

39.

weekly or more Never attends religious

ih)

services Note: *=data from 2002 Monitoring the Future (N=43,700); percentages may not add to 100 due to rounding. initial-cooperator survey, respondents (results not shown). These modest differences suggest that, when conceptualizing a survey respondent continuum from easy cooperators to seri us noncooperators—and differences between potential respondents who lie on that continuum—successful researcher efforts to convert initial refusals for inclusion in the final dataset increase the full representation of different types of respondents in the final sample, reducing possible sampling biases that would be found in surveys with less rigorous refusal-conversion methods. Because the NSYR successfully employed multiple, extensive, sustained measures to convert initial refusals into cooperators—who represent fully 10 percent of the final sample—it significantly reduced possible biases affecting measured variables potentially associated with respondents' propensity to cooperate with surveys, rendering its sample data more accurately representative. This assurance, together with the between-dataset analyses presented above, corroborates this methodological report's general conclusion that the NSYR provides a satisfactorily unbiased, nationally representative sample of the target population of US. teenagers age 13-17 living in households.

In addition to a main national sample of 3,290 cases, the NSYR conducted surveys with a modest oversample of Jewish households, 80 completes in all, to help obtain a large enough number of cases with which to conduct meaningful statistical analyses of Jewish youth. A national RDD sampling method screening out all but eligible Jewish households with teenagers would, theoretically, have been ideal, insofar as it would have provided a true probability sample yielding a genuinely nationally representative oversample of Jewish teenagers in propor-

Table B.3. Comparison of Weighted NSYR Initially Cooperating and Refusal-conversion Respondents

(Percentages) Initial Refusal Cooperators Conversions

(N=3,030)

(N=340) Census region Northeast

Midwest

South

West

Male parent

Male teen

Teen age

Race/ethnicity White/Hispanic 78 |

Black

Asian/Pacific Islander/

American Indian/ mixed/other Household type Married Couple

Income Less than $10K

$10K-$20K

$20K-$30K

$30K-$40K

$40K-$50K

$50K-$60K

$60K-$70K

o) gy)

$70K-$80K

$80K-$90K

$90K-$100K

More than $100K

Parental religious service attendance Parent attends weekly

or more Parent never attends

Note: Percentage may not add to 100 due to rounding.

APPENDIX B: SURVEY METHODOLOGY

tion to their actual geographic and social locations in the United States. However, because Jewish households with resident teenagers age 13-17 represent only about 1 in every 400 U.S. households (approximately 0.25 percent of all households), this method would have been unreasonably time-consuming and costprohibitive. As a more efficient alternative, the NSYR employed another standard survey method for obtaining a Jewish oversample, by calling a set of telephone numbers listed with one of 200 "Jewish" surnames agreed on by the National

Jewish Technical Advisory Committee and selected from White pages listings throughout the United States on a population-proportional basis. These numbers were obtained from the survey sampling firm Genysis, Inc., and screened in calling for eligible households with resident 13to 17-year-olds. Compared to yet a third alternative oversampling method—namely, the high-density Jewish sampling frame method of RDD calling of replicates of telephone numbers within geographic areas containing defined minimum Jewish residency rates, which, by definition, samples areas with higher concentrations of Jewish inhabitants, schools, synagogues, and other Jewish institutions and produces all of the associated sampling biases—the listed Jewish surname method employed by the NSYR has the distinct advantage of sampling Jewish youth from all social and geographic locations in the country. Nevertheless, the listed surnames sample method employed is by no means a nationally representative probability sample, because it systematically excludes both Jewish households with unlisted telephone numbers and Jewish households with surnames not considered by the survey sampling firm to be "Jewish-sounding."

To estimate any possible sampling bias involved in this nonprobability oversampling method, analyses compared key characteristics of the Jewish cases drawn from the nationally representative sample to those drawn from the Jewish oversample on a variety of key demographic and religious measures. Crosstabular comparisons of the unweighted Jewish national sample and oversample cases reveal no statistically significant differences with regard to the following variables: teen gender, age, race, religious service attendance, religious youth eroup participation, belief in God, importance of religious faith, type of school attending, household income, family debt or savings, family home ownership, parental marital status, parental religious service attendance, parental importance of faith, and father's education.'2 The two differently sampled groups, then, are remarkably similar along important dimensions of analysis. Therefore, it is deemed not necessary to construct or use special weights to compensate for any religious or demographic differences generated by the nonprobability oversampling method used here.

In sum, the National Survey of Youth and Religion may be taken as providing a nationally representative RDD telephone survey of 3,290 Englishand Spanishspeaking teenagers living in households in all 50 U.S. states in the years 2002 and 2003, between the ages of 13 and 17, and of their parents; the survey also includes 80 oversampled Jewish households (not nationally representative), bringing the total number of completed NSYR cases to 3,370. Every effort was made in project design, instrument construction, interviewer training, and survey fielding to produce the best possible results. Multiple diagnostic analyses demonstrate that the NSYR appears to provide a reasonably unbiased representative sample of its target population and so, when weights are applied, can be taken to accurately describe the population of U.S. teenagers age 13-17 and their parents living in residential households.

Table B.4. Demographic and Religious Traits of Religious Ideal Types of U.S.

Adolescents, Ages 13-17 (Percentages) 'Religious Type U.S.

Disengaged Sporadics Regulars Devoted

Ais} Ai

Female Race

White Black Hispanic Other race fi 8) Region Northeast Midwest South Mountain/Pacific

2) 17;

Parent marital status Married Living with unmarried partner Widowed Divorced Separated Never married WN on fr School type attending Public Catholic Private other Christian Jewish Private nonreligious Home-schooled Other or not going to school Cote RS ieee:

Religious type Conservative Protestant Mainline Protestant Black Protestant Catholic Jewish Mormon Other religion Not religious

ray WNWA KF aADH

wWnNnd Source: National Survey of Youth and Religion, 2002-3.

Note: Percentage may not add to 100 due to rounding; cells of <1 are reported as ~,

APPENDIX B: SURVEY METHODOLOGY

Table B.4 shows the demographic characteristics of the four religious idealtype categories used in the analysis of chapter 7.

↑ Back to top

↑