The Jewish Federation of Greater Philadelphia (JFGP) wanted to estimate the size of the Jewish population in the five-county Greater Philadelphia area (Philadelphia, Bucks, Chester, Delaware, and Montgomery counties). The goal was to describe a wide range of characteristics of Jewish residents, overall, for each county, and for a set of eight local communities (referred to as Kehillot). Figure 1 shows the boundaries of the target area. The gray areas in the western suburbs were assumed (based on JFGP contacts in each county) to have few if any Jews and thus were excluded from the survey.
Beyond the usual difficulties of conducting a population study, surveying the Jewish community brings additional complications. The Census Bureau, which is the source of demographic data often used in designing survey samples, does not collect information about religion in any of its surveys. Many Jews do not identify as religiously Jewish, but rather as culturally or ethnically Jewish, so may not answer “Jewish” to a question of “What religion are you?” Further, many Jews do not connect to any local Jewish organizations, so they are not likely to be found on lists of likely Jews from synagogues, Jewish community centers, or other organizations.
Previous Philadelphia Study
The previous study of the Greater Philadelphia area Jewish community was conducted in 2009. An RDD sample of landline telephone numbers was selected and interviews were conducted using computer-assisted telephone interviewing (CATI). (Earlier studies in 1984 and 1997 had also used RDD with CATI.) This was supplemented by lists obtained from local Jewish organizations. Due to changing telephone usage patterns, the use of a landline-only RDD survey in 2009 excluded the households that either did not have a landline telephone number or had a landline number but received all or almost all calls on cell phone numbers (e.g., only used their landline for a fax machine), estimated nationally to be 41 percent of households (Blumberg and Luke 2009). The excluded percentage was even larger for select subpopulations, for example, younger adults.
2019 Philadelphia Study
Collecting data by telephone has become more difficult and more costly as the population has transitioned from landline telephone service toward primarily or exclusively cell phone service. As stated previously, new technologies to identify and/or block incoming calls, and respondents having less “free time” to answer surveys has fueled the need to change the approach to collecting population-level data (Olson et al. 2019).
Sample Design. The 2019 Philadelphia study combined ABS with 50 lists from local Jewish organizations. The ABS frame for this study, which is based on the U.S. Postal Service’s Computerized Delivery Sequence file and is maintained by Marketing Systems Group (MSG), consisted of the set of all residential addresses in a list of ZIP codes that were identified by JFGP as likely having at least some Jewish population. Based on this knowledge from JFGP, using this frame provided almost complete coverage of Jewish persons living in households in the five-county area.
The 50 lists were deduplicated and matched to the ABS frame to partition it into separate strata for low eligibility list addresses, high eligibility list addresses, and non-list addresses. The low eligibility lists were either of college students (more likely to have moved) or a purchased list of “likely Jewish households” from a market research firm, where the proportion of households eligible to participate was expected to be low. The high eligibility lists came from organizations such as synagogues, Jewish community centers, and Jewish social services providers. Additionally, the sample was stratified by the eight Kehillot to facilitate geographic estimation. Out of 1.6 million residential addresses in the eligible ZIP codes, 3 percent were placed in the high eligibility stratum (where the proportion of households eligible to participate was expected to be high) and another 5 percent in the low eligibility stratum. A key to unbiased estimation is that respondents in each stratum were only weighted up to represent others in that stratum, so for example, Jews who are well connected to the community and thus appear on at least one high eligibility stratum list are only weighted up to represent others also found on such lists. Thus with ABS, the responses from these list strata are weighted up to only represent 8 percent (3% + 5%) of all households. The rest are represented by respondents in the non-list (generally less affiliated) stratum. This represents a major improvement over the approach used by Aronson as described earlier, where households on Jewish lists were assumed to represent all Jews, including less affiliated Jews.
One caveat of the ABS approach is that the sample represents the household population but does not include Jewish adults who are living in nursing homes, military barracks, and other institutional housing, nor does it include homeless Jews. This caveat applies to other sample designs as well. Those living in non-institutional residential settings, however, including most assisted-living facilities and non-barracks housing on military bases, were eligible for inclusion.
In addition to the major improvement in coverage of the target population, ABS provided additional improvements over the previous design. ABS allows for specific, accurate targeting of geographic areas of interest. Each ZIP code was connected to a specific Kehillot, allowing us to ensure a specific sample size was allocated to each. Each address is associated with a specific county. Not only does this ensure a sufficient number of completed cases in each target geography, but it also facilitates using area-level characteristics in statistical adjustments aimed at reducing potential nonresponse bias. With RDD, the characteristics available for nonresponse adjustments are aggregate characteristics for large geographic areas (typically, the primary ZIP codes associated with the telephone exchange). With ABS, we know the county, Kehillot, and (through geocoding) the census tract in which every nonresponding address is located, and from the American Community Survey (ACS) we can obtain characteristics of their census tract (e.g., percent renters, average income). This allows for the use of area-level characteristics for the particular area in which the address is located, resulting in nonresponse adjustments that are likely to be more effective in reducing bias in survey estimates.
Data Collection. Data collection occurred from late January through July of 2019. Each sampled address was mailed an initial invitation to complete the screener via the web, with a unique ID and PIN for each address. The mailing contained a $1 cash incentive to encourage participation. Households could request a hard copy screener survey but were encouraged to use the web. Follow-up postcards encouraged web response, followed by a paper copy of the screener mailed to remaining nonrespondents. As a result, 60 percent of screener respondents chose to respond via the web, including older adults.
In each letter and postcard, a toll-free telephone number and email address were provided for anyone with a question about the survey. This number was also available if someone preferred to answer the questionnaire over the telephone. Telephone data collection was only used for a few interviews (both English and Russian), but the option was provided to expand the potential methods of completing the survey.
The screener and the survey were offered in both English and Russian. A number of Jewish families from Russia and the former Soviet Union have moved to the Greater Philadelphia area over the last 40 years and many still speak only Russian at home. This necessitated offering a Russian language survey alternative to encourage this group to participate. Both the paper copy and web instruments were offered in the two languages.
Data were collected through a two-phase design, where the screener was used to determine whether any adult in the household was Jewish; if so, the household was eligible to complete the main questionnaire. Respondents were promised a $10 incentive upon completion. Web respondents to the screener continued seamlessly into the main questionnaire, while eligible paper screener respondents were mailed a paper copy of the main questionnaire, but were still offered the option of answering on the web. If the paper screener indicated that the only residents were ages 65 or older, then a tailored version of the paper questionnaire was sent with a larger font size to improve readability, eliminating all questions related to children in the household. Eighty percent of main survey respondents chose to respond via the web.
Who is Jewish? In the 2009 study, two questions were used to identify Jewish households:
“Is there anyone in the household who considers himself or herself to be Jewish?”
If no, then the screener respondent was asked whether either of their parents was Jewish, and if so, the household was classified as Jewish.
Research in Jewish demography (Pew 2021, 2013; Charme et al. 2008; Horowitz 1998) has revealed that there are multiple ways in which some people consider themselves Jewish, and that this determination may change over time. As a result, for 2019 we used a more detailed set of questions to determine whether the household qualified as Jewish. Respondents to the household screener were asked:
“What religion are you?”
(If not Jewish) “Are you Jewish by religion?” “Are you Jewish by ethnicity or heritage?” “Are you Jewish by culture?”
(If no to all of the above questions) “Were you raised Jewish or did you have a Jewish parent?”
(If no to the above question) “Does any other adult in the household consider himself/herself Jewish by religion/ethnicity or heritage/culture or had a Jewish parent?”
Jewish households were defined as those in which the respondent self-identified as Jewish (religiously, ethnically, or culturally) or had a Jewish parent or was raised Jewish (and didn’t have another religion), excluding Messianic Jews; or whose spouse/partner/other adult identified as Jewish (religiously, ethnically, culturally, or had a Jewish parent and no other religion).
These questions do not necessarily reflect an expansion of the definition of Jewishness relative to 2009, but they do clarify the many ways in which a respondent might identify. We believe this provides a clearer path to identifying all those who are Jewish. Also, by providing a neutrally worded initial question, we believe we encouraged those of all religions to complete the screener in a way that the 2009 questions did not.
Screening for those who were raised Jewish or had a Jewish parent, but who do not currently identify as Jewish (or any other religion) does provide a more inclusive definition of Jewishness. By examining the responses of those who qualified due to this last question, we found that they resemble many others who identified as Jewish via the first 3 questions. For example, they light Chanukah candles (21%), attend Passover Seders (13%), and attend High Holiday services (7%). While this group only represented one to two percent of Jews, they behave similarly to some other Jews and we found they were worthwhile to include in estimates of the Greater Philadelphia Jewish population.
Collecting Data in 2019. Collecting data on the Greater Philadelphia Jewish population in early 2019 posed several challenges. The Tree of Life synagogue attack in Pittsburgh, PA took place October 27, 2018. While we were collecting data in Philadelphia, not Pittsburgh, there are close ties between the two communities. During the data collection period (April 27, 2019) the Poway synagogue attack in California occurred, further putting the American Jewish community on edge.
Throughout the data collection period there were almost daily news items about Russian interference with the 2016 election, the Trump administration’s close ties with Russia, and possible impeachment. With the household screener offered in English or Russian, we encountered some suspicion and pushback from sampled households. For example, one sampled household suggested that we were “doing just what the Nazis did.” Another household worried that “given the results of the 2016 election, maybe Russian was going to be the second official language of America.”
Response Rates. A total of 79,486 addresses were sampled, from which 10,787 households completed the screener. There were 2634 screeners identifying eligible households (at least one Jewish adult) of whom 2119 completed the main survey. Table 2 provides detailed response rates overall and by stratum.
Overall, 12.2 percent of households completed the screener, with 78.6 percent of those eligible completing the main survey questionnaire. These response rates were similar for both the non-list ABS and low eligibility list strata. The high eligibility list stratum, however, had substantially higher participation rates, with 25.6 percent completing the screener, and 82.8 percent the main instrument. While the non-list ABS and low eligibility list strata had similar response rates, that is not true for eligibility; while only an estimated 12 percentFootnote 2 of households in the non-list ABS stratum were determined to be Jewish households, 47 percent of those in the low eligibility list stratum were Jewish households, and 87 percent of those in the high eligibility list stratum.
Studies have demonstrated that a well-designed ABS study attains response rates higher than those using RDD methods (Olson et al. 2019; Montaquila et al. 2013), especially when one considers that the non-list ABS frame is absent those who are most likely to respond positively to the named sponsor. In addition, knowing the location for ABS nonrespondents allows for adjustments in the weighting process that reduces a major source of potential bias in telephone-based data collection.
Estimating the Size of the Jewish Population. Each respondent was weighted with a final weight that accounts for the probability of selection, along with adjustments for eligibility and nonresponse. The weights were adjusted to county-level controls derived from the American Community Survey. The resulting estimates are much more accurate than those from earlier methods. Responses from Jewish respondents on high eligibility lists are only weighted up to reflect others on those lists. The same applies to respondents from the low eligibility lists. Finally, respondents from the non-list stratum cover Jews who are not found on any of the lists. In this way we ensure that responses from highly affiliated Jews are not used to represent those who are not affiliated with the Jewish community. In addition, since the location for all nonrespondents is also known, any differential response rates across geographies are controlled for in the weighting process, reducing the potential for bias in the estimates.
The methodological improvements in the 2019 survey approach contributed to significantly larger estimates of Jewish population in Greater Philadelphia relative to 2009. Table 3 shows the estimates from the 2009 survey, the 2019 estimates using a comparable definition of Jewishness (i.e., without the last question about being raised Jewish or having Jewish parents), the percent change, the best estimate for 2019 (including the more expansive definition of Jewish households), and a 95 percent confidence interval on that estimate.
As mentioned previously, the addition of the screening question on being raised Jewish or having a Jewish parent did not have a large impact on the estimated size of the Jewish population. The number of Jewish households in 2019 is over 60 percent larger, and the number of people in such households and the number of Jewish adults are both over 70 percent larger than the 2009 estimates. During this same time period, the overall population of the five-county area increased by only 3.5 percent.Footnote 3 While it is possible that the Jewish population has been increasing faster than the general population, that alone is likely not the only explanation for the large difference since 2009. Given the lack of evidence of massive recent growth, we attribute most of this change to the improved methodology. Leadership in the Philadelphia Jewish community agreed that growth over the previous decade was not responsible for the change; rather, they agreed that the change reflects the fact that the 2019 study captured parts of the community not previously included.