Introduction

Comparison and comparability lie at the heart of social science, but precise comparison is virtually impossible without using similar methods and similar data. Comparable historical data sets are especially scarce. Until very recently, this was particularly true for micro-level data on social and demographic behavior in past populations.

In recent decades, however, social demographers, historians, and economic historians have compiled and made available a large number of household- and individual-level data sets describing historical populations in North America and Europe (Ruggles 2014). The creation and release of these large data sets have allowed researchers to move beyond broad comparisons of aggregates and produce comparative insights at the meso- and micro-levels. Prominent examples include the vast collection of historical and contemporary census data available from the Integrated Public Use Microdata Series (IPUMS) and the North Atlantic Population Project (NAPP), as well as other Western historical data projects, such as the BALSAC Population Database, the Historical Sample of the Netherlands (HSN), Le Programme de Recherche en Démographie Historique (PRDH), the Scanian Economic Demographic Database (SEDD), the Umea Demographic Database (UDDB), and the Utah Population Database (UPDB).Footnote 1 These advances in historical population data construction have contributed enormously to the development of comparative historical demography in particular, and comparative social science in general.Footnote 2

In this article, we introduce five household- and individual-level historical panel data sets for East Asian populations that are similar enough in content and organization to be compared not only with each other, but also with their European and North American counterparts. These data sets include the China Multi-Generational Panel Dataset – Liaoning (CMGPD-LN) 1749–1909, the China Multi-Generational Panel Dataset – Shuangcheng (CMGPD-SC) 1866–1913, the Japanese Ninbetsu-Aratame-Cho Population Register Database – Shimomoriya and Niita (NAC-SN) 1716–1870, the Korea Multi-Generational Panel Dataset – Tansung (KMGPD-TS) 1678–1888, and the Colonial Taiwan Household Registration Database (CTHRD) 1906–1945. Altogether, these five data sets contain 3.7 million linked observations of 610,000 individuals, with more individuals and observations to come.

We divide this article into six parts. First, we summarize early efforts to produce systematic comparable data at the national, regional, household, and individual levels and their academic contributions. Next, we discuss the key features of these new East Asian data. The following two parts introduce the historical institutions that produced the original data and the subsequent processes by which these data were transcribed and reconstructed into individual-level panels. In the concluding two parts, we review the strengths and limitations of these data as well as their potential for social science inquiry.

Development of Data Comparisons

Large-scale cross-national comparative social science emerged in the mid-twentieth century with the creation and dissemination of increasingly detailed and systematic data sets, initially at the macro-level, and then at the meso- and micro-levels. The first such global comparative enterprise may well be the Human Relation Area Files, which beginning in 1949 made a collection of materials on human behavior, culture, and society available to the academic community, first in print and beginning from 1994, and then online (Ember 1997). In the 1960s and 1970s, such quantitative comparative projects as the Princeton European Fertility Project began to make use of meso-level data sets consisting of provincial social and demographic indices. However, it was not until the 1970s and 1980s—when studies began using individual- or family-level data from family reconstitutions and historical censuses—that historical demography emerged as a distinct subfield within population studies, historical sociology, comparative social science, and various subfields of health science. Moreover, it was not until the creation and release of the Integrated Public Use Micro Series (IPUMS) and other micro-level data sets largely beginning in the 1990s that these population-related subfields became a central focus of academic attention (Ruggles 2014).

The rapid increase in the numbers of publications referencing these major data sets or projects illustrates the contributions to scholarship of these successive advances in data and associated methods.Footnote 3 Figure 1 provides counts of publications since the middle of the twentieth century by Google Scholar that mention these data sets and/or projects by five-year period. In each case, the availability of systematic comparative data inspired a sustained stream of academic references. The first global data set (the Human Relations Area Files) and the first major historical demographic comparative project (the Princeton European Fertility Project) continue to be referenced by 1,000 and 100 papers, per five-year period, 60 and 40 years, respectively, after they were first created.

Fig. 1
figure 1

Google Scholar citations generated by comparative “big” social science data

Full size image

Even more striking is the recent, rapid, and still ongoing increase in interest in individual- and family-level microdata (Ruggles 2014). IPUMS, in existence for only two decades and until recently consisting solely of historical and contemporary census data from the United States, now generates almost 2,000 references per five-year period.Footnote 4 Finally, and most relevant to the East Asian data sets introduced here, even though the best-known complex longitudinal data sets describing individuals and households have been publically available only for at most two decades, they already generate nearly 700 new academic references per five-year period.

The simultaneous development of advanced statistical methods to analyze complex longitudinal data has allowed quantitative social science to move from comparison of descriptive aggregate statistics to examination of differences within populations, as well as measurement of associations between variables at the individual and family level. This shift has facilitated the development of more ambitious explanatory and causal models that link demographic behavior with current and past context and circumstances. However, these advanced methods demand increasingly complex and detailed data, including not just vital events and family composition but also occupation; socioeconomic status (SES); wealth; and economic conditions, such as food prices. Because the meaning of occupation, SES, and wealth varies across contexts, the data requirements for comparative historical demographic research are increasingly challenging.Footnote 5

Quantitative historical comparison through the application of these new data and methods has evolved from describing regional or national differences to uncovering similarities within differences, and most recently identifying differences within similarities (Lundh et al. 2014). The Princeton European Fertility Project, an early comparative project, tested existing explanations for the European fertility decline based on computation and comparison of national and provincial differences in demographic rates and socioeconomic indices throughout Europe (Coale and Watkins 1986). Although the Princeton project substantially improved our knowledge of the fertility decline, the aggregate rates and indices on which it relied did not allow for micro-level comparisons that link demographic behavior to individual or household context.

The Eurasian Project in Population and Family History is one recent example of comparative historical demography that uses micro-level data to identify similarities within differences at opposite ends of Europe and Asia in the past.Footnote 6 It makes use of individual-level longitudinal data from historical household registers to compare sociodemographic behavior in a variety of communities in southern Sweden, eastern Belgium, northern Italy, northeastern Japan, and northeastern China. These comparisons allow us to relate demographic behavior to individual, household, and community contexts across Eurasia.Footnote 7 The overall conclusions, summarized in three volumes published as the MIT Press Eurasian Population and Family History Series, focus on East-West divergence and convergence, and challenge current macro historical sociological theories without, however, proposing alternatives (Bengtsson et al. 2004; Lundh et al. 2014; Tsuya et al. 2010). The results suggest that before we attempt to produce new grand social theories at a global scale, we need first to make more detailed comparisons within East and West, focusing on communities that have similarities in terms of background and context.

Recognition of limitations to the focus on East-West comparison in the Eurasian Population and Family History Project inspire our new effort to map similarities and differences in East Asian population behavior through comparative analysis of population register databases: the East Asian Population and Family History Project. To distinguish from the earlier Eurasian Population and Family History Project (EAP I), we call this new project EAP II, which focuses specifically on neighboring populations in East Asia that are more similar in terms of background and context. Participants have already met three or more times at EAP II–related meetings, and have met less formally in other venues. These meetings have already yielded a collection of papers on migration in historical East Asia (Campbell 2013; Kim et al. 2013; Kye and Park 2013; Son and Lee 2013; Tsuya and Kurosu 2013). The coordination and cooperation that we hope to promote in EAP II will be the first step to such detailed comparison.Footnote 8

New East Asian Microdata

The five EAP II data sets are from four distinct regions in East Asia, identified on Map 1, scattered over an area of less than 4 million square km, measuring 1,700 km east to west and 2,300 km from north to south. These regions are geographically contiguous and share similar though far from identical social structures, cultural norms, and political institutions and ideologies.

Map 1
figure 2

EAP II study populations

Full size image

All five EAP II historical panel data sets are longitudinal in the sense that they contain linked records for individuals over time. Longitudinal data on individuals are valuable because they allow present behavior to be linked with prior circumstances. This allows researchers not only to describe patterns of behavior but also to explain their causes and consequences. Unlike aggregate-level time series that reflect only national, regional, or community averages, individual-level longitudinal data provide life histories for each individual, which makes possible disentangling the complicated relationships between individual behaviors at different time points.

These East Asian data sets, like the EAP I and most European historical panel data sets, are not nationally representative.Footnote 9 Each covers only a limited number of communities; however, unlike proportionally representative samples, they do so in their entirety. They are historical analogs to the contemporary data collected around the world by the participants in the International Network for the Demographic Evaluation of Populations and their Health (INDEPTH) Network (Sankoh and Byass 2012).Footnote 10 These data include information on vital events—such as fertility, mortality, marriage, migration, and longitudinal information—on household context and individual characteristics for all individuals in their respective registration areas. The EAP II data sets also record such details as occupation, kinship, (usually) property, and (sometimes) civil service examination attainment, which allow us to aggregate dynamic information on community and household context based on individual information. These data are usually constructed from household or civil registers that survive to the present day. Such historical sources were originally compiled by local governments in connection with population regulation, taxation, religious investigation, and other administrative functions (Ding et al. 2004; Hayami 1979; Kurosu 2002; Lee et al. 2010; Son 2007).

The five EAP II data sets are accessible online or in person subject to application. The CMGPD-LN and the CMGPD-SC and associated documentation are available from an Inter-University Consortium for Political and Social Research website.Footnote 11 The digital images and files for the KMGPD-TS, the Tansung household registers (THR), are also available online,Footnote 12 as are longitudinal links that connect individuals across registers.Footnote 13 The CTHRD is maintained by the Program for Historical Demography (PHD) at the Academia Sinica in Taipei. Researchers can apply for access to data through the PHD website.Footnote 14 The NAC-SN was originally constructed by Akira Hayami and his colleagues in Japan, and is now housed at the Population and Family History Project at Reitaku University. At present, researchers may submit a proposal to the Population and Family History Project at Reitaku University, and if approved, carry out the analysis at Reitaku University.Footnote 15

Additional data sets constructed from East Asian household registers exist and may eventually be made available. The China Multi-Generational Panel Dataset–Imperial Lineage (CMGPD-IL) describes 120,000 individuals over 13 generations who belonged to the Qing Imperial Lineage from 1616 to 1936 and is already entered and linked in its entirety (Lee et al. 1993). Choson dynasty Korean household registers from Daegu County and Jeju Island, similar to the original sources for the KMGPD-TS, have also been digitized, along with available data of urban population in Seoul transcribed from colonial household registers under Japanese government. Although the data derived from Japanese population registers presented here include only two villages, data for another three villages and one local town in the same region are already entered, and additional data from other regions in Japan are being entered.Footnote 16 The complete CTHRD includes 14 other locations that are also available for access through the PHD website. More locations in Taiwan are being added.

The EAP II data sets provide comparable and detailed information on demographic and socioeconomic characteristics of individuals and households. Table 1 summarizes their contents. They all provide the age of individualsFootnote 17 and annotate the recent occurrence of key demographic events, such as birth, death, marriage, and migration. Additionally, they all specify individual’s relationship to the household head, record the village of residence, and include information on occupation of males or at least household heads. The CMGPD-LN and CMGPD-SC record occupation only for state employees, such as local officials, artisans, and soldiers. The CTHRD, KMGPD-TS, and NAC-SN record other occupations, including servant and merchant. They all contain at least some information on physical disability and disease, but such information is less consistent, systematic, and comparable. The CMGPD-LN, CMGPD-SC, and KMGPD-TS indicate titles associated with the civil examination system, which are indicative of a high level of education. The CMGPD-SC, NAC-SN, and CTHRD also have information on household landholding.Footnote 18 Most importantly, all the data sets allow for longitudinal linkage of individuals and linkage of individuals to their various coresident and noncoresident kin,Footnote 19 which we describe in the upcoming section, Data Construction.

Table 1 Available information in the five EAP II data sets
Full size table

All these registers provide information that can be used to categorize individuals or households according to their social and economic status: occupational prestige for the CMGPD-LN; property entitlements for the CMGPD-SC; social status for the KMGPD-TS; household head’s occupation and property tax for the CTHRD; and a combination of social status and tax liability for the NAC-SN. These measures are not directly comparable across populations. Previous studies have used them, however, to categorize individual SES into high/middle/low and then examine gradients in demographic behavior. Analyses of the CMGPD-LN typically differentiate individuals according to the status of the state farm population with which they were affiliated and the official position they held, if any. Comparatively, analyses of the CMGPD-SC population usually divide the population into three categories according to their property entitlement: 64.4, 34, and 0 hectares (Chen 2009). The KMGPD-TS recognizes three broad categories in the original data: nobles (yangban), commoners (sangmin), and subordinates (nobi). The CTHRD allows for households to be differentiated by occupation of household head or taxed household landholding: high SES refers to households who paid more than 50 yuan (in colonial Taiwan currency) in land tax or the heads who had such white-collar professional occupations as administrative officials, doctors, teachers, or other professionals; middle SES refers to households who paid 1 to 49 yuan in land tax or to heads who had regular blue-collar or retail jobs; low SES refers to households who paid less than 1 yuan in land tax or to heads who were itinerant peddlers or heavy laborers (Hsieh and Chuang 2005). The NAC-SN divides households according to both principles. It records formal statuses, differentiating titled peasants (honbyakusho) who owned land from landless peasants (mizunomi). It also records household tax liability for titled peasants, who were assessed based on the productivity of their land, regardless of their formal status.

East Asian Household Registers and Historical Institutions

The registers from which these data sets were created are products of historical systems of civil, financial, and military administration. The CMGPD-LN and CMGPD-SC are transcribed from triennial and annual Eight Banner population registers, respectively, from Liaoning province between 1749 and 1909 and from Shuangcheng County in Heilongjiang province between 1866 and 1913, in northeast China.Footnote 20 The Eight Banner system was a civil and military administrative system organized by the Qing to govern the Manchurian and Mongolian provinces in Greater North and Northeast China, as well as the Qing garrison populations in China Proper.Footnote 21 The vast majority of the population in the CMGPD-LN were descendants of Han Chinese migrants who migrated from Shanxi, Hebei, and Shandong province to Liaoning after the founding of the Qing dynasty. There were also a small number of indigenous and descendants of earlier settlers who according to their surname or their registered status were Mongol, Manchu, or Korean. The CMGPD-SC population consisted of the descendants of migrants who arrived in Shuangcheng in the early nineteenth century. The original migrants were drawn from Eight Banner populations in Beijing and elsewhere in northeast China. According to the registered ethnicities recorded in the registers, they were a mixture of Manchu, Han, Mongol, and other groups.

The CMGPD-LN and CMGPD-SC registers are organized first by the administrative affiliation of the population, and then within register, by village of residence, household group, and household. Within households, individuals are listed according to their relationship to the head. Administrative affiliation is an important dimension of status and largely hereditary. Families remain affiliated with their original administrative population even after they move elsewhere in the region; and in the case of the CMGPD-LN, continue to be recorded in their original registers, although with their new location identified. Two households containing related individuals may be listed next to each other in the registers even though they reside in separate villages. As a result, both the CMGPD-LN and CMGPD-SC are valuable sources for migration and community studies because they not only provide the same basic information on households as other data sets but also allow for the tracing of households that move within the region and explicitly annotate individual departure from the region.

With a combined coverage of more than 800 communities across a diverse variety of geographic and socioeconomic contexts, analysis of the CMGPD-LN and CMGPD-SC should continue to produce findings that improve our understanding of general patterns of social and family organization in China, including the spatial dimensions of social organization (Lee et al. 2010; Wang et al. 2013). The physical locations of all the communities in the CMGPD-SC are known with precision, as are the physical locations of the 200 or so communities in the CMGPD-LN that accounted for 90 % of the population.

The KMGPD-TS is transcribed from triennial Korean civil household registers (hojŏk) compiled between 1678 and 1888 from Tansung County in South Korea.Footnote 22 Whereas the registers for the CMGPD-LN and CMGPD-SC record only those individuals affiliated in some way with the Eight Banner administrative and military system itself, the Tansung registers were intended to cover all people who actually resided in the area, without consideration of their political status or identity.Footnote 23 The population consisted of largely peasants but also local nobles and servile households or subhouseholds (nobi).Footnote 24 In the register, each individual was assigned to a household (ho). Then, households were organized into tong (five-household units), ri (village), and myeon (subcounty) in ascending order. The Tansung registers covered eight myeon, each in a separate register series. Because inclusion was based on administrative jurisdiction of residence, the Tansung registers sought only to record people who actually reside in Tansung. Unlike the CMGPD-LN, they do not follow individuals who leave the area except to indicate that they have left. Sometimes they specify destination.

The NAC-SN is transcribed from a set of Japanese population registers (ninbetsu-aratame-cho) from two villages, Shimomoriya and Niita, in northeast Japan between 1716 and 1870.Footnote 25 Each year, usually around lunar March, officials registered the residents in these villages and recorded any vital event that the individual experienced in the preceding year. Residents of the two villages were mostly peasants. Like the Tansung registers, the NAC registers also record individuals based on their administrative jurisdiction of residence. Although it is impossible to follow individuals after they leave the village, the registers record the year in which they left, and always record their reason for departure and their destination.

The CTHRD is transcribed from Taiwan household registers (hujiziliao) compiled by the Japanese colonial administration from 1906 to 1945.Footnote 26 The sample analyzed here covers eight locations in north and central Taiwan.Footnote 27 This sample includes some urban areas, but the majority of the population recorded in the Taiwan colonial household registers were farmers. In contrast with the annual or triennial CMGPD-LN, CMGPD-SC, KMGPD-TS, and NAC-SN, the colonial Taiwan registers, like the eastern Belgian registers in the EAP I, were updated continuously as vital events and other information occurred. Each household in the original register had one or more pages according to the household size, and each household member was represented by a column on that page in which their vital events and other information were recorded.Footnote 28 If changes occurred that fundamentally altered the household—for example, the household head was replaced—the original page would be crossed out, and a fresh entry started on a new page with a new household head. Although the Taiwan colonial registers do not follow individuals who moved out of the community or trace those who moved in, they provide information on the time of the move as well as the destination or origin. Importantly, the information on timing allows for the censoring of observations in event-history analysis of demographic behavior.

Data Construction

All five types of household registers require linkage of entries for the same individual in different locations to produce life histories that can be subjected to longitudinal analysis. Although their content resembles that of the large longitudinal databases of historical Western populations being constructed from linked parish and tax data (Mandemakers and Dillon 2004), the organization and format of the original data differs fundamentally, requiring a distinct approach to data set construction. The original registers from which the CMGPD-LN, CMGPD-SC, KMGPD-TS, and NAC-SC were constructed resemble annual or triennial censuses in the sense that they provide detailed snapshots of the population at fixed intervals in which individuals are observed repeatedly, while the Taiwan colonial registers consisted of one page for each household that was updated as events occurred. The annual and triennial household registers do not trace individuals from one register to the next, and they require manual or automated longitudinal linkage to produce the life histories that relate outcomes and behaviors to characteristics and context earlier in the life course. The continuous Taiwan colonial registers also require linkage of information about the same individual recorded in different households at different stages of their life to produce life histories. Although the page for each household offers complete records of events that occur during the period covered by the entry, the same individual may appear in the entries of different households at different periods of their life.

Through linkage, we have transformed these data into historical panel data sets that follow individuals across time and families across generations. In the CMGPD-LN, CMGPD-SC, and KMGPD-TS, we linked observations of the same individual in adjacent registers. Such linkage is straightforward in the CMGPD-LN and CMGPD-SC because households and their members are mostly listed in the same order in each register. Coders carry out linkage at the time they enter the data. In the KMGPD-TS, households do not appear in the same order in adjacent registers. We developed a process in which analytical software made a first pass and proposed candidate links based on name, calculated year of birth, and other information, and then coders adjudicated among the proposed links and created final links of their own.Footnote 29 After longitudinal linkage is complete, the software concatenates information from all the observations of an individual to produce life history information.

The NAC-SN and CTHRD had additional complexities. The transcription of the NAC-SN predated the contemporary era of database software. Individuals were first transcribed manually on time-series data sheets—called Basic Data Sheets (BDS), which were organized by household and then entered into databases. Household and individual histories were constructed based on unique household and individual identifiers (for specifics, see Ono 1993; Tsuya 2007). The transcription of the CTHRD relied on a specially designed data entry program, which allowed for dynamic linking of information about the same individual recorded on different register pages as coders entered data.Footnote 30

In the CMGPD-LN, CMGPD-SC, KMGPD-TS, and NAC-SN, the proportion of observations linked to an observation of the same individual in a subsequent register is generally high. Figure 2 displays these proportions.Footnote 31 Because out-migration for work, marriage, and even escape are specifically annotated in all five sources,Footnote 32 variations in linkage success across register pairs are largely the product of missing registers. When a register has not survived, links are made between observations in the extent registers. As discussed in detail in the upcoming section, Data Limitations and Implications for Comparability, individuals whose exits or entrances were recorded in the missing original registers appear or disappear with no explanation in the period between the two surviving registers. Longer gaps between surviving registers are associated with higher proportions of individuals whose first or final appearance was in a missing register, and who therefore cannot be linked.

Fig. 2
figure 3

Proportion of observations linked to the subsequent registers

Full size image

The annual NAC-SN and CMGPD-SC have as high as 90 % to 95 % pairwise linkage rate between registers. In the CMGPD-LN, the overall pairwise linkage rate is approximately 90 %. This percentage is especially high considering that the CMGPD-LN is based on triennial rather than annual registers, and it covers an area of more than 600 villages, which is much larger than the other data sets. In the KMGPD-TS, gaps due to missing registers are longer and more common, reducing pairwise linkage rates to sometimes as low as 2 %. However, if we consider only those linkage rates between surviving registers that are three years apart, they are 70 % to 80 %.

By concatenating the links made between pairs of registers, we reconstruct life histories for individuals. Table 2 presents the distribution of individuals according to years under observation in each data set. In the Years Under Observation column, “1” refers to individuals who are observed in only a single register and are not linked to any adjacent register; and “2–3” refers to individuals who have at least two linked observations, whether annual or triennial. CMGPD-SC has the lowest proportion of individuals who are only observed once (7.11 %), followed by the NAC-SN (13.38 %), CMGPD-LN (15.61 %), CTHRD (31.90 %), and KMGPD-TS (52.36 %). CMGPD-LN has the highest proportion of individuals who can be followed for 22 or more years: 43.57 %. In CMGPD-SC, NAC-SN, and CTHRD, the relevant percentages are 39.80, 34.69 and 23.57, respectively.

Table 2 Individuals by number of years of observation
Full size table

Linkage of individuals to their family members is based on the recorded relationship of each individual to his/her household head. When detailed relationship to the household head is recorded, it is possible to use it to identify relationships between any pair of individuals within the same household and link them with each other. Links between parents and children are especially useful because they may be cumulated across generations to reconstruct pedigrees and then identify distant kin, including those residing in other households or even other villages.

The success of such family linkage depends on the precision of the relationships recorded in the register and on whether individuals were ordered in a consistent fashion in the register. The original CMGPD-LN and CMGPD-SC registers always list wives next to their husbands and children next to their parents. They also describe individual relationship to the household head in great detail. In the NAC-SC and CTHRD, the registers also record relationship to household head with great precision so that the completeness of family linkage is comparable with the CMGPD-LN and CMGPD-SC. In contrast, the original registers of the KMGPD-TS list household members of the same generation together without further specification of their relationship to the head. As a result, although it is easy for coders and software to link wives with husbands and children with parents in the CMGPD registers, such linkage in the KMGPD-TS is much more difficult, and indeed, often impossible. For example, in a three-generation household headed by someone in the senior generation, we cannot link children with their parents and grandparents if the generation of the parents or grandparents contains more than one married couple.

Because of the time span, we can iterate through the links of children with parents to identify the distant ancestors of individuals. As shown in Table 3, the CMGPD-LN has the highest proportion of individuals who can be linked back across multiple generations: 85.46 % of individuals can be linked to the previous generation, the highest among all five data sets. Most of the unlinked individuals are in the earliest registers. They are unlinked because their father passed away before the earliest available register and was not recorded in an existing register. The CMGPD-LN is also remarkable for the number of individuals whose ancestry can be linked back for many generations: 55,243 individuals (20.76 %) can be traced back at least six generations: that is, to their great-great-great-grandfather. For individuals born in the late nineteenth or early twentieth century, the proportions are much higher. Even though the CMGPD-SC and CTHRD also have high proportions of children linked with parents, their limited time span means that multigenerational linkage is limited to four generations. Because the NAC-SN includes only two villages, linkage is limited to families that remained in the villages for multiple generations. For those families, multigenerational linkage is relatively successful. In the KMGPD-TS, multigenerational linkage is less successful, mainly because as discussed earlier, child-parent linkage is more difficult.

Table 3 Number of individuals by linked previous generations
Full size table

Data Limitations and Implications for Comparability

In the annual or triennial registers, individuals whose exit was recorded in a missing register disappear without explanation in the data set. When gaps between surviving registers are large, many individuals may disappear. In the CMGPD-LN, for example, many pre-1789 registers are missing, and all the registers between 1888 and 1903 were lost to fire. In the CMGPD-SC, only a few registers between 1866 and 1913 are missing, but because the Shuangcheng government archive was destroyed in 1865 during a local rebellion, there are no registers before 1866. Missing registers account for 74,420 unannotated individual exits from the CMGPD-LN (27.97 % of all recorded individuals) and 12,489 unannotated individual exits from the CMGPD-SC (11.61 % of all recorded individuals). In the KMGPD-TS, long gaps are especially common. We have consecutive registers for all eight myeon for only two short periods, 1729–1735 and 1780–1786. In other years, especially after 1789, many myeon are missing registers. As a result, of the 136,690 unique individuals in the KMGPD-TS, 71,823 (52.54 %) disappear with no explanation. The NAC-SN data are by far the most complete. Only a few registers—1720, 1729, 1846, 1850, 1858, and 1864–1867 for Shimomoriya; and 1742, 1758, 1796, and 1857–1858 for Niita—are missing; consequently, only 5.88 % (368) of recorded individuals disappear without annotation.

Even when there are no gaps between surviving registers, not all exits are annotated. Figure 3 compares the proportions of unexplained disappearances in such circumstances for all five data sets. The problem is the most serious in the KMGPD-TS, where 24.88 % of unannotated exits take place between surviving adjacent registers. According to Kim et al. (2013), this may reflect properties of the registration system. Individuals within the same household tended to disappear together, and the likelihood of unannotated exit was associated with social status. Unannotated disappearances between surviving adjacent registers are much less common in the other data sets. In the CMGPD-LN, CMGPD-SC, and NAC-SC, the proportions of individuals who disappear in such fashion among the total missing individuals are 4.26 %, 2.07 %, and 1.18 %, respectively.

Fig. 3
figure 4

Proportion of individuals without an exit annotation by reasons

Full size image

Although disappearances in the CMGPD-LN, CMGPD-SC, KMGPD-TS, and NAC-SN may be dealt with in a straightforward fashion, they are potentially more complex and difficult to deal with in the CTHRD. Because individuals in the annual and triennial data sets are observed at regular intervals, discrete-time event-history analysis may be used, and the observation immediately preceding the disappearance can be excluded. It is not easy to apply similar data restrictions to the CTHRD to address problems caused by disappearances because like traditional family reconstitutions and some European household registers, the CTHRD records only events and transitions. An individual who disappeared may not be distinguished from someone who remained in the household but experienced no additional events that require annotation. That said, disappearances in the CTHRD appear to be very rare. According to Li et al. (2011), at least before 1935, death registration was nearly complete. When individuals migrated out of the community, as noted earlier, the timing of the move and the destination were typically recorded.

Adult males and married and widowed women appear to have been recorded well, but in the CMGPD-LN, CMGPD-SC, and KMGPD-TS, many children were missing. Figure 4, which shows a distribution of ages recorded in the observations, summarizes the basic patterns in recording by sex. In the CMGPD-LN, CMGPD-SC, and KMGPD-TS, parents typically waited until their children were older before they recorded them, and in many cases never recorded children who died in infancy or early childhood. The CMGPD-LN and CMGPD-SC also appear to have omitted many daughters completely, while the KMGPD-TS omits many sons. In the CMGPD-LN, there are six to eight times as many observations of boys as girls. In the CMGPD-SC, there are about three times as many observations of boys as girls. In the KMGPD-TS, however, there are twice as many records of female children as there are for male children. For the CMGPD-LN and CMGPD-SC, the omission of daughters makes the study of assortative mating and female marriage timing difficult. It is difficult to trace wives back to their natal families, and the representativeness of the daughters who are recorded is unclear.

Fig. 4
figure 5

Observation pyramids of the CMGPD-LN, CMGPD-SC, CTHRD, KMGPD-TS, and NAC-SN

Full size image

Registration in the CTHRD and NAC-SN appears more complete in the sense that the sex distribution of the recorded populations by is relatively balanced.Footnote 33 The distribution of observations by age shows that the CTHRD, as a continuous record, records births and infant deaths relatively completely; while in such discrete registration systems as the CMGPD-LN, CMGPD-TS, KMGPD-TS, records of new births and infant deaths are relatively incomplete. Records of births and deaths in the NAC-SN are also incomplete, but the problems are much less serious than in the other registers.

The excess of individuals at very advanced ages that is most apparent in CMGPD-LN and KMGPD-TS reflects the failure to record the deaths of a small number of individuals. In both data sets, a small number of people who died did not have their death recorded and were carried forward from one register to the next. Before old age, there are too few cases to be of any significance. In old age, however, as everyone else dies off, these apparent immortals come to dominate the register population. Age-specific mortality rates shown in Fig. 5 confirm that the individuals who appear to have reached very advanced ages had unnaturally low death rates, presumably because they were already dead but were still being carried forward in the registers. Previous studies of mortality using these data (e.g., Campbell and Lee 2005, 2008, 2009; Dong and Lee 2014; Tsuya and Kurosu 2004, 2010) have addressed this by restricting analysis to individuals below age 75.

Fig. 5
figure 6

Predicted probability of death by next year by age (left: female; right: male)

Full size image

Meanwhile, Fig. 5 confirms that before old age, age patterns of mortality in these five data sets are broadly consistent with each other.Footnote 34 In other words, the EAP II data should be adequate for comparative analysis of patterns of differential mortality. The consistency of estimates in the CMGPD-LN, CMGPD-SC, NAC-SN, and the transformed person-year CTHRD are especially striking. From very early ages to 75, levels and patterns are similar for both males and females. The KMGPD-TS appears to have the least reliable recording of mortality. Even though estimates for females are in line with those in the other data sets, recorded mortality levels for males appear unusually low.

Marriage also appears to be recorded reliably in most of the data sets. Figure 6 presents the proportions of individuals who can be linked to a spouse. In other words, this measures the proportion of currently married individuals by age. Female marriage was early and universal. In all the studied populations, the proportion of females who are married increases very rapidly in the late teens and early 20s. By age 30, almost all females in these populations are married. In the CMGPD-LN, CMGPD-SC, and NAC-SN, the proportion is much higher than 90 %. The NAC-SN females tend to marry earlier than females in other populations. The proportion of married females identified in the CTHRD data is relatively low. Although the KMGPD-TS has much lower proportions of recorded married females, this is largely a result of the inclusion of nobi, who were less likely to be linked to a spouse. The proportions and patterns of non-nobi females with a linked spouse in the KMGPD-TS, however, look very similar to females in other data sets.

Fig. 6
figure 7

Proportion of observations with identified spouse by age (left: female; right: male)

Full size image

Male marriage was not universal. In every data set except for the KMGPD-TS, men began marrying at around age 15, and 70 % to 80 % were married by the time they were in their late 20s. Males in the NAC-SN married the earliest and in the highest proportions, probably due to their high frequency of remarriage. The KMGPD-TS has the lowest proportions recorded as married for males. Again, we suspect that this reflects underreporting. Nobi in particular were especially unlikely to have a spouse recorded, and it is unclear whether this is because they were unmarried, or married but did not report this event.

Additional available information varies across the data sets. The generational depth of the CMGPD-LN, for example, allows for measurement of the characteristics of distant kin. The CMGPD-SC records household landholding, including the location of plots and whether the land was allocated by the state or acquired separately by the household. Both the CMGPD-LN and CMGPD-SC record official position, administrative affiliation, and registered ethnicity. The KMGPD-TS records occupation and social background information for individuals as well as their mothers and fathers. The NAC-SN records many adoptions, as well as detailed information not only on household landholding but also on assets and farming animals of the household. The CTHRD also contains information on female footbinding and smallpox inoculation.Footnote 35

With the information on kinship and residence available in all data sets, we can embed individuals into a conceptual web of two dimensions: one is the relative position within the kin network, and the other is residential location and social position within the community. We can construct measures of community, household, and kinship context by aggregation of characteristics of relevant individuals. We can also construct relative measures that locate individuals within each of these units of organization. With these constructed measures, we can examine how community, kinship networks, and household context interact to shape individual behavior.

Comparison is facilitated by similarities in the cultural background of these locations. These sites are hardly identical, but they do have some features in common that distinguish them from Western populations: most notably, the emphasis on family and kinship as well as Confucian ideology. As suggested by the World Value Surveys (Inglehart and Welzel 2005, 2010), such similarities are still apparent in these societies. Given that the family is of the central importance of social organization in East Asia, individuals’ relationships are a key determinant of their standing in society and also their life chances.

By aggregating information across individuals, we can also construct measures of social and economic status at the level of the household, kin group, and community. All EAP II sources provide some level of detail on social or economic status, allowing for reconstruction of the community and kin group economic and social characteristics. Therefore, in addition to making use of the recorded statuses or occupations of individuals, we can interpret them relative to the social and economic standing of the kin group and the community. This is particularly valuable for research that compares social statuses or occupational groups on individual outcomes between different populations.

Potential for Comparative Social Science

The five EAP II data sets under discussion have proven useful for historical demography. They should also be valuable for comparative social science in general. Recent usage of similar longitudinal microdata of historical populations for Western populations shows the increasing importance of such records for social scientists not only in population studies, but also in health, economics, sociology, ecology, and other fields.

Ideally, micro-level historical data would combine longitudinal depth with spatial breadth. However, this ideal is rarely achieved because of the limited geographic coverage or survivorship of such historical sources.Footnote 36 Although census data such as IPUMS and the NAPP provide the broadest possible spatial coverage, longitudinal linkage to follow individuals across successive censuses is promising but still difficult.Footnote 37 In contrast, data transcribed from genealogy, parish register, or household register provide relatively complete longitudinal information on individuals and families, sometimes across many generations, but are available only for specific descent groups or communities and are not necessarily representative of their regional context, let alone national context.

The EAP II data are especially noteworthy because they combine longitudinal depth with geographic breath, provide the same information as the best Western longitudinal micro historical data sets, and added together roughly equal the population size of their European counterparts. All together, the five EAP II data sets cover four to eight generations residing in well over 1,000 villages or communities, with a total population size of 600,000 individuals, which is comparable with the combined population of the Scanian Economic Demographic Database (104,000 individuals); the Historical Sample of the Netherlands (78,000 individuals); the Italian Historical Population Database (17,000 individuals); the Umea Demographic Data Base POPUM (around 365,000 individuals); and the most newly released historical European intergenerational longitudinal population, the TRA database, which records 81,000 individuals. In fact, if we consider only linked life histories that allow for longitudinal/panel analysis at the individual level, the EAP II data contain even more available data than these five European data sets.

The five EAP II data sets all derive from common systems of household registration, and can potentially be standardized into identically formatted files and subjected to the same methods of analysis. In principle, it should be possible to combine these East Asian population data into a single large file, which would allow for comparisons of estimated coefficients of different populations within one statistical model rather than on statistical associations from a series of identical models on different data sets separately.

Overall, these East Asian data generate possibilities for important new comparisons on a continental scale. Although they are comparable within East Asia, they may be even more valuable as a basis for comparison with data from other societies and periods. They not only will improve our knowledge of populations in the past but also will contribute new insights into the processes that characterize contemporary populations. All such comparisons, facilitated by recent developments in micro-level “big” social science data worldwide will ultimately lead us to a better understanding of human agency and behavior in general.