Introduction

Appraisal can be broadly defined as the process of evaluating business activities to determine which records need to be created and captured, as well as how long the records should be kept (for different interpretations of appraisal, see, e.g., Frings‐Hessami and Oliver (2022)). In a narrower sense, (archival) appraisal is the process of evaluating whether records and other materials possess permanent (archival) value (Pearce-Moses 2005). In this article, we examine appraisal in the narrow sense and focus on the selection of information for permanent retention. However, we also provide some notes on the retention times of other information.

Appraisal has been described as the most difficult and challenging task in the archival profession. The issue has been a topic of discussion in archival literature since the early 1900s, when it became clear that archives needed to limit the influx of information. Archivists have developed ideas, approaches, and strategies to identify information that holds value beyond the needs of the organization that initially received or created the information as part of its functions.

Although appraisal is one of the core functions of archives, there have been few empirical studies conducted on it. In this study, we aim to fill a gap by examining the type of information that will be preserved in the archives of a Finnish municipality. The study is based on the Municipal Records Management Plan, which identifies the types of records generated or received in the functions and processes of the municipality, and defines the retention times for the information.

Literature review

The literature on appraisal is rich. It is impossible to provide a comprehensive explanation of it in a limited space. For a more extended yet still concise introduction, refer to Cook (1997). The value of records is a topic of debate in professional literature. Should one focus on the information content or information context? Who should make decisions about archival appraisal? Is the goal of appraisal to preserve evidence or construct memory? Should the usage of records in research determine their archival value? Should records of public administration be complemented with records from other sources? These are some of the questions that have been discussed in the field. For instance, one of the major figures in the history of archival thinking, Sir Hilary Jenkinson, argued that the archivist should be a neutral custodian. Consequently, the administration should make decisions about the appraisal of its records. On the other hand, another classical archival theorist, Theodore R. Schellenberg, considered appraisal to be the task of a records professional. He developed criteria for testing records’ informational and evidential value (e.g., Tschan 2002). Another well-known figure in appraisal literature is Hans Booms, who argued that societal values should define what is valuable and worth preserving (Booms 1987). Perhaps the most influential theorist in recent decades has been Terry Cook, who developed the theory of macro-appraisal (Cook 1999; 2000a, b; 2004; 2005). In macro-appraisal, the focus is on the functions that generate the records, rather than the information contained within them. Other influential approaches include the documentation strategy (Samuels 1986; 1991; 1992) and the Minnesota method (Hyry et al. 2002).

He and Tian (2018) identified seven major trends or theories in archival appraisal. These include the age determination theory, which posits that information becomes more valuable as it ages; the administrative official determination theory, which suggests that administration should be responsible for deciding about permanent retention; the function appraisal theory, which holds that decisions about retention should be based on the records creator’s position and functions in the administrative hierarchy; Theodore R. Schellenberg’s archival dual value theory; the utilization determination theory, which prioritizes historians’ actual and expected utilizations as the most important criteria; the social analysis and function appraisal theory, which maintains that an archive should reflect societal values which can be understood by examining records creators’ functions; and the literature strategy theory, which advocates for analyzing the background of the records, and Terry Cook’s macro-appraisal strategic theory. Boles and Young (1991) presented a comprehensive systematization of the factors that influence appraisal decisions. The model consists of three modules. The Value-of-Information module comprises four components: “Content Analysis”, “Functional Characteristics”, “Use”, and “Relationship to Other Documentation”. The Costs-of-Retention module includes “Acquisition”, “Preservation/Conservation”, “Reference”, “Processing”, and “Storage”. The final module, “Implications of the Selection Decision”, consists of two main components: “External Relations” and “Internal Policies”. The Boles and Young model demonstrates the potential complexity of appraisal decisions. There is no consensus on which criteria should be used for appraisal.

Appraisal has traditionally been discussed at professional forums, but academic research on appraisal has increased in recent decades. It encompasses a range of viewpoints. There have been several studies conducted on the accountability of archival appraisal (Marshall 2006), sampling methods (Buchholz 2011), how university archivists learn to appraise (Anderson 2011), the use of archives as a criterion in appraisal (Rhee 2011, 2012), the development of appraisal theory (He and Tian 2018), appraisal in South Africa (Ngoepe and Nkwe 2018), appraisal practices (Silva and Parrela 2022), archival value (Penn 2014), core terms of appraisal (Klett 2019), implications of digital collection takedown requests (Black 2020), adults’ decisions on what is worth keeping in their personal lives (Schoenebeck and Conway 2020), web archives (Summers 2020), and appraisal supporting animal rights activism (Jarvie et al. 2021). An emerging area of research in appraisal is algorithmic methods (Lee 2018; Makhlouf Shabou et al. 2020). We have been unable to find any studies analyzing records that are preserved permanently in the archives after appraisal.

Research setting

Archival legislation grants the National Archives of Finland the authority to make decisions regarding the permanent retention of records. The National Archives define which records should be permanently preserved in public administration, including those of municipalities. Public authorities are obligated by the Archives Act (831/1994) to define retention periods for their records and to have Records Management Plans (in Finnish, arkistonmuodostussuunnitelma or tiedonohjaussuunnitelma) that describe how the information generated or received in their functions is retained and managed. Identifying retention periods for ephemeral information and selecting permanently valuable records are integrated processes. The public authority creates a Records Management Plan and submits it to the National Archives for approval. The plan outlines which information should be permanently preserved in the archives, as suggested by the authority. The National Archives may either accept the suggestion or make changes to it. The plan guides records management within the authority and serves as both a disposal proposal to the National Archives and a list of information that holds permanent value.

Appraisal in Finland is a proactive process in which records are either destroyed or transferred to the National Archives in accordance with established plans. As municipalities have similar functions, the Association of Finnish Municipalities has, until recently, collaborated with the National Archives to draft general disposal schedules for these municipalities. However, legally, each municipality must negotiate with the National Archives individually (Arkistolain esiselvitystyöryhmä 2020). The current General Municipal Disposal Schedule, which consists of 15 volumes, was published in 2002. The web page of the National Archives lists (Seulontapäätökset) 38 orders, letters, and attachments that govern permanent retention in the municipalities, in addition to the schedule. The time range of these documents is 2001–2022, except for one. In 2015, the National Archives made a decision regarding the retention of records in the Helsinki City Electronic Records Management System, also known as “Ahjo”. There is no “grand theory” to guide the decisions of the National Archives regarding the permanent retention of records. Finnish appraisal is a product of various influences from multiple sources (Lybeck 2000). It combines a record-centric Schellenbergian approach with the Australian records continuum perspective (Valtonen 2015, p 98). In other words, there is a “continuum-like” understanding of the lifespan of records, and a sharp distinction between stages of the records’ lifecycle has been avoided (Lybeck 2006, p 22). The focus on records can be observed in the prevalence of Schellenberg’s value theory, which has had a significant impact on Finnish archival philosophy. One has also drawn inspiration from Germany, such as using administrative hierarchy as a preservation criterion, and from Sweden, where there is an emphasis on proactive appraisal (Henttonen 2019; Lybeck 2000). The term “semi-Jenkinsonian” refers to the National Archives’ practice of collaborating with agencies to determine which records should be permanently retained, as noted by Henttonen (2019).

The National Archives has published policy documents that describe the principles and goals of its appraisal process in 2008, 2012, and 2020. In the National Archives’ latest appraisal strategy (Kansallisarkisto 2020), the most apparent change is an increased emphasis on macro-appraisal. This refers to the evaluation of actors and functions rather than solely focusing on the information itself. However, macro-appraisal was already mentioned in the first policy document (Kansallisarkisto 2008). It can be assumed that the selection of permanently valuable information in the research data reflects the goals and principles outlined in policy documents.

A recent report on electronic records management in Finnish municipalities noted that the transition to electronic records management is still ongoing (Hänninen 2020). The City of Helsinki, with its 38,000 employees, is the largest employer in Finland. The creation of the City’s Records Management Plan, referred to as the “Electronic Records Management Classification System”, involved a wide range of personnel. Each branch of the administration had a project team that reviewed the functions and processes of the branch, identified record types, and recommended appropriate retention periods. At the final stage, the Helsinki City Archives reviewed and approved the suggestions (Helsingin kaupunki 2016, p 8).

At the heart of a Records Management Plan lies a functional classification scheme. The plan lists the organizational functions and record types that are received or generated within those functions. The plan includes process steps that establish connections between record types and functions. For each record type, there is a corresponding process step that occurs within a function. The functional classification has three or four levels, with the process name being the lowest level. An example of a process flow is when a person applies for adult education: the application (record type) is received, and they are subsequently admitted to an adult education center (process step). This takes place in the main function of Education and Cultural Administration.

Appraisal takes place at the records level, not at the level of individual files or series. Appraisal is determined by the combination of functional class and record type. The retention time is defined for each combination. Due to this, the appraisal process is fine-grained, and the retention time of a record type may vary depending on its functional context.

Unlike many English-speaking countries, records and archives management in Finland form a continuous process. The Finnish approach to records and archives management has been pragmatic and has avoided explicit theorizing. Nonetheless, many of David Bearman’s ideas align with Finnish recordkeeping practices. One of these is the integration of archives and records management (Kilkki 2004; Bearman 1995.). Conceptually, there was no difference between historical archives and current records (Orrman 2019). However, this distinction has been recently introduced in legislation (Act on Information Management in Public Administration). As a result, professional terminology does not include a concept that corresponds to the term “disposition”, which marks the end of the records management phase and signifies the transfer of records to an archive.

Records Management Plans are collaborative tools used by records managers and archivists. A record that is designated for permanent retention in the Records Management Plan becomes a part of the historical archive later. “Permanently valuable record” and “archival document” are synonymous terms in this article.

Research data

The research data for this study are based on the City of Helsinki’s Records Management Plan, which is referred to as the “Electronic Records Management Classification System” (ERMCS) on the City’s webpage. One can access the ERMCS on the internet at https://tiedonohjaus.hel.fi/ and download its contents. The application programming interface (API) for ERMCS is publicly available at https://dev.hel.ninja/projects/tiedonohjausjarjestelma/. The ERMCS contains 16,003 rows, each of which provides information about a specific record type within the system. The ERMCS provides information about the functional class (class ID and name, e.g., “09 05 01 01 Fire Safety Inspections”) and the process step (“Making safety inspection”) to which each record type (e.g., “memorandum”, “letter”, or “budget”) belongs. A record type is always associated with a class in the functional classification scheme through a process step. Additionally, there may be information regarding the stage of the process, such as “preparation and processing”, “decision-making”, or “appeal”.

The ERMCS has metadata element values for each record type that provide guidance on its publicity, storage, and other related aspects. In this study, we focus on the retention period. If the records are to be preserved permanently, the retention period is indicated as − 1. Otherwise, it is the length of time the records will be retained in years, for example, 10. If a record needs to be preserved for its period of validity or for the present, it is marked as 0. However, a weakness of the research data is that a record type may repeat itself multiple times throughout the process. The repetition may indicate that the same record type is related to more than one process step, in which case the information about the record type is repeated in the ERMCS. Alternatively, it may indicate that there are copies or manifestations of the record type, such as drafts or final versions, each with their own retention times. It is not possible to identify duplicated values with absolute certainty, but their occurrence is estimated to be between 2.5% and 7%. In principle, this may introduce some bias in the results, but there is no reason to believe that repetition favors certain types of records or retention periods.

In summary, the ERMCS provides insight into the functions and processes that generate information with enduring archival value. One can also observe the number of record types that hold permanent value, and therefore, the relative amount of information that is permanent compared to other classes and processes. The ERMCS does not provide information regarding the quantity of records. Thus, the ERMCS does not reveal the amount of information that needs to be archived. A class that contains mostly record types that possess permanent value may have less volume in the archives compared to a class where most record types are destroyed.

The research data (ERMCS) were complemented with other sources: the information governance manual of the city, the decision of the National Archives about retention times in the Helsinki City Electronic Records Management System, and discussions with the City information and records management specialist. Research questions are:

  1. 1.

    Which functions and processes in the city administration generate records that are assessed as permanently valuable?

  2. 2.

    Which factors govern how appraisal is conducted in reality?

Research method

The downloaded Excel file (ERMCS) included 18,874 rows. The original ERMCS is structured hierarchically. After exporting the data to a flattened spreadsheet, we removed repetitive information, which included upper-level data that were repeated at lower levels, as well as rows that only contained class names from the functional hierarchy. This left 16,003 rows. The data were imported into SPSS software for statistical analysis.

Next, the data were processed to enable calculations. The class hierarchy of the ERMCS consists of three or four levels. The code for the lowest level class was divided into smaller sections. For instance, from the level three code “09 05 01”, two new columns were generated with values “09” (level one) and “09 05” (level two). This made it possible to analyze data at higher levels of classification.

Finally, we generated tables, graphs, and lists of values from the data. This was primarily accomplished using Python scripts, supplemented by SPSS and Excel as needed.

Findings

The Helsinki City ERMCS identifies fifteen main functions (Table 1), 705 processes, and 16,003 process steps taken in these functions. Every process step is associated with a record type.

Table 1 Functional classes and number of processes, process steps and record types
Full size table

On average, one main function consists of approximately 50 processes and 1,100 process steps. However, there is significant variation. The smallest function has only 86 steps, while the largest has over 3300.

The ERMCS covers all records of the City, regardless of their form, in principle. The ERMCS does not make a clear distinction between electronic records and other types of records. However, the metadata element “record.informationSystem” indirectly reveals information about the nature of the records. The element contains a free-text value that lists information systems related to the record type, such as “Paper, ATJ, YPH, EFFICA”. In 8% of the cases, only one “system” is mentioned, which is “paper”. However, in 79% of the cases, “paper” is not referenced at all. Out of the total records, 13% are ambiguous as the element comprises multiple systems, including “paper”. This could imply that the record exists in different forms simultaneously, or that its form changes over time, or that a manual record is being monitored in an electronic system. Thus, either the lifespan of records, the recordkeeping process, or both are hybrid. Nevertheless, fewer than one-tenth of the record types exist only on paper, and the transition from paper to electronic information management has largely occurred.

The system portfolio of Helsinki City (dated 8th of March 2023) lists about 1100 applications, web pages, and other information systems that are used in the city administration. The values of the metadata element record.InformationSystem in the ERMCS contain almost 600 terms that denote information systems in which the records reside. Identifying matches between the values in the ERMCS and the systems in the system portfolio are not always possible, but from the length of the two lists alone one can conclude that the ERMCS covers only part of the city’s system portfolio, and many information systems are not in the scope of the records management planning.

All functional classes leave some information in the archives. In general, 20–50% of the record types within a functional class have been designated as having permanent value (see Fig. 1). The Administrative Affairs class 00 contains the highest proportion of permanently valuable record types, except for the smallest function (04 International Activities and Immigration Policy). Class 00 includes decision-making at the highest-level of the municipality, which involves the Mayor, City Council, and City Government, among other things. The highest-level decision-making covers all areas of activity in the City, which makes it a pseudo-function. Next, we have functions 10 Land use, construction, and housing, and 11 Environmental affairs.

Fig. 1
figure 1

Percentage of permanently valuable record types by functional class

Full size image

The ERMCS allows for the linking of process steps to an administrative process stage, but it is not mandatory. There are eight administrative stages: “Counseling and Guidance”, “Initiation and Commencement of Proceedings”, “Preparation and Processing”, “Decision-making”, “Notification”, “Implementation”, “Monitoring and Control”, and “Appeal”. More than four-fifths (84%) of the processes include at least one step that is associated with an administrative stage. The administrative stage that creates the largest number of record types (38%) is “Preparation and Processing”. Perhaps surprisingly, 10: Land use, construction, and housing are the function that creates one-fifth of all record types (Table 2).

Table 2 Number of process steps and record types by function and administrative process stage
Full size table

Processes without any linkage to an administrative stage usually (87%) belong to function 05 Social Services, in which service-oriented processes dominate.

According to Table 3, the decision-making process has the highest portion of permanently valuable record types (83%). This is more than double the amount that remains in the archives from the other stages. Next comes preparation and processing, during which an average of 36% of record types are retained permanently. This is followed by implementation, which retains 26% of record types. There are significant differences among various functions. For instance, in function 09 Security and Public Order, 97% of record types at the appeal stage are preserved, in contrast to only 9% in function 01 Personnel.

Table 3 Percentage of permanently valuable record types by function and administrative process stage
Full size table

The ERMCS identifies a record type for each of the approximately 16,000 process steps. There are a total of 93 record types. The ten most common types cover approximately 60% of the steps:

Attachment

2314 (13.9%)

Request

2016 (12.1%)

Announcement

1233 (7.4%)

Note

784 (4.7%)

Decision

713 (4.3%)

Plan

615 (3.7%)

Statement

612 (3.7%)

Report

562 (3.4%)

Minutes

556 (3.3%)

On average, a record type has six retention periods. In addition to permanent retention and one undefined retention period (“period of validity/for the present”), the ERMCS specifies seventeen retention times that range from one year to 120 years. However, most of the categories for retention period are rarely used (Table 4).

Table 4 Distribution of retention periods
Full size table

Some retention period categories are heavily utilized while others are rarely used. This manifests itself in many ways. In 76% of cases, the retention period is either permanent (39%) or ten years (37%). If a record type is preserved for 20 years or less, it usually has a retention time of ten years (72%). In records that are preserved for over twenty years but not permanently, the most common retention time is 30 years (75%). Interestingly, 90% of the information in the latter group comes from one function (05 Social Services).

After 20 years, 88% of ephemeral record types with defined retention times have been destroyed.

Discussion

When considering the findings, it is important to note that in a semi-Jenkinsonian Finnish environment, both records management and archives management play a significant role in determining permanent retention. If the appraisal process were solely in the hands of records managers or archivists, the results might differ.

How to select information that is permanently valuable has been a central theme in archival literature. Various criteria and approaches have been suggested, but based on the findings, it appears that appraisal decisions typically follow a pattern. While ephemeral information may lose its value in twenty years or less, records pertaining to permanent or long-standing objects and phenomena, or those with long-lasting consequences, are selected for long-term preservation. In addition to high-level decision-making records, information regarding the environment, constructions, land use, and people (excluding information about changing personnel) are more likely to have extended retention periods and may even be preserved permanently. This perspective has not received attention in archival literature.

Defining retention times for each record type in every function individually is laborious. There is no clear correlation between a record type and its retention period. When all functional and process contexts are taken into account, each record type has more than one retention period.

The findings show that while there are nearly twenty possible categories for retention time, records are typically kept permanently, for the duration of their validity, or for 10 or 30 years. According to legislation, social service records must be retained for either 30 years or 12 years (Act on the Processing of Social and Healthcare Client Information). Instead of focusing on every detail, an alternative approach using broader retention time categories as suggested by Cisco (2008), may also be effective for appraisal purposes. It may be more efficient to evaluate series or files rather than record types.

The transition from paper to electronic information management has largely taken place, but it is not yet complete. There are still paper records and hybrid processes. The same information can exist in multiple information systems or be related to them. “Information sprawl” or “digital sprawl” (Mapstone 2021) is a phenomenon where information exists in various forms and systems simultaneously, hindering its effective management. In principle, a Records Management Plan can provide clear guidance on how to manage a record but implementing it across the organization can be a daunting task.

Traditionally, the goal of appraisal has been to preserve 15–20% of the public sector records in Finland permanently (Rastas 1994, p 242). This percentage is considerably higher than in some other countries. For example, only about two percent of government records reach The National Archives in London (Rock 2016). In the United States, only 1–3% of Federal records are kept permanently (National Archives 2016). On the other hand, there are also opposite examples. In Iceland, all public records are deposited with the National Archives thirty years after their creation (Gunnlaugsdóttir 2006; Haraldsdóttir 2019, p 23). In an electronic environment, no quantitative goal has been established in Finland. Based on the percentage of record types that are considered permanently valuable (39%), it is possible that more information is being preserved permanently today than in the past. However, it is important to note that the actual volume of archived records cannot be determined solely from the number of record types. In addition, if the Records Management Plan does not include all the information (see below), the percentage of permanently valuable record types is lower.

While a Records Management Plan should encompass all information received or created in organizational functions, less than half of the City’s system portfolio was explicitly referred to in the Records Management Plan. This could indicate that some electronic information is frequently excluded from records management planning, although the planning should encompass all the information that is received or created in the organizational functions. Therefore, it would be vital to understand what information is excluded, why, and what implications this has for archival appraisal.

The City’s records management officer believes that the percentage of permanently valuable record types may be high due to gaps in the ERMCS when it comes to managing ephemeral records. This may also explain the high number of missing information systems in the ERMCS. The missing systems may only contain very ephemeral records.

The comment made by the records management officer highlights a clear limitation of the study. The research data itself provide the only window into the information, processes, and functions of the city. It is impossible to challenge the perspective presented by the ERMCS. One cannot inquire about the information that has been disregarded or challenge the validity of the decisions regarding permanent retention. The number of processes in the ERMCS (705) may seem low for an administration of this size and broad responsibilities. However, it is important to note that the process definitions are specifically designed to serve records management and may be adequate for that purpose. These issues require further study.

Conclusions

Although the results do not provide a clear rationale for selecting records for permanent retention, they do suggest that records pertaining to long-lasting phenomena are more likely to fall into this category. It is also evident that certain retention periods are preferred over others. Hence, it appears that there are rules, whether conscious or unconscious, that are followed during the appraisal process.

Thus, the study raises the question of how decisions about archival appraisal are reached. From archival literature, one may infer that appraisal is a theory-guided activity in which abstract ideas play a significant role. This study suggests an alternative view: appraisal is mostly guided by concrete and practical factors. However, the role that archival theory and other factors play cannot be understood without additional research.

Unfortunately, Finnish archival appraisal is a “black box”: what happens in discussions between the National Archives and records creators (public authorities that create Records Management Plans) is not documented, nor have there been studies about the appraisal process. The strategy and policy documents of the National Archives outline the broad principles that guide appraisal at national level. The documents are publicly available, but anecdotal evidence suggests that theoretical considerations play a small role when authorities draft Records Management Plans: besides legislation, practitioners tend to look at what retention times have been assigned to similar records in the past. The National Archives, on the other hand, takes a bird’s eye view when it comes to appraisal, being more conscious of the theoretical perspective. However, it is difficult to determine how appraisal theory is reflected in its individual decisions. The perceived value of information may depend on the level of detail provided about the functions and processes, as well as the knowledge possessed by the appraisers. Because of digitalization, technical considerations and the costs of disposal alternatives may also affect decisions. It may not be technically feasible or cost-effective to preserve information that has long-term value. There are no guidelines for resolving conflicts that may arise in an appraisal process.

Finnish appraisal might benefit from a more transparent appraisal process. Recently, there have been steps in this direction: in addition to the already publicly available appraisal policy documents of the National Archives, the National Archives has begun to ask for public comments about its general disposal schedules. However, both professionally and theoretically, it would be important to understand what reasoning is behind individual retention times that are assigned to records.

There are also societal factors to consider. Appraisal is a social process. National policy documents are created by one group of people, while general disposal schedules for municipalities are created by another group, and the disposal schedule for each individual municipality is created by a third group. At every stage, there is discussion and interpretation. At every level, the significance and meaning of the records are negotiated. Appraisal theory can alter the perception of value and, consequently, influence the selection of items for long-term preservation. It may also streamline the process by directing appraisers to ask the appropriate questions, thereby recommending the most effective and efficient approach. There is a lot that we do not know.