Keywords

1 Introduction

Performance at Transportation Security Administration (TSA) airport checkpoints must be consistent, efficient, and exception to skillfully alleviate and ultimately prevent national security threats and incidents. Security checkpoints are primarily maintained by Transportation Security Officers (TSOs). They are responsible for providing exceptional performance in three primary areas: consistent vigilance of potential threats, ethics to ensure that travelers are respected as their personal property is searched, and efficiency to handle the volume of people who go through each checkpoint. Although high performing TSOs are critical for the TSA to meet its corporate vision and national security goals, it is difficult to measure the human attributes that contribute to performance. Attributes such as cognitive ability (e.g., memory, critical thinking, and competence), personality (e.g., conscientiousness), and social skills (e.g., command presence, leadership) are latent variables that are not easily observed, measured, or quantified. Moreover, performance is a challenging variable to measure because it is difficult to quantify threat detection, travelers’ reactions, and efficiency in a dynamic operational environment like the airport security checkpoint.

Cognitive scientists at Sandia National Laboratories have developed a methodology to quantitatively link TSOs’ cognitive ability to their performance. The scope of this project was to identify attributes that distinguished high and low TSO performance for the duties at the checkpoint that involved direct interaction with people going through the checkpoint. This paper first discusses how the methodology was developed using a strict empirical process. For proprietary purposes, the fine details and results are not presented. We then discuss some of the strengths and weaknesses of the experimental design. Finally, a set of ideas for how this could be generalized to other non-TSA contexts are offered.

The TSOs’ job duties, specifically those interacting with passengers at the checkpoint, are challenging and require specific traits, skills, and attributes. Literature has supported the relationship between vocational interests and performance. Researchers conducted a meta-analysis, providing a quantitative summary of over 60 years of research [1]. Their meta-analysis indicates that vocational interests can be as predictive of job performance, tenure, and citizenship behaviors as can personality measures. It was concluded that it is important for an organization to understand the interest profile of the jobs for which they are hiring. Personality has also been linked to job performance. Another meta-analysis explored the predictive ability of personality inventories based on the Big Five personality traits for job performance. It was found that conscientiousness, agreeableness, and openness to experience were all positively correlated with performance.

There has been no study to date that investigates the knowledge, skills, and abilities (KSAs) and attitudes, aptitudes, and attributes (AAAs) that TSOs should have to optimally perform. Findings from a study of this nature would be highly valuable because it would inform how to manage TSOs at the checkpoint and may even influence how TSA hires personnel. The overall goal is to investigate how a TSO’s KSAs and AAAs may distinguish high and low performance.

Ultimately, the TSA headquarters research team is highly interested in moving past basic employee requirements and desires to better understand the cognitive and psychosocial factors that may help to improve TSO performance. The current study, designed and executed by the Sandia researchers, aims to examine the following questions:

  • How valid are measures shown to successfully predict TSO performance?

  • Are we able to use a battery of cognitive assessments to help TSA identify high performing TSOs?

2 Experimental Design

The overall goal of this study is to identify relationships between predictor variables and performance variables. Sandia researchers first started with identifying the performance variables in the TSA context. Through iterative conversations and trips to TSA locations, a number of performance variables were identified. These included, to name a few, annual test scores, human resources data such as attendance and awards, and performance review ratings. Note that the focus was on numerical data. Qualitative data such as supervisor comments and customer feedback were identified but were not used in this study due to their subjective nature. The performance data would ultimately need to be connected to the data we collected from TSOs so it was equally important to understand the identifiers in the performance dataset(s) in order to create a solid link between our collected data (predictor variables) to the TSA performance variables.

Once we established TSA performance metrics that were eligible for quantitative analyses as well as the needed identifiers, we turned our attention to the predictor variables. We classified predictor variables into two categories: (1) competencies and critical knowledge, skills, and abilities (KSAs), and (2) attributes, aptitudes, and attitudes (AAAs). Competencies and KSAs are defined in industrial-organization (I/O) psychology: “A competency refers to an individual’s demonstrated knowledge, skills, or abilities [3]. Note, however, that competencies go beyond the more traditional KSAs; they are KSAs that are demonstrated in a job context influenced by the organizational culture and business environment. for the job of interest [4]. These are usually defined by the organization, in this case, TSA. The next category of predictor variables, AAAs, is a concept defined by Sandia and TSA as the cognitive, social, and personality skills that can be measured and quantified and represent the latent, non-tangible KSAs. The process for identifying KSAs and AAAs is presented.

2.1 Knowledges, Skills, and Abilities (KSAs)

To identify KSAs, we used the standard I/O method, which is to utilize job task analyses. Commonly accepted best practices and industry standards call for a job analysis to be completed prior to modifying or developing job candidate selection criteria [4]. This process also fulfills the legal requirements in accord with the Uniform Guidelines on Employee Selection [5]. A Job Task Analysis (JTA) is the process of defining all the elements and work activities required for successful performance of a job and serves as the foundation for determining which assessments are appropriate for administration on TSOs. This includes all job duties, roles, and job tasks, as well as the Equipment, Machines, Tools and Technology (EMTTs) used to accomplish each job task. It also identifies at the employee level all of the knowledge, skills, abilities and other variables (KSA-Os) necessary to successfully complete each job task. Finally, a task analysis can come out of a job analysis providing a detailed explanation each job task’s relative importance, frequency of completion, and criticality. This type of task analysis is similar to the DIF (difficulty, importance, and frequency) analysis commonly utilized by HR and Training & Development departments. Fortunately, TSA’s Human Capital program had already conducted multiple JTAs which Sandia was able to leverage. However, to specifically address the aims and hypotheses we are interested in, which is connecting TSOs performance at the checkpoint, specifically related to TSOs interaction with the passengers, additional work was needed to connect the different JTAs.

Workshops using subject matter experts (SMEs) were organized. The workshops were used to get a group of SMEs to vote on the most critical KSAs and competencies TSA valued for each job task at the checkpoint. Multiple workshops were held across a variety of airports to get a diversity of opinions. After all the SME workshops were completed, the data was aggregated and analyzed until a SME-based ranking of valued KSAs and competencies was developed. The rankings represented what SMEs collectively reported as the most critical and influential KSAs for high performance on passenger-related (non-X-ray) job tasks.

2.2 Attitudes, Attributes, and Aptitudes (AAAs)

This ranked list of KSAs and competencies created categories that then enabled the Sandia researchers to link to AAAs. The difference between the KSAs and AAAs was that the KSAs were specific to the TSA tasks and duties whereas the AAAs were overarching group of attributes connected to self-report measures or tasks found in the empirical open literature. For example, command presence, independence, accountability, and assertiveness were ranked by SMEs to be important KSAs and competencies related to a number of passenger related tasks at the checkpoint. For the AAAs, these attributes were grouped into one category, “Strong Presence” and quantified using difference subscales from the Personality and Preference Inventory [5]. The purpose of grouping similar AAAs into a single category was to limit the number of hypotheses tested and therefore not inflate alpha levels during statistical analyses. Once the AAA categories were established, a “priority score” was calculated for each group. The priority score number represented how many times six or more SMEs ranked one of the KSAs as an essential ingredient for completing a critical task listed in the JTA.

Overall, the KSAs enabled us to understand what the most important AAAs were to the TSA environment and create a battery of cognitive, social, and personality measures that were either established in open, empirical literature or developed specifically for this study. Established measures in the battery were found to be validated and reliable with high internal consistency; those that were developed will be checked for their internal validity before use in data analyses.

2.3 Participants

This study aimed to collect data from a minimum of 200 existing TSOs across a minimum of eight TSA airports. A power analysis was conducted prior to data collection an estimated 200 TSOs to achieve a power of at least .80. Employees who participated in this study were required to meet the following criteria prior to data collection:

  1. 1.

    Currently hold the job title of TSO, Lead TSO (LTSO), or Supervisor TSO (STSO)

  2. 2.

    They must have had the job as an officer for over one year

  3. 3.

    They must be either passenger (PAX) or DUAL certified

  4. 4.

    Be at least 18 years old

2.4 Data Collection

Once the battery of measures was compiled, the study design went through a human subjects’ board (HSB) approval process at Sandia Labs and TSA Headquarters to obtain institutional review and approval. Then, permissions were obtained for the use of each measure. Finally, each measure was loaded onto SurveyMonkey. SurveyMonkey is an online platform designed for web-based data collection. Cellular connected tablets (version 6) were deployed to each airport so that participants could complete the online battery of assessments. Although this led to limited connectivity and data load time issues, the use of tablets was a safer method of protecting participant data because it utilized a cellular internet connection rather than a less protected network such as Wi-Fi.

Proctors were trained at each airport so that the Sandia researchers would only have to be onsite at each airport for 2 to 3 days but data collection could continue for up to two weeks to obtain a large sample from each airport. When participants arrived, a TSA research proctor trained by the Sandia researchers distributed all of the materials (an iPad and a red folder containing a consent form, a checklist, and participant instructions). The proctor directed the participants to read over an HSB stamped and approved informed consent form that detailed the study process and metrics. It stated that participation was voluntary and withdrawal was permitted at any time. If the TSO decided to participate in the study, they signed and dated the consent form and returned it to the proctor. All participants were offered a copy of the informed consent for their records.

Once all signed informed consent documents were collected, the proctor read a formal script addressing the participants, providing instructions on how to begin the battery of assessments, and offering to answer questions or provide other help. Once the script was concluded, the proctor provided additional help to the group (and one-on-one as needed) to help the participants create their unique identifiers and to start the first assessment. Proctors walked participants through the process of how to complete the online assessment on the tablets. The whole battery took approximately 4 to 6 h to complete. The battery was split into seven blocks to encourage breaks and help participants keep their place in the battery. Because participants completed the assessments at their own pace, breaks were taken individually, not as a group. To minimize the effect of cognitive fatigue on the final results the most important assessments were placed earlier in the battery order.

3 Data Analyses

Once data collection is completed at all the airports, all the raw data will be compiled and screened prior to analyses. The screening process will allow the researchers to check for any inaccuracies or issues that may have occurred during data collection. Data collected will also be scored according to the calculations instructed for the subscales prescribed each measure.

3.1 Data Characteristics

Data will be checked for missing data, assumptions of normality of distributions, linearity, homogeneity of variance, homogeneity of regression, and reliability of covariates. Each measure has been validated in the public domain. However, since TSOs is a new sample demographic for many of the measures used in this study, factor analyses will also be conducted for each measure and its corresponding subscales to examine internal consistency and construct validity.

3.2 Hypothesis Testing

Averages for each participant’s AAA subscales will be correlated with TSA performance data. It is expected that positive AAAs such as empathy and perspective taking, command presence, and emotional intelligence will positively relate to higher levels of performance. It is also expected that negative AAAs such as indecisiveness, the dark triad of personality, and touch avoidance will relate positively relate to lower levels of performance. Many of the measures have multiple subscales which will also be used to test hypotheses. Statistical analyses such as a profile analysis will be conducted to understand how AAA responses differ based on TSA performance ratings.

3.3 Demographics

Demographic data on age, gender, years of TSO experience, TSO level (TSO, LTSO, STSO) past military experience, education level, entry data, time in TSO position, airport/area, veteran (type) were also collected. Exploratory analyses will be conducted to how both AAAs and performance relate to demographics.

4 Strengths and Weaknesses

This methodology has enabled us to collect data across multiple airports and have large participation rates from a number of TSOs. The major strength is that this study provides a method to take latent variables important to the TSA checkpoint, such as command presence, visual acuity, and intuition, and quantify them so the relationship they have with performance variables can be statistically tested. This enables TSA to make rational, informed decisions on how to structure TSOs’ roles and responsibilities at the checkpoint and overall nurture high performance. Another strength is the portability of data collection. Through the deployment of tablets, the utilization of online assessments, and the partnership with TSA headquarters, data was able to be collected across airports to obtain a diverse sample. The large sample size will help both with looking at group differences within the sample (for example, low versus high levels of experience) and for increasing effect sizes to determine if statistical findings are robust.

One major weakness of this study is the limited metric we have of performance. Ideally, performance would be determined by comparing the number of threats identified with the number of threats unidentified. This ground truth is nearly impossible to obtain. Our performance variables were thus limited to how TSA as an employer rates its employees through annual performance review and employee behaviors such as attendance. Data included in statistical models were also required to be quantitative and on a numerical scale. We did not include qualitative data. Future research could utilize techniques such as text analytics to obtain data on TSOs’ cognitive processes and performance. For example, if performance records have comments for each TSO on file, the comments could be analyzed using text-based algorithms. One final weakness is the dynamic nature of TSA as a growing, maturing organization. The AAAs were based on the JTAs that listed out the tasks critical to the TSO job. The TSO job and its related standards of practice are occasionally updated to accommodate the changing security checkpoint environment (such as the introduction of the electronic boarding passengers and new screening equipment Research in this area should be updated every few years to maintain the relevance of its findings to the airport security checkpoint environment.

5 Additional Contexts

A major benefit of this research is the insight it provides to the Transportation Security Administration. However, this methodology contains elements that are applicable to many organizations. Most organizations have a set of performance metrics used to understand where employees are on a performance spectrum. Job Task Analyses and the use of I/O psychology in business settings are becoming a more widely used practice. Already existing JTAs can be leveraged or JTAs can be developed from scratch by I/O psychologists to understand what the critical KSAs are. At a minimum, SME workshops can be held to identify and rank the most important tasks and their respective KSAs/competencies. Once a list of predictor variables and performance variables as well as the identifiers that link the two have been established, researchers can utilize the open, empirical literature to locate measures that will quantify the predictor variables. The Sandia researchers have used similar processes to research domains to expand research into other relevant areas of TSA. These methodological principles have also been applied by Sandia researchers to different domains such as cyber security, the impact of the working environment on creativity and collaboration, analysts in military contexts, and other national security contexts.

6 Summary and Conclusion

In summary, this study applies a strict, quantitative methodology to empirically understand the relationship between AAAs and performance in the TSA security checkpoint context. The methodology we developed enables us to investigate how latent, human dimension variables such as memory, personality, and competence specific to a target population’s roles and responsibilities, such as TSOs, influence performance in that domain. This methodology is beneficial but not limited to TSA. Researchers that have access to performance metrics and Job Task Analyses could utilize a comparable methodology to accomplish similar goals.