Our lives are increasingly mediated through devices, with an estimated 6 billion smartphone users globally today (Statista, 2021b). Recent reports have suggested that individuals across different countries are spending an average of 4.2 h a day using mobile apps (Kristianto, 2021). The smartphone is also increasingly personalized and serves as a nexus where different facets of people’s lives converge. The variety of potential uses for smartphones is astounding, with the top apps across different countries showing great diversity, ranging from social media and messaging (e.g., Signal, TikTok, etc.) to productivity and finance (e.g., Microsoft Teams, Robinhood, etc.) (Kristianto, 2021). The centrality of smartphones in everyone’s life means that there is increasing interest from scholars across a wide variety of disciplines in understanding how people use it, what they use it for, and what kinds of effects different types of uses lead to (Meeus et al., 2020; Schneider et al., 2022; Twenge, 2019).

Despite scholarly interest, and the increasing diversity of use afforded by smartphones, measurements of smartphone use have often relied on traditional techniques such as surveys to make inferences about its uses and effects (de Vreese & Neijens, 2016). As Brinberg et al. (2021) argued, commonly used measures of digital media use such as self-reports and application logging – while useful for certain purposes – do not provide a robust account of the idiosyncratic nature of device use. Study participants vary in their ability to recall, commitment to the study, and consistency of data recording.

To capture digital device use more systematically and accurately, Reeves et al. (2021) pioneered an innovative approach called screenomics. The digital screenome is an in-depth record of users’ interactions with screen devices – such as the smartphone. Each screenome comprises a sequence of users’ screen interactions and records them in the form of high-frequency screenshots. This allows researchers to obtain data of individuals’ device use over a period in their original forms, resulting in an objective and comprehensive account of their device use suitable for both qualitative and quantitative analysis.

While Reeves et al. (2021) established foundational principles behind the screenomics approach, with invaluable suggestions for how a social scientist may establish their own workflow, there is currently no readily available software researchers can use to start collecting such data (Kaye et al., 2020). Furthermore, despite the value of screenomes, many social science scholars are not trained in software development and programming and may lack the technical skills required to develop such a data collection tool. Previous screenomics research has also been primarily conducted by Reeves and colleagues, and raising the accessibility of using such an approach can allow other scholars to replicate or refine extant studies.

To fill this methodological gap, we introduce ScreenLife Capture – an open-source and user-friendly Android application to capture screenomes – for scholars interested in the study of everyday digital life. We further document the results from an exploratory study with a sample of participants to understand users’ experience under such constant background surveillance and present an initial analysis of the data collected. Finally, we discuss privacy concerns which ought to be considered when utilizing such an approach to data collection.

What is the screenomics approach and why use it?

First proposed by Reeves et al. (2021), the screenomics framework consists of several key components. On its most basic level, the screenome is a sequence of screenshots collected continuously from an individual’s device. Reeves et al.’s (2021) data collection process starts with loading a screenshot capturing application onto a participant’s device. The application records screenshots at regular intervals (e.g., every 1, 5, or 10 s), stores them on each participant’s device, encrypts them in a folder, then transmits the data to secure servers. The screenshots then undergo pre-processing, automated information extraction and classification, before being loaded onto a database and analyzed.

This approach has already been utilized to examine different research questions such as task switching, how incidental exposure of news drives intentional information-seeking, within- and between-person differences in engaging with media content, and adolescents’ digital lives (Brinberg et al., 2021; Ram et al., 2020; Reeves et al., 2021). Existing studies have demonstrated the potential for the approach to provide unique insights into individuals’ smartphone use that were previously not possible, which can encourage the generation and testing of theories in different ways (Lee & Yee, 2020; Ram et al., 2020).

The screenomics approach can broadly be classified as a type of ambulatory assessment, which refers to research methods aimed at gathering ecologically valid and continuous data characterized by assessing people in real-world environments (Trull & Ebner-Priemer, 2014). As such, it shares many similar types of advantages when compared to traditional measures of media use. First, screenomes allow for the study of individuals’ media use in their everyday, natural environment, where it is intertwined with various environmental, interpersonal, and intrapersonal forces that are part of people’s lives but are difficult to replicate in a laboratory.

Second, retrospective self-reports of media use might not always be accurate, occasionally suffering from recall bias which leads to the over- or underreporting of media use (Scharkow, 2016). Previous research has highlighted the discrepancies between self-report and objective measures of media use, with a recent meta-analysis showing that the aggregate relationship between self-reported and logged media use measures over 44 studies to be moderate (Parry et al., 2021). In contrast, screenomes provide an objective and in situ measure of participants device use.

Third, researchers have sometimes used media diaries to obtain an account of people’s media use over time (Lim, 2009). In diary studies, participants are asked to record their media use in the form of diary entries over a period. Unfortunately, diary studies can suffer from recall bias, social desirability bias, and non-compliance (Trull & Ebner-Priemer, 2013). Screenomes offer a solution to this as the use of an automated screen capture application means less effort is required from participants and the data are recorded without their mediation that would introduce a degree of subjectivity.

Finally, there is an increasing number of scholars who utilize digital traces to examine media use (Peng et al., 2020). This might include collecting individuals’ social media activity using various application programming interfaces (APIs) or obtaining logs of mobile application use to extrapolate patterns of media use by participants (de Vreese & Neijens, 2016; Jones-Jang et al., 2020; Peng & Zhu, 2020). While it is possible to construct a sequential account of mobile phone use with different forms of digital traces, a considerable amount of effort is needed to manage and compile data across different platforms (Reeves et al., 2021).

Mobile application logfiles could also mask important content-related person-screen interactions which are of interest to researchers. Previous research has highlighted that low socioeconomic status (SES) individuals might exploit media platforms in different ways to high-SES individuals (Cho et al., 2003), with high-SES individuals potentially using social media to enhance social capital while low-SES individuals use it more for consumptive purposes (Micheli, 2016). Screenomes, however, provide both content-related and contextual information required to test such hypotheses. For example, screenomes enable researchers to map the specific content being encountered through smartphones across time (Brinberg et al., 2021).

This allows researchers to parse out specific smartphone related behaviors (such as application switching; Deng et al., 2019), content-based media effects (such as in examining the impact of meaningful media experiences; Janicke-Bowles et al., 2022; Krämer et al., 2021; Oliver, 2022; Oliver et al., 2016), and exploring digitally mediated social interactions (Ryan et al., 2017), among others. There have been increasing calls for research on digital media effects to be more content- and context-sensitive (vanden Abeele, 2021; vanden Abeele et al., 2022), and screenomes provide such an opportunity.

Beyond that, the convergence of different types of use into single mobile applications (apps), means that logfiles might not accurately capture the diversity of behaviors which can occur in one application. For example, Uber has over the years integrated food and grocery delivery into its app, following in the footsteps of other “super apps” such as Gojek in Indonesia, which provides a range of services from laundry and pharmacy services to banking and video streaming (McGee, 2019). More recently, Netflix has signaled plans to offer video games on their platform (Shaw & Gurman, 2021). Screenomes offer a clearer account of specific behaviors which cannot be obtained from logfiles alone.

Given the methodological strengths and potential theoretical contributions which can be developed from such an approach, it is important that an open-source, free, and accessible tool be made available to researchers. There are two main reasons why we believe our tool – ScreenLife Capture – is an important contribution to the field. First, as mentioned earlier, there are currently no existing programs readily available to researchers to capture screenomes on smartphones. Existing screenome studies have been based solely on the Stanford Screenomics application which is not freely available for use by other researchers (Brinberg et al., 2021; Ram et al., 2020; Reeves et al., 2021).

Second, while Reeves et al. (2021) provided an initial framework for collecting screenomes, it is not available for widespread use, since setting up the entire system requires substantial software development and higher-level programming skills. On the other hand, ScreenLife Capture and its accompanying participant onboarding and data management software (DMPO) offers a largely point-and-click interface with minimal coding. This means that researchers can download the program and quickly start data collection.

ScreenLife Capture

ScreenLife Capture is an open-source Android application which comes with a companion DMPO software. Figure 1 provides an overarching framework of the workflow and how screenshots are captured, transferred, and processed before analysis. As detailed below, there are three main stages to the workflow process of collecting screenomes from smartphones: (1) participant onboarding, screenshot collection, and encryption, (2) transfer and cloud storage of encrypted screenshots, and (3) decryption and data processing.

Fig. 1
figure 1

Overarching workflow

Full size image

Step 1: Participant onboarding, screenshot collection, and encryption

To begin the participant onboarding process, a researcher must download and boot up the DMPO on a research computer. At the start of a new project, the DMPO will prompt the creation of a passphrase. The passphrase is used to authenticate the researcher and allow access to the project. It must be securely kept and remembered by the researcher. Once the passphrase has been created, researchers can start onboarding participants for their study.

Next, a user installs the front-end Android application on their personal or research smartphone. To enroll participants in the study, the researcher must use the DMPO’s “onboard participant” function and key in a unique participant ID for the user. Once a participant ID has been created, the DMPO automatically generates a unique Quick Response (QR) code. Using the ScreenLife Capture front-end application, the user must then scan that QR code to complete their enrolment into the study. This results in a randomly generated encryption key being provided to the device’s secure element. This key is used to encrypt screenshots collected through the application. As the QR code is generated on the researchers’ computer, participants can be enrolled remotely through videoconferencing or in-person at a research lab. Once onboarded, participants can select the option to start the recording of screenshots via the application.

When the ScreenLife Capture application is active, it captures screenshots at researcher-defined intervals using the MediaProjection API, with the default interval set at every 5 seconds. To respect the autonomy of participants, users have the option to suspend screen recording when they want to. Encrypted screenshots are stored on the user’s devices and uploaded to the cloud storage at pre-defined intervals, after which screenshots are deleted. Using the default resolution of 720 pixels by 1280 pixels, each screenshot can range in size from 1 kilobyte (KB) to 300 KB depending on their visual propertiesFootnote 1. This means that the amount of storage used on participants’ devices will vary depending on how they use their phones across the study period. In our experience, the amount of storage required ranged from approximately 100 megabytes (MB) to 600 MB per user per day.

Step 2: Transfer and cloud storage

All encrypted screenshots are stored on the participants’ device until a random period between 1:00 am and 3:00 am (device time). During which, using the WorkManager API, the app will prompt the transfer of the encrypted screenshots over Hypertext Transfer Protocol Secure (HTTPS) to a researcher-specified Google Cloud Bucket, where the screenshots will be stored in encrypted formFootnote 2. It is important to note that this automated transfer will only occur if the participant is connected to a wireless network. This is to ensure that participants are not burdened by the cost of cellular data incurred during the transfer of images. Once the data has been successfully transferred, it will be deleted from participants’ devices to free up their storage space. For participants who worry about device storage being overburdened, there is also an option on the user interface of the application to upload and clear stored images when they are connected to either a wireless or cellular network. If participants are connected to cellular data instead of a wireless network, the application will prompt the user to confirm that they want to use cellular data to upload their data. Finally, once the transfer is complete, the data are stored in its encrypted form on Google Cloud. This means that individuals with access to the bucket cannot view the data without the encryption key.

Step 3: Decryption and data processing

At regular intervals, or at the end of the study, researchers can download the encrypted data into the research computer, before emptying the Google Cloud Bucket. Once all images have been downloaded into a research computer, the computer is disconnected from the Internet before we decrypt and process the data. Using the DMPO, researchers can simply decrypt the screenshots using an encryption key that was generated during the participant onboarding process. We have simplified this to a single point-and-click option on the graphical user interface (GUI) of the DMPO.

Once the data has been decrypted, we utilize an automated personally identifiable information (PII) removal module – using various open-source deep learning libraries and scripts – to remove some level of personally identifiable informationFootnote 3. First, multitask cascaded convolutional networks (MTCNN) is used to both detect regions of the screenshots that contain a face and thereafter to obscure these detected faces (ipazc, 2021; Zhang et al., 2016). Next, Tesseract Optical Character Recognition (OCR) is used to extract, identify, and remove certain strings of text (Smith, 2007). All strings containing the character “@”, numbers containing more than four digits, and character strings referring to a “person” as recognized by spaCy’s EntityRecognizer are removed from the text (explosion, 2022).

These techniques offer a quick way to screen and remove PII such as faces, names, usernames, e-mail addresses, phone numbers, and bank account numbers among others. However, as we will discuss in our limitations section, the complex visual characteristics of each screenshot mean that the level of accuracy for Tesseract OCR is too low for real-world use. To ensure as much PII is removed as possible, we assign one research team member to remove PII missed by the automated PII removal module. To manually remove PII, we use a custom-built software that allows research team members to quickly specify bounding boxes around identified PII for each screenshot screened. Images that have been screened can then be exported for analysis. Based on our existing work, we note that, after familiarizing themselves with the software, research team members can manually screen and remove PII at a speed of approximately 1000–1500 images per hour. Once PII has been removed, we delete the raw screenshots, and keep only the screenshots without PII for analysis. All our screenomics data are stored in an encrypted external hard disk stored in a locked facility within our university.

While researchers do have access to the raw screenshots throughout the process, only de-identified screenshots were analyzed in our study. From a research standpoint, retaining raw screenshots would provide a greater amount of informational detail in the dataset. However, the decision to remove PII was considered following extensive consultation with the university’s ethics board. This was determined based on a careful consideration of (1) the purpose of the study and (2) ethical concerns surrounding third-party consent. First, in considering whether PII should be removed, researchers ought to consider if there is a research question which requires such PII to be retained (Wade, 2007). The two main purposes of our research were to pilot the ScreenLife Capture application and explore possible relations between exposure to specific content and well-being. Hence, there were limited reasons for the retention of PII in our study. Second, researchers and their university’s ethics board must consider their ethical position surrounding third-party consent. For example, even though participants might have consented to taking part in the study, family and friends of the participant might not consent to having their PII recorded by researchers. These pieces of information might be unknowingly revealed to us in their private communication with a participant. Based on a risk–benefit analysis (retaining PII did not provide additional information that was necessary for the fulfilment of our research objectives) and concerns surrounding third-party consent, we decided to remove PII in our study.

Data security protocols

A major consideration in the development of a screenome collection tool is data security and privacy. In its current iteration, we utilize several industry-standard protocols to ensure that participants’ data are protected. Our system provides data security under the following threat model: (1) we assume the operating systems at both the researchers’ machine and users’ device are secure (malicious attackers can run malware alongside the DMPO application and ScreenLife Capture); and (2) we assume that the cloud storage can be compromised by software vulnerabilities, insider threats, or by malicious tenants sharing the cloud platform.

First, all screenshots are encrypted using Advanced Encryption Standard (AES) in Galois/Counter Mode of Operation (GCM). The keys are stored securely at two locations. At the user’s device, it is stored in the secure element, using the Android Keystore API. At the researchers’ machine, it is derived directly from the passphrase and therefore never stored on disk. Only the salted hash of the passphrase is kept on disk, and we use expensive hash function, namely bcrypt, for added protection against brute force attacks. In our system, all communications are over TLS, and data stored on the cloud are encrypted. Therefore, it is safe against network attacks and a compromised cloud account. Finally, raw screenshots are never stored on the user’s device. To mitigate the impact of potential software vulnerabilities in our system, we delete all data after pre-defined intervals.

Customizing and setting up the front-end app and DMPO

As described above, a researcher needs only two pieces of software to begin data collection – the front-end ScreenLife Capture APK to be installed on participants’ or research devices, and the DMPO. However, there is a need to ensure the setup of the front-end application syncs with the DMPO through a cloud services provider, so researchers need to set these up and customize the source code for ScreenLife Capture to integrate these services before building it into an APK for data collection. To do that, we provide a simple step-by-step guide to set up Google Cloud Functions and Cloud Storage (accessible at https://www.andrewzhyee.com/screenlifec). We further provide the source codes for the programs, which can be easily edited on any text editing software (accessible at https://github.com/ScreenLife-Capture-Team). It can then be built via Android Studio into an APK for researchers to load onto participants’ phones.

Exploratory study

Participants and procedure

An exploratory study was conducted to test the application and provide a preliminary analysis of screenomes collected in Singapore between December 2020 and January 2021. Using convenience sampling, we recruited 20 college students aged between 18 and 26 (M = 21.95, SD = 1.96). 14 (70%) of the participants were male, while six (30%) were female. Most of the participants were Chinese (15; 75%), while the rest were Indian (3; 15%) and Malay (2; 10%).

Prior to data collection, we obtained approval from our university’s institutional review board to conduct the study (IRB-20-00348). To recruit participants, we sent out a recruitment e-mail to all undergraduate students at our university. Participants were invited to indicate their interest to participate in the study if they were above 18 years old and used an Android smartphone. Informed consent and participant onboarding were conducted via videoconferencing due to Covid-19 restrictions. During the informed consent process, participants were briefed about the objective of the study, the types of data that would be collected, the potential risks and benefits to them, as well as measures they can take to stop or suspend their participation in the study. For participants below 21, we also obtained parental consent on top of participants’ assent to take part in the study. Once participants have agreed to take part in the study, we sent the ScreenLife Capture APK to install on their devices. Participants were asked to utilize the application for 14 days, filling out a short survey questionnaire every night, and to complete a post-study interview to better understand their experience of having their smartphone use under constant background surveillance. At the end of 14 days, participants were compensated $100 for their time and effort taking part in the study.

Preliminary analysis and key findings

A total of 743,498 screenshots were collected over 14 days, with a mean of 37,174.90 screenshots (SD = 16,311.84) collected per participant. This suggests participants averaged about 3 h, 41 min, and 16 s of screen-on time per day, resulting in an average of over 51 h of smartphone use per participant over 14 days. We offer a preliminary analysis of the results using human-coded data to conduct a descriptive analysis of individuals’ everyday smartphone use. To do this, we extracted eight participants’ 24-h screenomes on a randomly selected day (January 5, 2021) and hand-coded each screenshot to a particular type of smartphone use. To provide some context, January 5, 2021 was a Tuesday and before the start of the university’s Spring semester. For most students, this was during their semester break, and their screenomes should be interpreted with this context in mind. In total, 22,635 were hand coded.

Smartphone use types

To analyze the data, we used a two-step approach which included both deductive and inductive approaches to create coding categories to label each screenshot. First, we utilized a top-down approach first by coming up with initial coding categories through a thorough discussion with the research team, referring to existing literature where needed. The initial categories were (1) Notifications / Home, (2) Active Social Media, (3) Passive Social Media, (4) Gaming, (5) Information Search / Browsing, (6) Finance, (7) Shopping, (8) Music, (9) Video, (10) News, (11) Calls, (12) Camera and Photos, and (13) Life Admin.

Briefly, Home / Notifications referred to screenshots of smartphone home screens and notifications. We also categorized all transitioning screens under Home / Notifications (e.g., when participants are swiping between apps).

Based on existing research, we categorized behaviors on social media platforms (which included both social networking sites like Instagram and TikTok to messaging platforms such as WhatsApp and Telegram) which involved direct and intentional interaction with other users as Active Social Media use (Escobar-Viera et al., 2018; Thorisdottir et al., 2019; Trifiro & Gerson, 2019). This was visually characterized by the use of the on-screen keyboard and typing of texts, the use of cameras or audio-recording within a social media application, or by visual indicators of reacting to (e.g., likes, loves, etc.) and sharing of posts. In contrast, Passive Social Media Use involved the use of social media platforms without any direct engagement (e.g., scrolling).

Screenshots classified under Gaming involved any form of mobile gaming, while Information Search / Browsing was a category for screenshots which displayed the use of the web searches and browsers (including browsers which opened from social media and e-mail applications).

Screenshots classified under the Finance category involved any kind of personal finance related use, which included the use of personal banking and investment apps. We also classified digital payment applications under Finance.

We classified screenshots which reflected any kind of e-commerce activity under the Shopping category, which also included the use of food delivery apps for purchasing of meals. The Music category included screenshots which reflected the use of music apps, such as Spotify and Deezer, while the Video label was used to indicate the use of video-centric apps. This included Netflix, Disney Plus, Prime Video, Hulu, and YouTube. Though content on TikTok and Instagram Reels were largely short-form videos, we chose to categorize the use of such apps under either Active or Passive Social Media use instead.

We categorized the use of news apps – such as the New York Times, and news aggregators such as Google News – as News. While participants can sometimes come across news during information search and web browsing, as well as during social media use and inve-mail applications, the News category reflected more intentional browsing of news, as users had to open an application to scroll through different news. This contrasted with the less intentional and algorithmically driven news exposure which individuals may be exposed to on social media platforms.

Calls involved screenshots that reflected either voice or video calls, while the Camera and Photos category was used to classify screenshots which reflected both the use of the phone’s camera to take photos, as well as browsing of past photos in a phone’s photo application. Life Admin was used to categorize the functional use of different apps to organize and plan one’s life. This included the use of map apps for directions, review apps (like Google Maps’ reviews of restaurants), calendars, calculator, e-mail, and alarms, among others. E-mail was categorized as Life Admin instead of social media largely because our participants’ use tended to reflect functional uses of e-mail (e.g., looking at event details).

As we labeled the images, some screenshots were more difficult to map onto the categories above. For such screenshots, we set them aside and engaged in further discussions to decide on the most suitable category for classification. We made several decisions based on those discussions. First, some screenshots involved the use of educational apps like Coursera and DuoLingo. We classified these under a new category – Education. Second, we noticed one participant heavily using a variety of dating apps. Hence, we classified the use of these apps under Dating. Third, we saw the use of very niche apps for unique interests, such as sheet music, religious, and fan apps (e.g., NBA fan app). We classified this under Other Interests. As with the News category, we observed many instances where such content was also found during participants’ social media use – including Passive Social Media use. However, we chose to keep them separate, since Other Interests as a category reflected more intentional use of those apps.

Fourth, we noticed that participants sometimes utilized a social media platform’s search function to look for information. As such behaviors did not fit into the concept of either Active or Passive Social Media use, we classified these screenshots under Information Search / Browsing. Similarly, participants could search through e-mail applications to retrieve information, or open a link sent via e-mail – which then opens in a browser. These instances were also categorized as Information Search / Browsing. Finally, we also noticed that LinkedIn was sometimes used as a social media platform, and at other times as a job search platform. Hence, we classified job searches on LinkedIn under Information Search / Browsing, while non-job searches were classified either as Active or Passive Social Media use.

In total, we found that almost all 22,635 screenshots could be satisfactorily hand-coded in one of these 16 categories, which reflects a rudimentary way in which screenomes can be classified and studied. These 16 smartphone use types include (1) Notifications / Home, (2) Active Social Media, (3) Passive Social Media, (4) Gaming, (5) Information Search / Browsing, (6) Finance, (7) Shopping, (8) Music, (9) Video, (10) News, (11) Calls, (12) Camera and Photos, (13) Life Admin, (14) Education, (15) Dating, and (16) Other Interests.

Figure 2 provides a graphical representation of eight participants’ screenomes over a 24-h period, with their pseudonyms provided on the left. Meanwhile, Table 1 provides a detailed breakdown of each use type’s frequency count and relative proportions. Overall, Passive Social Media Use was the most common type of smartphone use among our participants, with 27.93% of all the screenshots coded belonging to the category, much higher than the 15.05% of time spent on Active Social Media use. This was in line with previous research which have found reported passive social media use to be much higher than active social media use (e.g., Escobar-Viera et al., 2018).

Fig. 2
figure 2

Eight participants’ 24-h screenomes

Full size image
Table 1 Smartphone use frequencies and counts by user
Full size table

Key findings

Heterogeneity of smartphone use

One common theme identified in previous screenome studies was that the nature of smartphone use was highly heterogeneous both within- and between persons (Brinberg et al., 2021; Ram et al., 2020; Reeves et al., 2021). As with these previous studies, we found substantial intraindividual heterogeneity of use, especially when viewing smartphone use across time. Using the 24-h screenomes illustrated in Fig. 2, we can see that Sophie primarily used her smartphone for entertainment purposes, watching videos on video platforms between approximately 11am to 3pm. However, her usage patterns shifted towards social media and messaging later in the evening. This intraindividual variation can also be observed with Andy, who switched between many different social media platforms during the day but used dating applications more heavily later in the evening.

The screenomes presented also illustrates how smartphone use can differ substantially between individuals. Some participants switched applications constantly and rapidly across large parts of their day, while others might use single applications over long stretches of time. The purpose of use also differed substantially, with some participants using it sparingly and for specific purposes such as contacting others and for information search, while others might use their smartphones to play videogames or passively scroll through social media feeds.

Threads of media experience

Similar to previous studies, our analysis of the screenomes also indicated that much of our participants’ smartphone experience is “threaded” (Reeves et al., 2021). A threaded media experience refers to how individuals’ interaction with media content on smartphones can cut across applications and content categories – such as moving from video entertainment to information search and/or news consumption – and yet be centered on one cohesive use experience. To identify threaded media experiences, we first segmented smartphone use into distinct sessions. We classified a smartphone use session as a single stream of continuous active screen on time. Each session is distinguished from another when there is a lapse of more than 7 s between two recorded screenshots. Once each session was isolated, we examined the screenshot to see if there were app switches. When app switches were identified, we made a subjective decision to categorize if each switch was relevant to the content in the previous screenshots. If it was assessed as relevant, we then classify those screenshots as belonging within one threaded media experience.

In an example of a threaded media experience, Andy was passively scrolling through LinkedIn when he encountered another LinkedIn user’s profile that caught his interest. He looked at the profile of this individual, switched to a browser, conducted a cursory search of that person online, before browsing the website of the company that individual was working for. Next, he switched back to LinkedIn and looked at job openings within that company, switched back again to the browser to conduct an information search on salaries, before moving back to passively scroll through Instagram.

Diverse possibilities for the conceptualization of media use

One commonly used strategy in conceptualizing and operationalizing media use is to define it as frequency or duration of use (Gil de Zúñiga et al., 2017; Khan et al., 2021; Orben & Przybylski, 2019; Rozgonjuk et al., 2020; Twenge, 2019). During our inductive coding process, it was evident that time-based operationalizations of digital media use – including smartphone use – were severely limited in a time where so much of life is digitally mediated. This is in line with previous scholars’ call for the need to pay greater attention to the range of content and interaction types when considering the effects of digital media on people (Antheunis et al., 2015; Ohme et al., 2016).

Screenomes provide a level of informational depth and enable a diversity of ways one can conceptualize media use. Our human coding of screenomes identified different approaches and “layers” of conceptualization – which we call tagsets. For example, one approach to code the data is by smartphone use type – as we have done in Fig. 2 and Table 1. Going further into deeper layers, we could distinguish between different types of news – political, entertainment, sports, and financial, among others. If it is of interest to the researcher, deeper layers and sub-categories of these different news types can be further explored – from liberal versus conservative political news content, to positive or negative news coverage. Other approaches which surfaced include operationalizing media use in terms of application switching (Deng et al., 2019), types of incidental exposure to information which then serve as gateways to news and information (Feezell & Ortiz, 2021; Lee & Kim, 2017; van Damme et al., 2020), types of media-enabled social interactions and behaviors (Ryan et al., 2017), and highly visual versus text-based social media content (Marengo et al., 2018), among others.

Relatedly, in Table 1, we see that some participants – like Andy, Clint, and Nick – spent more than half their time on their smartphone on Passive Social Media use. In fact, we see that most participants spend a substantial amount of time passively using social media. Though some researchers argue that passive social media use may be detrimental to well-being (Escobar-Viera et al., 2018; Thorisdottir et al., 2019), there is a growing number of scholars who suggest that the relationship could be more complex than previously thought (Kross et al., 2021). More recently, Valkenburg et al. (2022) suggested that merely measuring time spent on active or passive social media use may be too imprecise, and that researchers should consider other characteristics such as the content of use when examining social media use.

Beyond allowing us to differentiate between active and passive social use of mobile phones, the informational depth of screenshots allow us to go deeper into exploring the types of Passive Social Media use. After isolating all the screenshots reflecting Passive Social Media use, we saw stark differences in the types of uses. Andy and Mike were following their favorite sports team on social media platforms and are a part of Instagram and reddit communities built around the support for their sports team. This has the potential to cultivate a sense of community and relatedness, which are crucial nutrients to well-being (Ryan & Deci, 2017). Meanwhile, portions of Alice’s passive social media use include content she potentially finds joy in – such as in art and design. Likewise, while Sophie was actively engaging others on social media platforms, her passive use consisted of her browsing and reading about her hobby, cosplay.

The impact of constant background surveillance: Findings from in-depth interviews

Given that screenome studies involve high levels of surveillance, we wanted to understand participants’ experiences taking part in such studies, so that it can help inform future research protocols and policies which might alleviate participants’ concerns surrounding privacy, security, and psychological discomfort. All 20 participants took part in semi-structured interviews over videoconferencing or an online audio-only call. We broadly asked questions about their emotional experience and how they felt throughout the 2-week data collection period, the reasons behind them, and in what situations they suspended screen recording. We also asked how they thought we could make the experience better for them. The conversations were recorded and transcribed, before being analyzed using the constant comparison approach (Strauss & Corbin, 1990). A researcher read the transcripts line-by-line and coded the data for emerging themes and concepts. We then compared the codes across transcripts and synthesized them into broader themes which describe their experience of going through a screenome study.

Six themes emerged from our analysis, all of which broadly reflected different dimensions and consideration of the impact of being part of a screenomics study. First, participants noted certain emotions brought about by being constant surveilled. Second, these emotions led to a greater awareness of the delineation between personal private and public behaviors. Third, participants adapted and got used to surveillance after some time. Fourth, despite getting used to it, participants did exhibit some level of behavioral changes. Fifth, some level of self-discovery emerged out of the entire process. Finally, we identified two areas of concern that researchers using this method ought to take note of.

Constant background surveillance led to heightened sense of awareness

When asked about the experience of having an application record their smartphone activity, participants noted that there was a heightened sense of awareness of their smartphone use. One participant recounted:

Basically, I think it is just [that] sometimes I feel quite self-conscious. For me [the feeling] is kind of neutral. I guess I kind of knew what I was getting myself into when I signed up for the study. So, it is not exactly that I wasn’t prepared that like “oh my data could be like looked.” It is just that you’re more aware of the stuff that… At least I was more aware of what stuff I was doing on my phone.

While a heightened sense of awareness was common across all participants, some felt that it was unfamiliar and difficult to describe, with one participant calling it “weird” and another describing it as a “quirky” feeling. One other participant described it as neither “sad nor unhappy”, but perhaps “like an itch that you can’t scratch”. This was described by some as a feeling of being watched, which is slightly psychologically discomforting. One participant said:

Uh, although I did feel like, uh, a bit unnatural because I kind of, do remember sometimes, you know, that I’m being watched.

One reason for the state of discomfort experienced by participants could be related to the fact that the constant surveillance from ScreenLife Capture undermined their sense of privacy. As Pedersen (1997) argued, privacy can “be viewed as a boundary control process in which the individual regulates with whom contact will occur and how much and what type of interaction it will be (p. 147).” The presence of a screen capturing software on a smartphone – which is perceived as a highly personal and private device which the user normally has extensive personal control over – threatens the level of control a user has over his communication with multiple parties. The sense of heightened awareness and discomfort could perhaps be due to a diminished sense of privacy by the participants (Lombardi & Ciceri, 2016).

Interestingly, participants did not discuss the similarities or differences between ScreenLife Capture and how other entities – such as social media companies – conduct background surveillance and data collection (Zuboff, 2019). This could be due to the normalization of a surveillance driven by a familiarity with such practices (Lyon, 2017). It is possible that since the presence of ScreenLife Capture was unfamiliar, it generated a more salient feeling of being surveilled.

Delineation between personal private and public behaviors

Relatedly, participants reflected that such feelings of heightened awareness and discomfort manifested most when they enact what they perceive to be personal and private actions, including in private conversations with family and friends and engaging in activities that are usually done in solitude. One participant said:

I think, as I mentioned earlier, it seems like [some] conversation[s] are only meant to be read by two people, right? No one else.

Another reflected that there is discomfort when he engages in certain activities that he might normally undertake in alone:

When I am looking at TikTok, then I am like “hmm, will people judge me if I am looking at this?”. You know sometimes bikini pics come out? Yeah. More like I am watching some web content or like some weird content. Like “Hmmm… Don’t judge me.”

Despite this, several participants noted a difference between private and public behaviors enacted on their smartphones, pointing out that these were distinct types of behaviors which affect them in different ways during the study. Social media activity, as well as entertainment (such as watching videos and playing mobile games), were viewed as public when compared to private conversations on messaging applications, since they could be accessed by others anyway:

I think I was more conscious of messages like maybe WhatsApp and Telegram. With regards to social media, I wasn’t really like, conscious. I wasn’t really like scared of it. I mean, social media we all just surf stuff and, you know, see the news, and all that stuff. I think I was most calm about entertainment la, like going on YouTube, or watching Netflix.

This delineation between private and public activity was reflected by another participant, who said:

For example, I am talking to my family chat or something. Like talking in group chats. I mean, in these chats, there are already a lot of people who can see what is going on. So, if someone were to look at it, it doesn’t really matter to me. But things like maybe swiping Tinder or like stuff, then it feels like “Wait, that means someone who is watching this will know what’s my type of person, etc.”.

Previous research has shown that when online, people engage in strategic impression management to avoid “context collapse”, where they present different facets of themselves to accord with the norms of different social groups they belong to (Lim et al., 2012; Wesch, 2009). Since the screenomes captured all that he did across multiple platforms and social networks, this participant was more acutely aware of how his different (and possibly dissonant) ‘selves’ could appear to the researchers.

Behavioral changes and omitted screens

One feature in the design of our application is that participants could suspend screen recording at any point in time, ensuring that participants retained some sense of control and autonomy over their data. However, this comes at a cost of collecting less complete screenomes. On top of suspending screen recording, participants also revealed that they sometimes used another device, such as a laptop, to perform certain online activities, since the ScreenLife Capture application was running on their phones. There were two primary drivers for such behavioral change. First, some participants did so for security reasons, such as when keying in passwords, or when accessing financial services. One participant said:

I think when I was using banking and more investment platforms, I was more thoughtful about it. Um, and then I did those things through my laptop more, but besides that I think I was fairly comfortable with everything else.

Second, some types of smartphone use, usually engaged in solitude, proved too embarrassing and uncomfortable to be performed under surveillance. This included what one participant call surfing for “not safe for work” content. One participant summed up both reasons in his response:

Like when I’m doing more sensitive stuff, like, when I’m doing my banking stuff. Or, like when I, you know, arguing with my girlfriend, that kind of stuff then– I’ll like, feel a bit uncomfortable– more uncomfortable than usual, so I pause the app.

Beyond missing screens, some participants revealed that they might have altered the way they used their smartphones due to the feeling of being watched. One participant mentioned they used certain social media platforms less, as they did not want to give the impression that they were excessive social media users. Another participant mentioned that he “stalked” fewer people during the 2 weeks of data collection, because he does not want to “give the wrong impression”. These findings suggest that even an objective measure of media use such as screenomics is potentially susceptible to social desirability bias (Fisher, 1993).

Adapting to constant background surveillance

Despite feeling a sense of heightened awareness and psychological discomfort, several participants noted that these feelings faded after an initial period of adaptation. One participant said:

I think in the first few days I was a bit nervous, obviously, cause it’s like someone is watching what I’m doing all the time. But then after a while, I think, it kind of became okay for me. I think about, by the fifth or sixth day, I was fine. It just became a background thing that I didn’t really mind too much.

Previous research examining the impact of ubiquitous surveillance of individuals in their homes also noted this phenomenon of participants becoming accustomed to surveillance over time (Oulasvirta et al., 2012). For some, this desensitization to the constant surveillance extended to personal and private activities, such as in one-on-one conversations between family and friends:

Texting wise, the first few days again, I was just – should I be a little bit more careful of what I say, kind of feel? But then after a while, I became used to it, and then I just, you know, started typing without worrying too much about what was happening.

Self-discovery

One unexpected finding was that some participants reflected that the process of taking part in the study led them to greater knowledge of themselves. One explanation could be that being under constant background surveillance brought about a greater level of self-awareness, which results in participants focusing on themselves and learning more about their emotions and thoughts (Morin, 2011). For example, one participant noted that he had always believed that he did not care how others saw him. However, taking part in the study made him realize that he was conscious about his actions and how the researchers would view him:

I want to say that I don’t really care about how people think about me. But I think this experiment proved me wrong. Because when I do certain things, I don’t want people to think that I am like this.

Arguably, the awareness that his thoughts and actions were being captured for subsequent analysis served as a reminder than even in daily mediated communication that seems otherwise mundane and inconsequential, he is in a sense still on the ‘front stage’ (Goffman, 1969).

Trust and the researchers’ role in protecting the participants

Finally, we found that there was a need to uphold participants’ trust and an obligation to protect their privacy. Like Reeves et al. (2021), our conversations with participants noted that one reason participants were cooperative in keeping the screen recording enabled for most of the study was because they trusted a university research team to respect their privacy and keep their data secure. This trust must be protected in highly sensitive research like screenomics. A small minority of participants noted that their discomfort lingered beyond the end of the study, and researchers are obligated to follow-up and enact procedures to manage such incidences. This discomfort was described by one participant:

It was a bit weird. Cause it feels like I am always being watched. Actually, even until now, I feel like I am being watched. I deleted it already. But it is a lingering feeling like someone is watching me.

Discussion

This paper introduces ScreenLife Capture, a user-friendly and open-source screenome data collection tool with a companion DMPO software which allows researchers to start their own screenomic projects. We further detailed an exploratory study which replicated some of the findings from the earliest screenomics research (Brinberg et al., 2021; Reeves et al., 2021). Finally, we conducted an in-depth exploration on the impact screenomics research can have on participants, which has important implications for the design of future screenome studies.

First, and perhaps most importantly, we have provided one of the first freely available tools for researchers to collect screenomes. To facilitate further research in this area, we have made ScreenLife Capture’s source code fully open access and provided a simple and detailed guide for researchers who may not be familiar with programming to start their own screenome studies. This may be useful as it allows diverse teams of researchers to fully utilize the depth of information packed inside individuals’ screenomes. Some scholars have argued that there is a need for researchers to use objective screen data to tackle the conceptual ambiguity of screen time. Specifically, Kaye et al. (2020) noted that there were insufficient freely available resources for the collection of high-quality screen time data such as screenomes. ScreenLife Capture can fill this methodological gap, and by virtue of being freely accessible, enables researchers with fewer resources to utilize the technique. We have also designed ScreenLife Capture to work through cloud-based solutions, which reduces the cost and effort needed to run screenome studiesFootnote 4.

Second, our exploratory study is one of few studies outside of the Stanford Human Screenome Project to provide an in-depth analysis of screenomes. Overall, we replicated previous findings and found strong heterogeneity of smartphone use. Individuals vary their smartphone use across socio-temporal contexts, and different individuals use their smartphones very differently. Echoing other scholars (e.g., Barr et al., 2020; Brinberg et al., 2021; Kaye et al., 2020; vanden Abeele et al., 2022), we argue that screen time as a concept is insufficient in capturing the idiosyncrasies and diversity in which individuals engage with digital devices. We also found that media experiences are oftentimes “threaded” through multiple application switches, and that studies examining task or application switching cannot take on face value that an “app switch” indicates tasks switching (Deng et al., 2019).

Building on this, our process of labeling and coding screenshots have suggested that there could be a diversity of approaches and layers in which smartphone use can be captured. Previous research has suggested that the different ways in which smartphones are used by individuals can have different impact on outcomes (Elhai et al., 2017). With close to 2.9 million different applications available on the Google Play Store (Statista, 2021a), we believe that future research using the screenomics approach can enable novel theoretical conceptualizations of smartphone use. For example, previous studies have suggested how exposure to certain types of social media posts (Kreling et al., 2022; Meier et al., 2020), playing and participating in video games and its culture (Oliver et al., 2016; Przybylski et al., 2012; Yee & Sng, 2022), and even viewing meaningful memes (Rieger & Klimmt, 2019) can be beneficial for well-being. Such meaningful experiences with media (e.g., Oliver, 2022) – even during passive social media use – can potentially be understood better with screenomes. Specifically, future research utilizing ScreenLife Capture can provide more in-depth descriptions of digital media use beyond the active/passive divide. For example, scholars have recently used the screenomics approach to study the concept of attentional inertia (Brinberg et al., 2022), as well as interpersonal relationships and smartphone interactions (Sun et al., 2022).

Third, our in-depth interviews highlighted the impact of participating in studies involving constant and ubiquitous surveillance. First, researchers must expect that participants will feel some level of discomfort at the start of their participation. This is possibly due to a loss of control and sense of privacy. To mitigate this, researchers should display empathy and remain sensitive to the participants’ comfort level and be ready to terminate data collection if participants indicate a desire to withdraw their consent from the study. Other than displaying empathy, researchers should always highlight to participants that they have the option to withdraw from the study at any point in time. Furthermore, researchers can also highlight that the ScreenLife Capture application by default allows participants to suspend screen recording as and when they see fit. These conversations can help reinforce a sense of autonomy and control among participants.

Fourth, participants distinctly delineate between private and public smartphone use actions, and some interactions perceived to be extremely private might be omitted from their screenomes. Some participants also revealed that they might interact with their devices in ways that might be perceived to be more socially desirable, further suggesting that collected screenomes might differ slightly from their regular smartphone use. There are two things researchers can do when utilizing the technique. The first is to encourage participants to keep the screen capturing on as much as possible, and to remind them that the data would only be analyzed on an aggregate level. Next, researchers must always interpret screenomes with the knowledge that some aspect of participants’ smartphone use may be omitted from the dataset. Future work should go into better determining what types of smartphone use are people more likely to refrain from sharing with researchers through such data collection applications.

Finally, the interviews revealed that trust in the university and research protocols is a crucial criterion for participating in screenomics studies. Researchers ought to value and protect the trust which participants place in the integrity of academic research by placing participants’ interest and well-being first, and by strictly following data security and privacy protocols.

Ethical considerations

As screenshots may reflect deeply personal and private information, researchers planning such studies must address several key ethical considerations. These include issues surrounding autonomy, beneficence, justice, transparency, the right to privacy, and dual use. These facets are intertwined and require researchers to carefully plan and implement study procedures which consider ethical issues related to their study.

As discussed above, there is a significant loss of privacy and sense of autonomy when participants enroll in screenomics studies. Participants also reported feelings of discomfort in the initial phase of participation in a screenomics study. To ensure that participants autonomy, privacy, and well-being are protected in accordance with the Belmont Report, we designed ScreenLife Capture and our accompanying study protocol around three main practices.

In the frontend application, participants could choose to stop screen capture at any time. This function is aimed at offering a sense of control and minimizing the psychological discomfort participants encountered during the study. For example, if participants felt highly uncomfortable typing in passwords or accessing their bank accounts while screen capture was occurring, they had the option to access the app and temporarily suspend screen recording. Second, we implemented a policy where participants can contact researchers anytime during the study to remove data which they do not want included in the analysis. They could do so by informing us of the date and period in which said data are located. Following that, researchers would then delete the specified data in its encrypted form. Finally, participants were free to delete the application and leave the study at any point in time, and researchers were obliged to delete all collected data.

During the process of recruitment, we were also completely transparent about the use of participants’ data and provided a detailed informed consent procedure which fully described the study without any use of deception nor concealment. During these consent-taking sessions, we also provided a frequently asked questions (FAQ) document for participants which addressed some common concerns raised by participants. These were worded in non-technical terms and could be easily understood by participants. These practices were in line with the basic ethical principles of autonomy, beneficence, and justice as expressed in the Belmont Report.

Beyond adhering to these ethical principles in our study, we also respected the right to privacy for both the participant and third parties. It is important to note that personal and private information contained within screenshots do not only pertain only to participants’ but also to their contacts who have transmitted information to them via certain smartphone applications. For example, a participant’s acquaintance might take a selfie and send it via a messaging application to the participant, intended only for the participant’s consumption. ScreenLife Capture would inadvertently capture such third-party PII sent to the participant. In our pilot study, we made the decision to mask both first- and third person PII to respect both the participant and third-party individuals’ right to privacy, as we were unlikely to be the intended recipient of information being shared by a third party. As discussed earlier, we made the decision based on two reasons. First, the purpose of our study did not warrant the analysis of both first- and third-party PII. Second, we collectively committed to an ethical position on third-party consent and their right to privacy, after discussions with our university’s ethics board.

While we do not make a formal recommendation for or against the removal of PII (both first- and third-party), we encourage researchers considering the use of ScreenLife Capture to engage in a similar process of ethical deliberation. First, researchers should consider if their research question requires PII to be retained and conduct a risk–benefit analysis to ascertain if the benefits of retaining outweigh the risks of removing PII (Wade, 2007). Second, researchers should engage with their research team and university ethics board to develop their ethical position on third-party consent and individuals’ right to privacy.

Finally, researchers must be aware of potential situations in which dual use of data may occur and find ways to manage it. Dual use – in the context of research – refers to the idea that research can have multiple potential uses (Miller & Selgelid, 2007). The first is the intended purpose – or the purpose in which the research was designed for. In the context of screenomics, this is usually to address a media-related research question. The second involves the use of the data for purposes outside of its original intended purpose. There are several situations in which a dual use dilemma may occur for researchers utilizing screenomics. One potential situation is when the researcher comes across information in the research data which indicates potential harm or criminality. For example, what should a researcher do when they come across information from a participant which indicates that he or she is being abused or is on the brink of self-harm? Similarly, how should a researcher respond when they come across criminal information in their analysis of the screenomes?

In both cases, researchers may be either legally obliged or morally convicted to deviate from the original intended use of the research data. However, we urge researchers to carefully consider the ethical principles which guide their research and implement study protocols to reduce the probability for such dilemmas to occur. In the case of abuse or self-harm, researchers must maintain a balance between respecting the autonomy of the participant and protecting their well-being. When researchers judge that the likelihood and severity of harm is high, they may be compelled to breach confidentiality and report the information to relevant parties who can stop the harm from occurring. In the case of criminality, researchers who work in jurisdictions with legal obligations to report a crime may be compelled to do so. While there is strong debate in academic ethics as to whether it is ever right to breach research confidentiality (Lowman & Palys, 2014), such dilemmas are often difficult for researchers to wrestle with (Surmiak, 2020).

Instead of prescribing a solution to these dilemmas, we can try to mitigate them from occurring in the first place. First, researchers can include clauses in the informed consent form which spells out the conditions in which confidentiality may be breached. For example, researchers can make it clear to participants that information about criminal activity will always be reported to relevant authorities. Second, researchers can include an embargo before working on the data. As the reporting of criminal activities and harm/self-harm are often time-sensitive, an embargo will reduce the chances of encountering relevant time-sensitive information which warrants a confidentiality breach. While these may not totally remove such dilemmas from occurring, they can be useful in helping researchers reconcile and resolve them were it to occur.

Limitations

There are several limitations that researchers must be aware of. First, the volume of data collected via ScreenLife Capture is exceptionally high. The existing automated PII removal tool is blunt, and we currently require substantial manual labor to mask all PII from the data collected. An alternative approach would involve no “human-in-the-loop” in the process of data collection and analysis. This means that data are collected, decrypted, and analyzed without a human having access to the raw data. Future research ought to examine how that approach can be implemented in studies utilizing screenshots.

Relatedly, in our preliminary analyses, we manually hand coded each screenshot. To speed up the process, we developed a customized software which (1) breaks each participants’ collection of screenshots into separate consecutive use sessions (use sessions are separated when the two consecutive screenshots have a 7-s time difference), (2) allows us to view multiple consecutive screenshots across a large screen, (3) relies on “tagsets” – which are a collection of labels that can saved and loaded for each set of screenshots we are analyzing, (4) uses keyboard entry for quicker two-handed labeling, (5) allows us to label consecutive screenshots as a group, and (6) quickly exports the labeled data into formats which could be readily analyzed in statistical softwares such as R. This significantly sped up the time spent labeling the screenshots, as consecutive screenshots with similar visual properties could be quickly labeled at one go. Having said that, the manual coding of screenshots can be a significant barrier for researchers who want to scale up such studies. To overcome this, future research should focus on developing machine learning models to automate the process of PII removal and labeling of data. One way to collectively speed up this process is for researchers to share models they have trained to label screenome data, so other researchers can quickly implement them in their own research.

Next, other than the process presented in the preliminary analyses, we did not provide an in-depth discussion on the possible analytical procedures in interpreting and making sense of screenshot data. There are two reasons for this. First, careful conceptualization and operationalization of media use requires expertise and time, and we have referenced several important pieces of work from colleagues who can provide far more in-depth discussion of those phenomena. Next, our position is that the work of finessing the techniques in which screenomes are analyzed can only advance as far as these types of data are being collected by research groups. While future work should be dedicated to discussing specific techniques in which screenome data can be analyzed, this paper remains focused on the collection of screenomes through the ScreenLife Capture application framework.

Second, while ScreenLife Capture can provide highly in-depth objective data of smartphone use, it does not consider the spatial and psychological dimensions surrounding that data. The Stanford Screenomics application tags geolocation data to the collected screenshots, which can help provide an additional dimension to researchers’ interpretation of screenome data. Future researchers can build on the open source ScreenLife Capture application to include options for the collection of such metadata to provide more contextual information. Additionally, screenomes alone do not convey the psychological state of users at the moment of each screen interaction – such as their emotions, motivations, and goals. Future research can combine screenomes with ethnographic methods such as self-confrontation interviews to better understand their mental processes at the moment of action (Lim, 2002). These methods rely on confronting participants with detailed records of their activity so that they can reconstruct their mental state during the activity (Lahlou, 2011). Researchers can also tap on ecological momentary assessments to collect measures of temporal psychological states for use in relation to the screenome data (Meers et al., 2020). Such mixed-method techniques can potentially help researchers examine and test theories from multiple approaches, including self-effects and reception effects of media use (Valkenburg, 2017).

Third, it is important for researchers using such approaches to be aware that there are media-related activities not captured through smartphone screenshots. For example, ScreenLife Capture does not capture screenshots when the screen is not active, which means activities which continue while the screen is not active (such as participants listening to music) is not captured. Likewise, smartphone audio is not captured via the application, which means that the content of non-text-based conversations are always omitted.

Fourth, our application is currently only developed for the Android mobile operating system, which limits participation only to Android users. This means that a large group of smartphone users using other operating systems – such as Apple’s iOS – are not eligible to take part in studies using ScreenLife Capture. Despite this, with approximately 70% of the global smartphone market share (GlobalStats, 2022), there remains a sufficiently large set of potential participants for us to conduct research on. Alternatively, if resources allow, we suggest the use of research mobile phones in future studies. When participants are assigned research devices, this will also minimize possible technical issues which may arise from the different versions of the Android operating system adopted by various phone manufacturers. Ideally, research mobile phones should run the stock Android operating system.

Finally, since screenome projects typically involve the collection of sensitive personal data, researchers must consider the laws surrounding data protection and privacy in the countries they are intending to collect data in. There is a possibility that researchers’ ability to conduct screenomics research may depend on each country’s prevailing privacy laws and practices. While we have integrated specific features within ScreenLife Capture to respect key ethical principles of autonomy and non-maleficence, future research should consider the development of specific protocols surrounding the management and handling of screenome data.

Despite these limitations, we believe that the ScreenLife Capture application, our exploratory study, and the in-depth interviews examining the impact of screenomics on participants offer some methodological contributions to the study of digital media. Our hope is that this paper and our accompanying solutions can spark new theoretical and conceptual approaches to study media interactions across the globe.