Keywords

1 Introduction

Due to European privacy laws, particularly the European ePrivacy Directive (ePD) and the General Data Protection Regulation (GDPR), websites must obtain users’ consent for tracking. To comply, websites commonly display consent notices, asking users whether they accept or decline tracking, such as for advertising purposes. While refusing tracking has traditionally incurred no monetary cost for users, European websites increasingly adopt pay-or-tracking walls, sometimes known as “consent or pay models,” “cookie paywalls,” “pay-or-okay walls,” or “accept-or-pay cookie banners” [21, 22, 24]. These pay-or-tracking walls require users to either pay a fee or consent to be tracked in exchange for access to the website. While the latter (“tracking option”) involves no payment but the user’s consent for tracking, the former (“pay option”) costs money, requires no consent for tracking, and comes with a text that highlights its privacy advantage, such as “without tracking,” which we refer to as the “privacy promise.”

However, initial evidence suggests that websites might continue to track users even when they purchase the pay option for accessing the website without providing consent for tracking [21, 22]. Such reports raise questions about how websites using pay-or-tracking walls fulfill their stated privacy promise for paying users and whether they align with the actual tracking practice.

If websites break their privacy promises by tracking users even after receiving payment, it will not only deceive users but might also breach legal standards regarding consent under Article 5(3) of the ePD, unless such tracking is deemed strictly necessary for the operation of the website. Conversely, if the allegations are false and the pay option’s implementations are privacy-preserving, it would show that pay-or-tracking walls are better than their reputation and an adequate alternative to protect users’ online privacy while supporting websites’ monetization.

This paper assesses whether websites employing a pay-or-tracking wall keep their privacy promises and offer pay options that safeguard user privacy. It addresses the following research questions (RQ):

  • What do websites promise users who purchase the pay option? (RQ1)

  • What tracking do websites implement in pay-or-tracking walls’ choice alternatives? (RQ2)

  • To what extent do websites keep their privacy promises? (RQ3)

2 Overview of Tracking

2.1 Value, Privacy Risk, and Technologies of Trackers

Tracking is often defined as collecting users’ online activities over time (e.g., [15, 20]). It can be particularly harmful to privacy when the data recipients collect a relevant portion of the user’s online activities across multiple websites [15], a practice referred to as cross-site tracking. This risk is particularly present when websites integrate widespread third-party services operating as a tracker. The latter enables websites to embed relevant functionalities, such as advertising (e.g., DoubleClick to manage ad sales), site analytics (e.g., Google Analytics to analyze website traffic), or social media (e.g., Facebook Social Plugins to integrate a widget). However, the integration may also enable the tracker to collect the user’s online activities across multiple websites, for instance, by storing a third-party cookie on the user’s device that allows re-identifying the user on different websites. While third-party cookies are a well-known method, trackers employ a range of technologies, also including fingerprinting or tracking pixels (e.g., [3, 25]).

2.2 Tracker Vs. Essential Third-Party Service

The GDPR requires that any processing of personal data must be carried out by data controllers based on a legal basis (as listed in Article 6), such as consent or legitimate interest. Additionally, tracking is further restricted by Article 5(3) of the ePD, which mandates consent when operations involve storing and gaining access to information on the user’s device, as in the case of trackers, and regardless of the specific technology used [7]. Under the GDPR, Article 4(11) and Article 7 define consent as a voluntary and explicit choice (among other requirements). Consequently, websites must ask users for consent to track them and embed trackers.

However, websites’ technical functionality frequently depends on third-party services, and these essential third-party services often store and retrieve information from the user’s device. Accordingly, Article 5(3) of the ePD provides exceptions to the consent requirement, which can apply when being deemed as “strictly necessary” for the website’s operation. Legal authorities interpret these exceptions rather narrowly and typically refer to purely technical necessity (e.g., [4]).

Examples of essential third-party services that websites deem technically necessary involve, for instance, tag managers. Websites need tag managers to centrally manage scripts, integrate dynamic elements, and streamline privacy compliance efforts. Another example of essential third-party services are site analytics tools, which are crucial for generating aggregate performance statistics, detecting navigation issues, or optimizing technical performance.

However, since these essential services, including tag managers and site analytics tools, are often maintained by third parties, they may transmit user information, such as IP addresses, or store third-party cookies, potentially enabling the third-party service to track the user (even across websites). Therefore, websites should ensure that essential third-party services are designed and configured to minimize the risk of being misused for tracking purposes beyond the essential technical purpose. Further, websites rely on third-party services to handle the transmitted information responsibly. Due to these risks, companies that provide anti-tracking protection, such as Ghostery, often classify essential third-party services as trackers [13].

Moreover, in the case of third-party site analytics services, legal authorities may consider them essential but only under strict conditions. For instance, the French Data Protection Authority mandates that such third-party site analytics services must produce anonymous statistical data only, not use a global user identifier across different websites, and limit its sole purpose strictly to site analytics to be exempt from consent [10].

While essential third-party services can pose a risk to user privacy, their technical necessity and legal recognition for being exempt from consent suggest a distinction between trackers and essential third-party services, which we consider in the analysis of this paper.

3 Related Work

3.1 Discrepancy Between Promise and Reality for Cost-Free Decisions to Refuse Consent or Opt Out

Many privacy laws, such as the European privacy regime, and industry initiatives, such as Apple’s App Tracking Transparency framework, aim to mitigate privacy risks by empowering users via explicit consent (opt in) or enabling users to object to (default) tracking activities (opt out). Thus, in addition to the tracking option, websites and apps typically provide a second alternative, allowing users to access a website or app cost-free without tracking.

However, previous research has revealed discrepancies between privacy promises and actual tracking practices in the context of such cost-free and supposedly tracking-free choices. For instance, Matte et al. [19] document instances of websites storing positive consent even when users refuse consent. Further, Bouhoula et al. [1] find that websites often continue tracking despite users choosing to refuse. Similarly, Bui et al. [2] show that embedded third-party services on websites may continue tracking despite users choosing the cost-free opt-out option, contradicting the privacy promise not to track opt-out users. Similar discrepancies have been observed for apps [5], with multiple studies highlighting that privacy labels for apps are often misleading and diverge from the tracking reality (e.g., [16, 17]). Further studies aim to explain such discrepancies. For instance, Utz et al. [26] report that web developers often lack awareness regarding data collection of third-party services and that privacy plays a minor role in their implementation decisions.

3.2 Potential Discrepancy Between Promise and Reality in Pay-or-Tracking Walls

Providing a tracking-free alternative and keeping the privacy promise is even more critical in the context of pay-or-tracking walls. The pay option offered by European websites represents the legally required possibility to refuse consent for tracking under European privacy laws, and it is a service the user pays for.

Three studies have examined pay-or-tracking walls, with two indicating that paying users may still be subject to tracking [21, 22], while a third study reports the opposite finding [24]. First, these studies do not address whether websites transparently include certain tracking activities within the pay option as part of their pay option’s privacy promise. Secondly, the three previous studies exhibit limitations that prevent a conclusive determination of whether paying users are being tracked, involving no measurement of embedded trackers [21], no distinction between trackers and essential third-party services [22], and no measurement of tracking beyond third-party cookies [24]. Thus, this study aims to comprehensively assess whether websites keep their privacy promises and offer pay options that safeguard user privacy. We elaborate on the details of each of the three previous studies on pay-or-tracking walls and this study’s contribution in what follows.

First, Morel et al. [21] highlight the growing trend of European websites charging users for data protection, identifying 431 websites employing a pay-or-tracking wall via an automated approach. They further examine the websites’ implementation of the Transparency and Consent Framework (TCF), an industry standard for websites to streamline consent decisions and disseminate them with the website’s implemented third-party services. They find that even if users choose the pay option and refuse to consent, they might be tracked for inappropriate purposes, such as advertising, which do not fall under the consent exemption of Article 5(3) of the ePD as outlined by the authors. However, the analysis focuses on the disseminated consent string via the TCF but includes no measurement of embedded trackers on the websites. Thus, it is unclear whether only the TCF consent string is unlawful or whether users are actually tracked.

Second, Müller-Tribbensee et al. [22] analyze the popularity of pay-or-tracking walls, design, price, and user reactions among top European websites. Regarding tracking activities, they examine 26 top European websites using a pay-or-tracking wall and discover the presence of trackers on most websites even after purchasing the pay option. However, it is unclear whether some of the identified tracking activities are essential third-party services deemed technically necessary for the website’s operation.

Third, Rasaii et al. [24] developed an automated approach to detect pay-or-tracking walls and analyze tracking cookies in the pay option of 219 websites. They find no evidence for tracking after purchasing the pay option. Their analysis focuses on a group of websites offering a joint subscription for the pay option, allowing access to all participating websites. The group is managed by an independent provider, which the authors refer to as a subscription management platform. Such websites participating in the group that offer a joint subscription potentially implement a different tracking regime than other websites, as the group may need to implement the same tracking policy. More importantly, the study focuses on tracking via third-party cookies. Thus, whether the websites continue tracking paying users via different technologies, such as tracking pixels, remains unclear.

Motivated by the insights and mixed findings from prior studies, this paper’s contribution entails examining the alignment between the stated privacy promises and the actual tracking practice for pay-or-tracking walls. Further, this study covers various tracking technologies (e.g., third-party cookies, pixels) across different website types (e.g., those participating in a group offering a joint subscription or not). It also considers the distinction between trackers and essential third-party services. Lastly, this paper analyzes the potential discrepancy between what users pay for and receive.

4 Setup of Empirical Study

4.1 Sample

This study uses the website list of pay-or-tracking walls provided by Morel et al. [21], including a country and website industry classificationFootnote 1 We focus on German-speaking countries, i.e., Austria, Germany, and Switzerland, representing 341 websites and 79% of all identified websites using pay-or-tracking walls. We visited the selected websites between October 6 and November 26, 2023, from an IP address in Germany using the Google Chrome browser (version 117.0.5938.150). We purchased the pay option for each website and collected data about their stated privacy promise and actual tracking activities.

4.2 Collecting Privacy Promises

As shown in Fig. 1, a pay-or-tracking wall typically gives users the choice between the pay and tracking option upon their first website visit. While the latter asks for consent and informs the user about the tracking purposes, the pay option requires no consent and may include additional benefits, such as reduced advertising or access to premium content [22]. Further, the pay option is presented along with a privacy promise, which involves a text highlighting its privacy advantage, such as “without tracking.”

Fig. 1.
figure 1

Exemplary pay-or-tracking wall of the website “0180.info”. Notes: Screenshot taken on October 6, 2023. Pay option (“... or with contentpass”) displayed on the right half, promising website access “completely free of personalized tracking” and without advertisements.

Full size image

We retrieve the privacy promise associated with the website’s pay option using a manual approach, allowing us to reliably analyze the relevant texts from the pay-or-tracking wall’s first layer (i.e., what users see upon the first website visit). For each website visit, we use the Google Chrome browser in Incognito mode, imitating a first-time visitor without browsing history. We refer to that configuration as a “clean” browser instance. We further configured the browser’s language settings so that websites would show their content preferably in English. This led to websites displaying their pay-or-tracking wall, if available, in English and otherwise in German. We document each pay-or-tracking wall with screenshots and translate German privacy promises into English.

4.3 Measuring Tracking Reality

To detect the presence of trackers during our website visits, we use the browser extension Ghostery Insights [12], which is maintained by Ghostery, a well-known privacy protection company that offers anti-tracking and ad-blocking software. The browser extension exposes trackers during a website visit by logging network traffic and comparing embedded third-party requests on the website with their database of known tracking domains. It detects trackers using various technologies, such as third-party cookies or tracking pixels.

The browser extension allows for extracting two files for each visited webpage containing information on the detected trackers. The first file offers detailed information, such as the specific tracker domain linked to the tracker, while the second file provides a summary, including tracker names and tracker categories. The tracker categories are closely related to the tracker’s functionality for the website, such as advertising, site analytics, social media, or essential [13], and have been used in previous research settings on online tracking (e.g., [18, 23]).

The tracker categories are relevant to our setting for two purposes. First, it allows us to compare whether the privacy promise aligns with the actual tracking practice, particularly in pay options involving a privacy promise excluding specific tracker categories. For instance, if a website promises a pay option “without ad tracking,” we can measure the presence of trackers from the advertising category to compare the promise with the tracking reality.

Secondly, the tracker categories enable us to distinguish between trackers and essential third-party services, with the latter being utilized by websites for technical purposes, not for tracking user behavior. To identify and exclude such essential third-party services from our analysis, we refer to Ghostery’s “essential” category, described as “Site requests that may be critical to website functionality, such as tag managers and privacy notices” [13].

Additionally, we acknowledge the technical necessity for websites to perform site analytics, provided these third-party services are privacy-friendly. We refer to the list published by the French Data Protection Authority [10], which includes third-party site analytics services that can be configured to fall within the scope of the consent exemption of the ePD and produce anonymous statistical data only. Moreover, we exclude the privacy-friendly third-party site analytics service of VG Wort, a German non-profit organization distributing royalties on behalf of authors and publishers. The latter’s sole purpose is to derive the aggregate number of page views for an article, and it qualifies for the consent exemption of the ePD as outlined by the provider VG Wort [27].

In summary, we exclude from our list of detected trackers 1) those identified as part of Ghostery’s “essential” category and 2) privacy-friendly third-party site analytics services, including VG Wort and those listed by the French Data Protection Authority [10]. Specifically, the latter list includes the third-party site analytics services of AT Internet, Etracker, Matomo, and Piwik Pro, which were identified during our data collection. However, essential third-party services still represent a risk to users being tracked due to a wrong configuration of the third-party service. Thus, we also conduct robustness checks by including essential third-party services in an additional analysis, particularly those classified as essential by Ghostery.

Regarding the procedure, we measure each website’s tracking activities 1) before making the choice between the pay and tracking option, 2) after selecting the tracking option, and 3) after purchasing the pay option, allowing us to evaluate the pay option’s privacy advantage compared to the other scenarios. Further, we manually visit each website to reliably ensure the selection of each alternative option and the proper login to the pay option. Using a clean browser instance, we first detect tracking activities before making the choice between the pay and tracking options. Next, we select the tracking option and detect tracker usage on the website’s start page. Since tracker usage on subpages like news articles may differ from the start page, we further analyze trackers on two additional subpages. We picked these subpages randomly among those featured on the website’s start page. Third, we again open a clean browser instance and purchase the pay option. We then log in to the pay option and measure the tracking activities on the website’s start page and the same set of two subpages. After collecting data on websites’ tracking activities, we reviewed the list of detected trackers and removed the previously described essential third-party services for our main analysis.

Despite the advantages of capturing tracking across various tracking technologies and distinguishing between trackers and essential third-party services, the presented approach has some limitations. First, we may not detect all trackers, for instance, due to the collection procedure only involving a subset of the website’s subpages or the browser extension Ghostery Insights relying on lists of known tracking domains. Although the latter is a well-known approach in the research community (e.g., [6, 14]), tracker companies may circumvent detection, for instance, by registering new domains that are not yet included on these lists [9]. Second, although embedding a tracker into a website enables the company behind the tracker to collect and process data, we do not always know whether the company actually stores and uses such personal data. However, embedding trackers increases users’ vulnerability to potential misuse or unauthorized access, thus harming privacy.

5 Results of Empirical Study

5.1 Sample Description

We refined the dataset by excluding 49 websites from the initial list of 341 German-catering websites. Out of the 49 excluded websites, 43 redirect users to other websites in our sample, one website did not use a pay-or-tracking wall, and another did not cater to a German-speaking audience - the other four exclusions comprised websites with technical issues during the purchase of the pay option. Despite efforts to resolve these problems by contacting the websites, they remained unresolved or unanswered. The final sample consists of 292 websites.

Additionally, we faced technical problems after purchasing the pay option on several websites, which we outline in more detail in the Appendix A in Table 2. Some issues involved websites displaying the banner for the pay-or-tracking wall even after logging in to the pay option. However, most websites resolved this problem after we contacted them. Moreover, some websites lacked a dedicated login button for the pay option on their pay-or-tracking wall. Notably, three websites suggested clicking the “accept tracking” button as a workaround to either remove the persistent banner of the pay-or-tracking wall or gain access to the login button for the pay option on the start page.

As depicted in the left upper panel of Fig. 2, the final sample of 292 websites consists of websites from Germany (92.1%), Austria (7.2%), and Switzerland (0.7%). Regarding the website industry (depicted in the right panel of Fig. 2), the majority concerned News (23.3%), followed by Business (16.8%), Computers and Technology (14.4%), and a variety of other industries (45.5% in total and depicted in more detail in Fig. 2).

Moreover, we observe two types of pay-or-tracking wall offerings. As portrayed in the lower left panel of Fig. 2, the first one comprises websites selling their “Own Pay Option,” allowing users to access only one or a few websites of a single publisher (36.6%). The second type involves a group of websites offering a joint subscription for the pay option, allowing access to all participating websites. The group is managed by an independent provider known as Contentpass, which charges a collective fee for the pay option and enables users to access multiple websites from various publishers. We refer to the latter group as “Multi-Website Provider” (63.4%).

Fig. 2.
figure 2

Sample description by website country, website industry, and type of pay-or-tracking wall offering. Notes: \(\hbox {N} = 292\) websites. Website industry and country classification adopted from Morel et al. [21]. “Multi-Website Provider” refers to websites participating in the group of websites offering a joint subscription for the pay option. “Own Pay Option” refers to websites selling their own pay option, allowing users to access only the website(s) of a single publisher. Percentages may not sum up to exactly 100% due to rounding.

Full size image

5.2 RQ1: What Do Websites Promise Users Who Purchase the Pay Option?

As illustrated in Fig. 4, we identified 17 distinct formulations of privacy promises for the pay option among the sample of 292 websites. The most prominent formulations include “Without Personalized Tracking” (29.1%), “Without Ad Tracking” (22.6%), “Completely Free of Personalized Tracking” (20.2%), “Without Ad Tracking and Marketing Cookies” (9.9%), and “Largely Free of Tracking” (7.5%). Various less prominent formulations comprise 10.7% in total, as depicted in more detail in Fig. 4.

In terms of clarity, the promise “Largely Free of Tracking” stands out as a vague statement. Additionally, some formulations, such as “Without Personalized Tracking,” include the term “personalized,” the meaning of which remains unclear. Regarding content, most privacy promises involve a general commitment not to track users. In contrast, the remaining privacy promises only exclude tracking activities related to advertising, representing a weaker privacy promise than not tracking users.

We utilized the difference among the promises’ content to manually categorize the 17 distinct formulations of privacy promises into the two groups: those with a general commitment of “No Tracking” and those with a weaker commitment of “No Ad Tracking.” As portrayed in Fig. 4, the “No Tracking” category comprises 65.4% of websites, while the “No Ad Tracking” category accounts for 34.6%. Although we classified the formulation “Largely Free of Tracking” under the “No Tracking” category, we will also analyze this vague statement separately in our subsequent analysis when comparing the privacy promises to the tracking reality.

5.3 RQ2: What Tracking Do Websites Implement in Pay-or-Tracking Walls’ Choice Alternatives?

In Table 1, we report on the websites’ tracking activities before making the choice between the pay and tracking option, after selecting the tracking option, and after purchasing the pay option. Firstly, we do not detect trackers for a share of 76.4% of the websites before making the choice. The remaining share of 23.6% tracks users even before users choose between the pay or tracking option. Unsurprisingly, we detect trackers in all websites’ tracking options (100.0%).

Regarding the pay option, we do not detect any tracker for a share of 62.7% of the websites. Further, in preparation for the latter comparison with the privacy promises, we measured whether the pay options include trackers from the advertising category, which we refer to as “ad trackers.” The analysis reveals that a share of 12.0 % of pay options includes no ad tracker but other trackers, such as for site analytics. The remaining 25.3% of pay options have ad trackers (and possibly other trackers). While trackers can be present in the pay option, their number is lower compared to the tracking option. The number of trackers typically ranges between 0 and 3 in the pay option, referring to the lower 5% and the upper 95% percentile. In comparison, the tracking option typically includes between 6 and 91 trackers.

Table 1. Descriptive statistics of tracking activities per option.
Full size table

In the Appendix A in Fig. 5, we further analyze the prevalence of each tracker category in each option (i.e., before making the choice, pay option, tracking option) in more detail, including essential third-party services not considered as trackers (e.g., tag managers). If the website’s pay option and before making a choice include tracking, the detected trackers are typically part of the advertising or site analytics category. The tracking option may include additional tracker categories, such as customer interaction or social media.

Moreover, we analyze those websites embedding tracking in the pay option and before making the choice in more detail. As portrayed in the right panel of Fig. 3, most websites embedding tracking in the pay option include one (55.0%) or two (24.8%) trackers. Thus, for around 80% of the websites still embedding tracking, a tracking-free pay option could be achieved by removing only one or two trackers. Similarly, the left panel of Fig. 3 provides an overview of tracking before making the choice, revealing that most websites embedding tracking incorporate one (55.1%) or two (23.2%) trackers.

Fig. 3.
figure 3

Number of trackers for websites still embedding tracking (before making the choice and pay option). Notes: \(\hbox {N} = 69\) websites (before making the choice), N = 109 websites (pay option). “Before Making the Choice” based on tracking measured on the starting page. “Pay Option” based on tracking measured on the starting page and two additional subpages. Excluded trackers: trackers from “essential” category, AT Internet, Etracker, Matomo, Piwik Pro, and VG Wort.

Full size image

5.4 RQ3: To What Extent Do Websites Keep Their Privacy Promises?

When assessing the extent to which websites keep their privacy promises, we assume that the website should not involve any tracker to keep the promise of the “No Tracking” category. Similarly, promises of the “No Ad Tracking” category require websites not to embed ad trackers.

As depicted in Fig. 4, the analysis reveals that a share of 32.9% of the websites do not keep their pay option’s privacy promise. Despite websites breaching their privacy promise, some websites provide more tracking protection than promised. For instance, those websites that only commit to “No Ad Tracking” but, in fact, do not track the user at all. Further, we specifically examine websites that use the vague privacy promise “Largely Free of Tracking.” Our findings show that these websites do not embed trackers, effectively making them tracking-free.

Next to our main analysis revealing that a share of 32.9% of the websites do not keep their pay option’s privacy promise, we also conduct a robustness check by including essential third-party services, particularly those classified as essential by Ghostery. As depicted in the Appendix A in Fig. 6, the share of websites not keeping their privacy promise would increase to 36.3% when treating third-party services from Ghostery’s “essential” category as trackers.

We conduct further analysis to identify heterogeneities between websites. While the websites’ industry and country do not indicate clear differences, the two types of pay-or-tracking wall offerings do. The websites that sell their own pay option, each providing users only access to one or a few websites, break their privacy promise with a share of 60.7%. Conversely, websites participating in the multi-website provider offering a joint pay option allowing access to multiple websites break their privacy promise with a significantly lower share of 16.8%.

We contacted Contentpass, the company behind the multi-website provider, to learn more about the possible reasons why participating websites better safeguard users’ privacy than others. The company stated that this is likely due to their ongoing external control of website tracking activities, which include running daily crawls to detect trackers on participating websites and requesting them to remove trackers in their pay options.

Fig. 4.
figure 4

Privacy promise vs. tracking reality. Notes: \(\hbox {N} = 292\) websites. “Privacy Promise Formulation” refers to the collected promises associated with the pay option. “Privacy Promise Content” describes the categorized content of the privacy promises. “Tracking Reality” shows the actual tracking activities detected after selecting the pay option (excluded trackers: trackers from “essential” category, AT Internet, Etracker, Matomo, Piwik Pro, and VG Wort). “Privacy Promise Kept?” shows whether the “Privacy Promise Content” aligns with the “Tracking Reality”. Percentages may not sum up to exactly 100% due to rounding.

Full size image

In response to why websites keep embedding trackers in the pay option, Contentpass reports that website developers and editors often unintentionally integrate trackers due to their lack of awareness of the trackers’ data collection practices. Moreover, many websites have evolved organically and were not originally designed with the explicit objective of prioritizing user privacy.

6 Summary of Results and Implications

6.1 Summary of Results

This paper analyzes whether websites using a pay-or-tracking wall keep their privacy promises. Data collection preceding the analysis shows that some websites have implemented pay-or-tracking walls inadequately. While the implementation problems only concern a minority of websites, we could not purchase the pay option on four websites despite contacting them. Moreover, we encountered several problems after purchasing the pay option on some websites, such as no dedicated login button for the pay option on the banner displaying the pay-or-tracking wall.

The data analysis of 292 websites reveals that websites use 17 distinct formulations for their pay option’s privacy promise. The content of these promises falls into two categories: “No Tracking” and the weaker promise of “No Ad Tracking.” Regarding tracking, we detect trackers in 37.3% of the websites’ pay options, while 62.7 % are tracker-free, thus truly safeguarding user privacy. The websites that still integrate tracking in the pay option typically embed one (55.0%) or two (24.8%) trackers. Moreover, we find that 23.6% of the websites track users even before making the choice for the pay or tracking options. When comparing the pay option’s privacy promise with the actual tracking reality, we find that 32.9% of websites do not keep the privacy promise associated with their pay option.

The group of websites offering a joint subscription for the pay option (referred to as “Multi-Website Provider”), allowing paying users access to all participating websites, noticeably keeps their privacy promise more often than websites selling their own pay option. This difference may be explained by the ongoing efforts of the group to detect tracker usage on participating websites.

6.2 Implications and Conclusions for Websites

We observe that 37.3% of the websites’ pay options are not tracking-free, and 32.9% do not keep their privacy promise, so websites should review their tracking practices criticallyFootnote 2. For 80% of the websites still embedding tracking, a tracking-free version could be achieved by removing only one or two trackers. Conversely, the websites embedding even more trackers in the pay option should comprehensively overhaul their tracking practices. As shown by the example of the multi-website provider, websites may achieve a better offering by implementing an ongoing tracker detection mechanism. Furthermore, making the results and actions from such monitoring transparent to users and regulators would enhance trust and transparency.

Regarding the privacy promises made to paying users, websites that currently use the weak promise of “No Ad Tracking” but do not track paying users should consider changing to the more privacy-enhancing promise of “No Tracking” to describe their offering more accurately. Similarly, websites that use the vague promise of “Largely Free of Tracking” and do not track paying users should consider changing to the promise of “No Tracking.” While consent management platforms often provide the technology for the banner, websites can typically customize the text on a pay-or-tracking wall to suit their specific needs, allowing them to make these changes accordingly.

Moreover, websites typically embed essential third-party services, such as tag managers or site analytics tools that produce anonymous statistical data only. However, websites rely on these third-party services to handle the transmitted information responsibly. Consequently, websites should consider requiring such essential third-party services to undergo an external audit of their source codes to demonstrate they are privacy-friendly and handle the data responsibly.

6.3 Implications and Conclusions for Users

Most importantly, 37.3% of websites’ pay options are not tracking-free, and 32.9% do not keep their privacy promise. Thus, users choosing the pay option risk wasting their money. Users who want to safeguard their privacy may need to continue using additional (and often cost-free) privacy protection tools, such as browser extensions blocking trackers. Additionally, 34.6% of the pay options come with the weak privacy promise of “No Ad Tracking” instead of “No Tracking.” Consequently, users considering a pay option should thoroughly examine the website’s offering.

6.4 Implications and Conclusions for Regulators

Our findings reveal that 32.9% of websites fail to uphold their privacy promise to paying users, and 37.3% of websites’ pay options are not tracking-free. These results show that implementing tracking-free websites remains challenging. Regulators may consider supporting (or even mandating) websites to implement measures to safeguard privacy more continuously, such as an ongoing tracker detection mechanism. Such measures may help improve user privacy and foster a fair, competitive landscape among websites. The latter becomes particularly relevant for regulators to maintain an equal level playing field between websites devoting more resources to ensure user privacy and those that do not.

Moreover, users interested in purchasing a pay option cannot be sure whether a website’s pay option is entirely tracking-free. As tracker detection is complex, users may benefit from a standardized privacy label that externally validates whether a pay option is privacy-friendly. Such a privacy label could also include an open-access repository, such as a website, that transparently publishes the results of the (continuous) privacy checks. Regulators should consider initiating or supporting such initiatives to improve transparency and user trust. Some regulators have already taken first steps to increase transparency for users, such as the French Data Protection Authority, which offers a browser extension enabling users to identify cookies stored on their browser [11].

Lastly, the adoption of pay-or-tracking walls is fairly advanced in some countries, such as Austria and Germany. Despite the latest discussions on the general compliance of pay-or-tracking walls under European privacy laws [8], this trend may expand to other European countries with similar regulatory environments. Thus, other national regulators should prepare for the privacy challenges that arise with the advent of pay-or-tracking walls.