Reflections: Bringing Privacy to Practice

We interviewed a panel of 13 applied researchers to understand why applied and academic privacy researchers do not collaborate more often. While many agree about the benefits of collaboration, they simply do not collaborate due to real and perceived barriers, such as timelines, goal differences, and data-sharing difficulties. We synthesize the findings and provide actionable recommendations to help bridge the gap between academic and applied research.


Introduction
In our work across academic and applied settings-from research agencies to inhouse research teams, from scrappy start-ups to established organizations-we have identified a hole. Academics and applied researchers are not collaborating, and those collaborations that exist are rare. The collaborations are rare enough that we, and the editors of this book, decided to raise awareness by publishing this book. We hope you, the reader who has made it to the final chapter, agree.
One might wonder why bridging applied, or "industry," and academic privacy research is an important issue. As the other chapters in this book have demonstrated, academics are tackling big, important privacy issues. However, they are not necessarily the same big, important privacy issues that we in industry attempt to tackle. We believe it is essential to work together: 1. In order for applied privacy researchers to utilize the foundation that privacy academics have built. This can inform our very fast-moving applied work.
2. In order for privacy academics to understand the problems that applied industry is facing. This can make academic research farther reaching and applicable to real-world problems.
In order to provide a broad and balanced opinion informed by diverse experiences, we believe collaboration is essential. Industry folks are moving fast and are often thinking about issues in an applied context, which sometimes varies from academic researchers. Academics, on the other hand, have the wealth of knowledge from both their own past work and that of others and the time to ponder issues at length. It is only by collaborating that we can get these diverse viewpoints in the same room.
There is a lot of distance between privacy in the books and privacy on the ground. Scholarly debate about privacy is important-it helps guide discussion.
-Trevor Hughes, International Association of Privacy Professionals In an effort to share the landscape, we believed it was important to talk to fellow applied researchers working outside of academia and to synthesize their views alongside ours. We conducted one-on-one interviews with a panel of 13 applied researchers who have engaged in academic collaborations and/or other cross-industry research endeavors. Further, we solicited input and feedback from the Privacy Research Teams at Google and Facebook. What follows in this chapter is a synthesis based on these researchers' and our experiences, advice, and hopes for the future of privacy research across academic and applied contexts.
Our panel consisted of: • Anja Dinhopl: Google Privacy Safety & Security Team. Anja is a UX research manager at Google, responsible for ensuring that privacy settings are communicated in the most understandable and intuitive way to users under the age of 18. Anja previously worked at Facebook, leading a research team focused on understanding how children, teenagers, and families use and get the most from online services. • Carol Smith: Carnegie Mellon University's Software Engineering Institute.
Carol worked previously at Uber ATG (self-driving). In a 20-year career spanning work across multiple industries and now working in academia, Carol has a unique point of view between them. Since 2015, her work with AI systems has increased her concerns about preserving privacy. In response, she has been developing methods and tools to support responsible system development.

Applied Research Benefiting from Academic Research
Applied researchers often need to move quickly-our science is in service of informing upcoming decisions about products, programs, policies, etc., and those outcomes come with hard deadlines. However, this also means that we cannot always comprehensively investigate a given topic from "top to bottom" due to time constraints. Industry benefits from working with academics because it can create the opportunity to look at a problem more objectively and over a longer term. In order to benefit from this long-term academic thinking, the work that applied researchers and academic researchers do together needs to be scoped to occur over several years-a true long-term partnership.
The deep knowledge and experience in the specific space is so good for us. We get deeply expert in one space, but then we get moved to another space and become an expert at that. We are never deeply understanding a space like academics.
-Rebecca Destello, Facebook Another way that applied researchers can benefit from academic research is via academics' expertise in working with particular topics or sensitive populations (e.g., older adults, teens and children, people experiencing cyber abuse). As discussed in Chap. 15, vulnerable populations face unique privacy risks that not only challenge designers' preconceptions about privacy but also are often overlooked in decisions about privacy design and policy. Further, existing frameworks often overlook the privacy concerns of people who experience heightened risk. Applied sectors have a vested interest in serving diverse consumers or clients well; however, that doesn't always mean that the right course of action is for applied researchers, many of whom have more generalized research experience, to reach out directly to potentially vulnerable populations for research. Rather than industry trying to break into communities, it's better to work with people who are trusted in those environments. Many times, those are the academics.
At Walmart, I studied how people budget in preparation for holiday shopping. I learned that people usually set a budget for different categories, ranging from bills and rent to savings, entertainment and "me-money." Some people used envelopes for cash for food, bills, transportation, etc. to categorize expenses. Customers part of a different segment did not use envelopes, but they also had ways to categorize and control expenses, for example, one checking account and three savings accounts-one for emergency, one for kids education, and one for monthly unexpected events like if the washer broke. People had a way of controlling their budget, whether physical or in their heads. While analyzing the data I collected, I did a quick review and learned about a solid concept in the literature, called 'mental accounting,' that described exactly what I had been seeing in my studies.

Academic Research Benefiting from Applied Research
Some of the problems applied research is positioned to solve can benefit academics because of the potential for impact. Many of these problems are the exact "big questions" that motivate academics to study privacy in the first place-e.g., How should technology companies protect the privacy of potentially vulnerable people? How should governments collect census data in ways that promote necessary social programs but also responsibly collect, use, and store sensitive data? How should IoT technologies provide in-home, everyday value to people's lives while also protecting their privacy? Academics' opportunity for impact in the applied sector-to influence the experiences of billions of people's lives on a daily basis-is unparalleled. Another way that academics can benefit from such collaborations is access to resources. Despite developing rigorous theory-derived hypotheses in academia, academic researchers sometimes lack the resources to appropriately test their hypotheses with large enough or diverse enough samples. One way that academic researchers may benefit from applied research is to capitalize on the possibility to (a) collaborate, and thus have access to larger samples (with appropriate privacy protections), and/or (b) bidirectionally replicate work with multiple samples across academic and applied settings. In addition to sheer sample size, an additional benefit of applied research is the ability to access diverse, global samples through resourceintensive methods.

How Academics Can Best Contribute to Applied Outcomes
Until there is a direct application for academic research results and they influence our ability to build something, they simply won't be used. This means that in order for applied researchers to benefit from academic privacy research, it is paramount that academic researchers do not just imply or suggest how their work "might" or "could" be applied, but they need to take the next step to understand and to collaborate with the industries they seek to influence. This type of impact starts with academics gaining as much context as possible about the applied sectors they seek to influence. Industry researchers do not have unlimited manpower, technology, etc., to create the solutions academics are calling for, and this lack of consideration for the business reality of solving real-world problems can make implementing academics' recommendations untenable. Academics should reach out to applied researchers and ask questions about how their work can influence applied outcomes. Then, they can incorporate that knowledge into their future research and especially into their recommendations for applied sectors.

Successful Relationships
Some have mastered this, and it is up to the rest of us to learn and follow suit. For example, Anja Dinhopl, a UX Research Manager at Google, shared an example of a successful partnership between applied researchers and academics. For research she conducted, she collaborated with academics for foundational work that she and her team then wanted to build on, rather than starting from scratch. She said that the applied team was "able to move much faster than most industry groups." She also argues there need to be feedback mechanisms between communities, industry researchers, and academic researchers. Ideally, this would include a way to share back to the community and a stronger commitment for industry to share with academia and legislatures and maybe even competitors. This can be done via white papers, collaborative meetings, conferences, published articles and blogs, and more. There is a need to give transparency of why we are doing what we are doing, and with that comes building on existing research. We cannot simply conduct research in a silo, but we must share with other researchers and organizations so others can learn as well.
Heather Desurvire from Google also shared an example of a successful partnership. When she conducted game user research, she saw a need for partnering on a recurring problem in many games she worked on. While she ran a research agency, she recognized that she was answering the same problems: AAA game companies were producing games that were showing the same critical player issues over and over. For example, the game tutorials were consistently too open world or too pedantic-neither are optimal for learning the tools to play the game. There was a need in industry to find a more principled way to create optimized designs. And the only way to do so properly and to have impact industry-wide was to understand the principles behind optimizing the player experience. Heather was able to offer consistent insights to all the game studios. She partnered with academics she had met at conferences, through her association with USC and a fruitful collaboration with the Microsoft games research group who were also geared toward raising the bar for all game studios via shared knowledge. Heather conducted theoretical work with authors and academics and shared it back to the game companies. Together, they came up with principles. They then had the task of actually applying them. This occurred eventually via publications in conferences and game research books, as well as teaching them to up-and-coming game designers, as a faculty member at USC's Interactive Media and Games department. An example of a play principle they developed is: (1) the game presents overarching goals early as well as shortterm goals throughout play, (2) the game does not put any unnecessary burden on the player, (3) mechanics are easy to learn but hard to master, (4) the player should not lose any hard won items, and (5) the game world reacts to the players and remembers their passage through it.
Another way Heather bridged the gap was to take a faculty position, teaching future game publishers game user research. By having future designers understand the player experience, it was a "way to plant a seed on how to optimize player experience by teaching them the concepts." To Heather, planting the seeds was a way of raising the bar. And she has been teaching now for over a decade and has seen those seeds flourish in industry. In fact, many game researchers in the gaming world, who were there since the beginning, came from academia. That bridge happened naturally in the gaming industry, as a result of this dual focus. Heather and these others helped define and pioneer the game research methods that are now industry standard.
Ideally we should work towards a multi-partnership-academia, government, non-profits, and industry. Academia is known for methods, detailed inquiry, and long-term goals. Applied research focuses on how findings impact prioritization and development of products, programs, processes, etc. Government makes the rules, but may wrestle with the right approach when designing privacy experiences is a complicated endeavor without obvious answers. Government agencies may not always trust companies to come up with design principles, but overly detailed mandates can also be problematic or restrict innovative problem-solving. While academics, policy folks, and industry researchers seem to have conflicting interests, they actually are all working toward common goals of helping people benefit from using products while protecting their privacy. Each sector differs in how they prioritize and work toward these goals. We simply have a fundamentally different way of framing things. But in the end, we are all working toward the same outcomes and can help one another get there if we work more closely together.
Companies face a variety of commercial, legal, ethical, and reputational risks that serve as disincentives to sharing data for academic research, with privacy-particularly the risk of reidentification-an intractable concern. For companies, striking the right balance between the commercial and societal value of their data, the privacy interests of their customers, and the interests of academics presents a formidable dilemma.
To help support data sharing for research by companies, we have developed model contracts, a special ethics review committee for projects not covered by IRB review and an award to honor successful industry-academic teams that collaborate using company data safely. We hope this will move the ball forward.
-Jules Polonetsky, Future of Privacy Forum

Why These Partnerships Are Challenging
So why don't more of us foster these relationships? Why aren't all applied researchers partnering with academic researchers? There are many reasons, but one that stands out to us and many in our panel is timelines. Simply put, the timelines that academic researchers and applied researchers work with are different. Applied researchers are moving quickly, sometimes conducting a project from start to finish in just a few weeks. Academic researchers, on the other hand, take much longer in order to conduct research that can lead to publication of peer-reviewed results. Projects can last anywhere from a few months to many years. This discrepancy poses a bit of a problem. It is harder to adopt what you learn in academia in industry because we need to do it fast-we aim to solve immediate problems. In addition, because academia aims for knowledge that is generalizable and scrutable by third-party reviewers, there is a general sense that there are so many steps one has to go through to conduct academic research, such as IRB approvals, statistical testing, theoretical validation, and generalization rather than focused and contextual findings. Heather Desurvire from Google serves as a bridge herself, and she shared a story about her experiences working with a new field: "Academia funded the research, and the work was adopted because the field was brand new. There were no real methods in industry. We applied what we learned in academia in industry." But it was not always easy, and everything was not readily accepted. "Some work was criticized as not statistically valid, and some qualitative research was looked down upon." But she did not stop. While the early work received some initial criticism, over time the qualitative work was appreciated for its depth, and the academic partners utilized it. It is often important to use both qualitative and quantitative work together, for example, when studying gaming and the player experience: "You can look at quant, but need the qual to understand WHY. This was taking place in industry, not academia. Academia did the theoretical work, not the applied work." And if it is not something you can eventually use, why is the work being conducted at all?
Applied researchers are working on improving products, and while time and other constraints play a role, is it simply that applied researchers don't care about the theoretical work? Well, not necessarily. Gretchen Gelke started working on privacy at Google about 3 years ago. Prior to this role, she worked on security software, hardware, and smart devices-privacy and data have been key issues for her for a long time. Compared to other companies, her privacy specialty was never really the focus of her role . . . until Google, where she realized her specialty was an asset. The team she joined, while very applied, was also extremely academic. They focused on products (e.g., helping users find privacy settings) and also foundational research both across Google and externally. The team is frequently asked to present the foundational work to product teams to inform thinking and inspire change in products, overarching strategy, or to inform entirely new directions. The team hired people with PhDs in privacy and security to help inform policy and set context for the product changes being recommended. These efforts were critical for Google. Having the academics in house helped the team make better applied product decisions.
Unfortunately, applied research teams reinvent the wheel constantlyconducting research that has likely already been conducted in academia or by previous colleagues. Sure, it would be easier to go to someone who has done the work, but we want to make sense of it too. So how do we stop reinventing the wheel and shift to doing this together?
So what do we do? Do we abort the mission of trying to bridge? Of course not; we certainly don't think so (or we would not be writing this chapter). We believe that collaborations are essential for building on existing work and applying that work to real-world problems. Academic researchers have been studying many of the things we are grappling with in applied research for years! We need to figure out how to make those timelines match up. If we have problems we are interested in examining in a year or two, it would be nice to be able to predict that so academics could get a head start. Applied researchers need to be proactive, to predict future needs, to move earlier to foster those collaborations.

Actionable Strategies for Making These Partnerships Work
In order to properly bridge, we believe that industry researchers need to be mindful about what academic institutions are worried about, and academic researchers need to be mindful about what industry researchers are worried about. We need to make sure the collaboration is good for both sides. We believe we need to also work against some of the biases, for example, the incorrect stereotype that companies do not actually want collaboration, they just want to be told they are right. We believe we need to build a strong relationship model that simply does not exist for most of us right now. So how do we actually DO that? How do we better understand and communicate with each other? We have identified some actionable strategies that have worked for us and our panel, and here we synthesize them for you, the reader who would like to implement these strategies.

Provide Funding Sources
We heard time and again from our panel that one way that applied privacy researchers learned about academics' work is through participating in industryfunding programs (e.g., serving as a reviewer for proposals to a private sector research grant). Relevant research may be discovered this way, and to further academic research so that we can all make more progress, industry should help fund it. This helps academics accelerate their work, which also helps industry get to results quickly. For instance, rooted in the advocacy of internal researchers, Facebook launched a new research funding proposal to fuel academic research on inclusive privacy (see https://research.fb.com/programs/research-awards/ proposals/peoples-expectations-and-experiences-with-digital-privacy-request-forproposals/). As a part of this funding, awarded academic researchers provide updates to Facebook research partners throughout the progression of the research so that those working on related, applied questions at Facebook can benefit quickly from the latest results. Additionally, the academic researchers can learn from how their industry counterparts are considering using the research to shape product development in order to make their work and its implications more valuable to the applied sector upon publication. Touchpoints should be created during academicindustry partnerships so that applied researchers and academics can learn from one another.
Likewise, Mozilla has funded privacy research grants over the last several years with diversity and transparency at the heart of this program. Privacy research that resulted from this funding led to direct impact on Mozilla's products such as changes to what Mozilla displays when one uses private browsing mode. In addition to funding research practices, funding academics in nontraditional ways can also greatly contribute to productivity and mutual benefit, such as providing funds for childcare over the course of the grant.
Sure there are existing grants, but are they good enough? A true program/collaboration would involve long-term relationships, like conferences or "labs" that connect academia and industry, and they would involve ongoing commitments, not just a project in a single point in time. Current grants are often perceived as pertaining to a specific topic and project goal. We need to foster relationships and open the time and space for ongoing relevant questions. Companies can also give money to foundations or other organizations who can then determine and fund research based on their own values. This type of intermediary can be important to be sure the results are not influenced or perceived to be influenced by the industry group who is funding the work.

Invest in the Next Generation of Scholars
Applied researchers can align their efforts with privacy academics' overarching goals by investing in the next generation of privacy scholars. Because academic researchers spend a great deal of time mentoring and teaching students, focusing joint energy on bolstering student skills and experiences will foster better collaborations with applied researchers, not just now, but into the future. For example, many large technology companies fund capstone course projects for graduate and undergraduate students studying computer science. Students receive a real-world problem that applied researchers are currently facing and are provided resources to study related concepts and report back with proposed solutions at the end of the semester. Importantly, these courses allow for cross-academia and applied sector collaboration during which both applied researchers and professors provide guidance to students.
Applied researchers should be actively involved in the training of future privacy researchers in academia, regardless of whether students end up going into applied or academic careers. Providing students with context about applied problems, solutions, and careers will ultimately strengthen their research and impact across privacy science as a whole. For example, both Facebook and Google invest in conducting "Research Jams" and other collaborative hands-on workshops with industry researchers and students at different universities aimed at developing rigorous applied research plans about privacy topics. These events provide students with exposure to solving applied problems through research as well as to applied research career trajectories.
In my work, I get to partner with professors and students on projects-they do research on their own, and I bring it together for our government customers. I advise on the work, and sometimes collaborate, but there is still separation. The strongest partnerships I've seen between academic and industry partners are through CMU's Human-Computer Interaction Institute's Capstone courses. As both a teacher and an advisor, I've been thrilled to observe student teams successfully collaborating with corporations and nonprofits to improve all types of experiences. In these situations, they are sharing and learning from each other resulting in positive outcomes including employment for graduates.
-Carol Smith, Carnegie Mellon University

Sharing Work
Both academic researchers and applied researchers need to find ways to regularly and relevantly share their work with one another. Academic privacy researchers should consider publishing summaries of their work for mainstream audiences, send relevant papers to applied researchers they have previously connected with, and present at both academic and applied research conferences. We may need more and new mechanisms to do this well. For instance, perhaps we need to expand the open source and open data movements into the privacy sector and create shared repositories of both academic and applied research by topic. Although there are challenges to doing so (e.g., private sector confidentiality), there are ways to start to build these bridges. For example, the Facebook Privacy Research Team has partnered with Trust, Transparency, and Control Labs to share privacy research insights with external audiences so all can learn from one another. In addition to these types of forums being useful for sharing results across academia and applied research, applied researchers across different companies and industries, in and of themselves, also benefit from shared knowledge bases. Members of our panel described times when academic partnerships hadn't worked out, when there were concerns or disagreements in how results would be shared at the end of a project (e.g., one academic wanted to publish based on collaboratively collected data without having the partner applied researcher review the final product). These disagreements can lead and have led to collaborations failing. Thus, it's vital that applied researchers scope collaborative or funded academic research projects with publication as the ultimate goal and that both sides of any collaboration talk openly about and agree to publication guidelines (in writing) at the very beginning of the project.
Lastly, academics can help make their work more applicable in industry settings by ensuring insight-based recommendations are actionable, plausible, and in digestible language for non-scientists. This allows applied researchers to easily translate academic findings and implications into design and engineering "languages" for stakeholders on the ground who might be implementing academics' suggestions.
Provide a '5 things you need to know about privacy' document that accompanies any academic publication. This type of document is not about dumbing down the work; rather, it is a way to translate the work for all to read, like how we should explain concisely and clearly to our users! -Julie Schiller, Google

Sharing Data and Resources
It would be remiss to not identify one of the ways that privacy academics consistently ask for partnership with applied researchers-they request data to be shared with them for their own use and analysis. On the surface, this may seem like a simple ask. However, the sensitivity of sharing consumers' data from a business or clients' data from a nonprofit generates, ironically given the topic of study, a number of privacy considerations that need to be addressed (e.g., Has consent been obtained? Can data be effectively deidentified? Will the analyses directly benefit consumers/clients?). Sometimes it is easy to answer all these questions and other necessary questions with "yes." For example, Facebook has provided aggregated and de-identified data sets to academics working on research for social good. But sometimes the answers to the privacy questions above are more fuzzy, and in those cases, applied researchers simply cannot justify sharing data.
The best practice here is not to ask these questions post hoc but instead to define how data can be shared at the beginning of a collaboration so the right security and privacy measures can be put in place. This will allow academic and applied researchers to work together to determine what inputs are essential to project goals and then work from there to determine if, when, and how data sharing will be executed.
One technique to include more academic work in applied research is to include a literature review as the foundational part of setting up projects. Julie Schiller, a UX Research Manager at Google, thinks that those who have academic training can bridge by "using the skills we were taught academically to give a thorough overview of existing work for the team to better understand the project and broader problem." Janaina Pantoja, a UX Researcher and Manager at eBay, starts most of her research with literature reviews. She takes advantage of the huge amount of research that has already been conducted on e-commerce, in general, and on eBay, in particular: "If I don't take the time to understand what is in the literature, I may not be able to synthesize and analyze my data properly. Sometimes it is about language gapwe study topics that others might have studied outside of industry, and there are concepts and terms that have been formalized elsewhere . . . sometimes you are saying the same thing that was said in academia already." One might think that adding literature reviews to the process is time-consuming and inefficient. What many applied researchers may not know is that it actually does not take that much extra time to conduct the literature reviews that Julie and Janaina find so valuable. Many articles are publically available on Google Scholar, ResearchGate, and at local libraries, and conducting a literature review upfront can save you time later.
Collaborating with academics for literature reviews is an excellent way to bridge the gap. Academics can recommend the articles to read to get up to speed quickly on a topic. In fact, putting out a little bit of funding for various literature reviews might be a good way to incentivize academics (likely students) to do this. New students are always looking for new topic areas, and this is a mini step toward shared knowledge and better partnerships without having to have done a lot of research in the area already and without having to commit to the area just yet. This is why they hire us-to know the rigor and then to make those short cuts, in ways that do not detract from the proper study.
-Janice Tsai, Google Similarly, another way to bridge is to have academics join industry. They can bring the rigor that is second nature to them to industry and teach industry folks to slow down just a bit and learn from what already exists in academia. We have seen this type of cross-over to be very successful at organizations like Facebook and Google, who often hire academics or sponsor them for a period of time (e.g., a sabbatical). Employing academics and grad students as interns and student researchers to move projects/programs forward is also a way to bridge.