Starting in May 2018, we conducted three deployments at three different scales: a small-scale pilot, a medium-scale deployment, and a large-scale deployment.Footnote 3 To date, 3400 individuals in Accra downloaded our mobile app, called DumsorWatch; 457 people installed our plug-in sensor, called PowerWatch; and over 4000 surveys were performed to directly measure socioeconomic outcomes. In December 2019, we surveyed an additional 462 participants to understand their experiences and to collect updated measures of time-varying socioeconomic outcomes.
We present the technology deployed, the design of our deployment, and where our planning and assumptions failed or caused unexpected problems. While doing so, we attempt to categorize our key challenges and describe the steps we have taken to overcome each of these challenges, emphasizing issues that arose that were more complex and/or costly than originally forecasted. We find that our experiences differed depending on the scale of the deployment, each scale uncovering its own complexities.
3.1 Innovation: Data Collection Instruments
We developed two types of data collection instruments – sensors and surveys – to achieve the goals described in Sect. 2.3. These instruments collect the required data to estimate SAIDI and SAIFI independently of utility participation.
We developed two different sensors that detect the presence and absence of grid power: an app called DumsorWatch that is installed on a participant’s mobile phone and a sensor called PowerWatch that is plugged into a power outlet at a household or business.
DumsorWatch is an Android app installed on the everyday-use smartphone of a participant who lives and/or works in Accra. “Dumsor,” the local word for power outages, was used for branding and association with power outages in the Ghanaian context. DumsorWatch automatically senses power outages and power restorations through a combination of on-phone sensors and cloud services (Klugman et al., 2014).
PowerWatch, our plug-in sensing technology, integrates power reliability sensors with a GSM radio to send measurements in near-real time to a cloud database (Fig. 6.3). By designing PowerWatch to plug into a participant’s home or business, as opposed to connecting directly to the electric grid, we avoid the need for prior approval or cooperation to deploy the sensors and therefore maintain independence from the utility, a primary goal of our deployment. PowerWatch senses power outages and power restorations timestamped to the millisecond, as well as GPS-based location, voltage, and grid frequency.Footnote 4 PowerWatch contains a battery to allow for continuous reporting throughout a power outage and will queue data if there are GSM-network-connectivity problems, to be uploaded once GSM connectivity is restored.
How Do PowerWatch and DumsorWatch Work?
PowerWatch consists of an outage detection sensor that plugs into an outlet and is installed in homes and businesses and reports the state of the grid over a cellular backhaul to the cloud. Every minute, the sensor takes a reading of power state, grid voltage, grid frequency, GPS, and cellular quality. It also records the number of nearby Wi-Fi signals as secondary validation, as wireless hotspots may be grid powered. In addition, upon changes in power state, the device records the timestamp (from an on-board real-time clock) and current acceleration. All these measurements are stored locally on an SD card and transmitted to the cloud when a cellular connection is available. Acceleration signals that a participant is interacting with the device, making it likely that any charge-state change at that time is a false positive and allowing us to more easily reject the data point. The sensor contains a 2000 mAh battery, which can run the sensor for several days, longer than most outages in Accra. When the sensor is on battery power, it still reports data at the same frequency to our servers, a feature necessary for calculating outage duration.
The primary sensors used by the DumsorWatch app are the phone’s location sensors (GPS), the phone’s charging state which monitors if the phone is connected to a power source or not (and notifies DumsorWatch when that state changes), and the system clock to give the time of any observed events. Secondary sensors in the app help refine the likelihood that a change in charge state corresponds to a power outage or restoration. For example, the accelerometer can measure if the phone was moving when its charge state changed (as it would if a charging cable was inserted or removed from the phone side), and the phone’s Wi-Fi radio can report the presence or absence of wireless hotspots. On-phone processing and analysis of microphone recordings may be able to detect the presence of a 50 Hz “hum” of grid mains. Additionally, the users of the app can manually report a power outage and power restoration by pressing a button in the app.
For both technologies, a cloud-based analytic system searches for outage reports from multiple devices to ensure the validity of an outage. To perform this search, we cluster outage reports into density-based clusters in both space and time. This lets us reject noise from a single sensor (i.e., a single participant unplugging a device or a pre-paid meter running out of credit) and ensures that only true outages are reported by the system.
A 60-min socioeconomic survey accompanied the deployment of each PowerWatch device, and all participants who received a PowerWatch device also downloaded the DumsorWatch app. A shorter survey was administered to participants who solely downloaded the DumsorWatch app. All surveys were completed using SurveyCTO, and all participants received an airtime transfer as a thank you for participation (SurveyCTO, 2019). We conducted high-frequency checks to address any obvious data quality issues. Example data collected includes:
Demographics: name, age, education, income.
Electricity attributes: appliance and surge protector ownership, usage of electricity and generators.
Recall of power quality in the past 2, 7, and 30 days.
Social media usage and perceptions of the energy crisis.
Along with providing data for the economic study, the survey was used to support the development and deployment of the technology itself. For example, the survey recorded a unique code for the PowerWatch device and DumsorWatch app and the participant’s phone number and GPS location. To inform DumsorWatch debugging, we asked how they used their mobile phones, how many phones and SIM-cards they use, and how frequently they upgrade their phones. To inform the PowerWatch deployment, we asked whether the participant turns off their electricity mains at night and whether they had any safety concerns about PowerWatch.
3.2 Innovation: Deployment Methodology
To support our deployment as scale increased, we designed and implemented a novel set of deployment management tools. While our methodology evolved to support each deployment scale, its general structure remained fairly consistent. First, we developed criteria for site selection that allows us to answer specific socioeconomic questions. Next, we devised a sampling procedure that gave sufficient coverage of each chosen site, as well as sufficient redundancy to enable cross-validation of the new measurement technology. Finally, we worked with a team of field officers to deploy in the chosen sites, employing our deployment management tools to maintain and monitor the system. The rest of this section considers each of these components in detail.
3.2.1 Site Selection
We selected a subset of the sites where infrastructure upgrades are planned (“treatment sites”) and then quasi-randomly selected a set of sites that are comparable in observable characteristics (“control sites”). For each site, we defined a geographic surveying area that is the intersection of a 200-meter radius from the site centroid and a 25-meter region extending from the low-voltage network being measured. We wanted the area to be relatively small, so that we could have a high degree of confidence that customers within the area were all connected to the same infrastructure, but it needed to be large enough to have a sufficient number of residents or firm owners for us to enroll into the study. We performed this analysis using GIS tools operating on a newly created map of Accra’s grid constructed by an independent firm that had been contracted by MiDA as part of the improvements funded by the Ghana Power Compact. Using these GIS maps, we produce a series of maps marking the geographic area bounding each site. Field officers used these maps, along with the GPS coordinates for the sites, to identify the surveying area and deploy sensors accordingly.
3.2.2 Sampling Strategy
We deployed our sensors at the home or place of work (or both, if these are co-located) of Accra residents, targeting a 50/50 split between households and firms. Installing PowerWatch at consumer plugs and DumsorWatch on consumer phones allows us to not depend on direct access to utility infrastructure such as transformers or lines and to measure power quality at the point where it is least understood: the customer (Fig. 6.1).
We planned a deployment of three PowerWatch devices and 20 DumsorWatch app downloads at each site. Our strategy is built around redundant sampling such that multiple sensors are placed under a single transformer. When multiple sensors in this group report an outage at the same time, we can be confident it was due to an issue affecting the transformer rather than a single customer. Further, when we observe sensors below multiple transformers reporting outages simultaneously, we can infer the outage occurred at a higher level of the grid. This sampling strategy is shown in Fig. 6.2.
3.2.3 Deployment and Surveying Team
We hired local staff who supported our continuously operating deployment. One team member works full time as a field manager to oversee initial roll-out and ongoing maintenance of the system and an auditor to follow up with participants who report problems or whose sensors are no longer functioning.
To implement our medium- and large-scale deployments, we temporarily employed a team of ten field officers and three team leads. Prior to the start of the deployment, the field officers were trained extensively to ensure the correct protocols were used to obtain consent, conduct surveys, plug in the power strip and PowerWatch device at the respondent’s home, download the app onto the respondent’s phone, and conduct any necessary troubleshooting related to the technologies. Field officers find potential participants, get informed consent, and screen their eligibility. They then conduct the survey, install the sensors, and answer any participant questions. We conducted multiple training exercises where each team member learned about the technologies being deployed and practiced completing the survey and deploying the technologies.
Field officers visited sites in groups of two to alleviate safety concerns. We provided team uniforms, shown in Fig. 6.4, to make clear they are part of an official project. We also provided backpacks to carry supplies, tablets to conduct the survey, Wi-Fi hotspots to upload the survey and download the DumsorWatch app, flashlights for safety, and feature phones to verify the phone numbers of participants to ensure we know where to send the participation incentives.
3.2.4 Dependence on Participants
The placement of PowerWatch sensors directly in homes and firms – where participants may unplug them, run generators, or experience power shutoffs due to nonpayment – increases the noise of our data relative to a deployment on utility-owned equipment such as transformers. Similarly, the DumsorWatch app may be uninstalled from respondents’ phones, reducing coverage and leading to a potentially under-sampled signal. A key challenge was thus ensuring that we only enrolled participants who had the ability and desire to participate for the full study duration and then to minimize any cause for participants to choose to withdraw consent from participation.
In a preemptive attempt to decrease the statistical noise caused by human factors, we screen participants for specific criteria including owning a phone with Android version 4.1–8.1 and being an active customer on the grid. In order to minimize attrition, we explain the goals, risks, and benefits of the project, as part of the consent process. Finally, we provide a phone number to call if participants have any questions or concerns.
To further encourage continued participation, we compensate participants monthly with airtime credits on their mobile phone. All participants whom we recruited to download the DumsorWatch app received 5 Ghana Cedi (0.93 USD) of airtime for initial recruitment and 4 Ghana Cedi (0.75 USD) monthly for keeping DumsorWatch installed. Participants who also installed a PowerWatch device received an additional 10 Ghana Cedi (1.86 USD) for installing the sensor and 5 Ghana Cedi (0.93 USD) monthly for keeping PowerWatch installed. Additionally, participants who have a PowerWatch sensor placed at an outlet in their home receive a power strip so that the sensor does not take up a needed outlet.
3.2.5 Deployment Management Tools
We developed three software subsystems to support the deployment: (1) an automated incentive system to transfer the airtime incentives; (2) a deployment management system to (a) track sensor and participant status and (b) display deployment health to the field management team; and (3) a data visualization and analysis system. We discuss these systems, and the experiences that led us to develop them as the deployment scaled, in Sect. 3.5.
3.3 Evaluation: Overview
For each of the small-, medium-, and large- scale deployments, we report problems that occurred, techniques used to mitigate their impacts, and the effectiveness of the mitigation. To emphasize the parallels between challenges exposed at different scales, we organize this discussion around four categories of challenges: organizational, cultural, technical, and operational. Organizational challenges relate to procurement, hiring, and finances; cultural challenges relate to how cultural considerations impacted the deployment and operation of the technology; technical challenges relate to the development, manufacturing, and functioning of the technology; and operational challenges relate to the successful deployment and operation of the technology.
3.4 Evaluation: Small-Scale Pilot
The first activity we performed was a deployment of 15 PowerWatch sensors and 5 DumsorWatch app downloads. The goal of this deployment was to validate that the technology can reliably sense power outages and transmit this information over many weeks in the field. We performed no survey work and no site selection work for the small-scale pilot: devices were not deployed with participants enrolled from the public but in the private homes of our research partners. The primary challenges were related to producing the technology, connecting the PowerWatch sensors to the cellular network, and building enough local capacity to deploy PowerWatch and DumsorWatch.
In addition to testing the technology, we worked to build relationships to support future scaling. We reached out to local stakeholders for feedback on the assumptions driving our sensor design, speaking with engineers and managers at ECG, MiDA, and several independent contractors involved in the Ghana Power Compact. We also received data from ECG that helped validate our hypothesis that their existing estimates of SAIDI and SAIFI could benefit from higher-resolution measurements.
Even at a small scale, we experienced unanticipated technical challenges. To connect the PowerWatch devices to the cellular network, we initially used SIM cards sold by Particle, the US-based manufacturer of the cellular technology used in PowerWatch, in part because these SIM cards had been advertised to work in any country. But in practice, their ability to maintain a stable network connection was worse than that of a local SIM. We therefore decided to use SIM cards from the largest local carrier (MTN), but we encountered a three-SIM-card-per-person limit upon purchase. Although we were able to circumvent this by visiting different stores, purchasing SIM cards in stores was not an option for future scale.
Another challenge was keeping SIM cards functional over the full study period. Prepaid SIM cards require data plans, which are purchased using an unstructured supplementary service data (USSD) application that can only be run from within Ghana; there is no web-based account management or top-up available. We initially solved this problem by purchasing a 90-day data plan, the longest available. This was sufficient for our small-scale pilot but would not be viable for future deployments.
3.5 Adaptation from Small-Scale Pilot Experience and Evaluation of Medium-Scale Deployment
In our medium-scale deployment, 1981 individuals downloaded the DumsorWatch app, and 165 individuals installed PowerWatch sensors. After an initial 1-week training, field officers first deployed the PowerWatch sensors, which included also administering a detailed socioeconomic survey and installing the DumsorWatch apps on the phones of PowerWatch recipients. Once the deployment of the PowerWatch sensors was complete, field officers spent 3 weeks deploying the DumsorWatch apps among an additional set of participants, which also included a shorter socioeconomic survey. Once initial deployment was complete, the monitoring activity continued for 7 months.
Unlike the small-scale deployment, this scale required implementing our full deployment design, including hiring a full local implementing team, recruiting and incentivizing participants, choosing deployment sites, extracting value from the data streams, and implementing the survey instruments. We enumerate the changes experienced as we increased from small- to medium-scale in Table 6.1, paying particular attention to the challenges extracted.
The medium-scale deployment was large enough that the financial responsibilities were significant. We had to start managing multiple monthly payments for cloud services and payments to local companies for cell network connectivity and incentive transfers. Most of this increase in complexity was ultimately handled by staff at the University of California, Berkeley, but establishing payment schedules took a large effort from the research team. Bureaucratic requirements at the university also caused frequent delays in payments, especially when payment was needed in a short time frame (1–2 weeks). Increased flexibility, for example, at an independent organization or private sector company, might better suit technological deployments that have some complexity in administrative streams like finances and hiring.
Because prepaid SIM cards were not available at the quantities we now needed, we had to enter into a contract with the cellular provider, MTN. To alleviate concerns about whether our application was legitimate, we visited the MTN main office in our university shirts, gave a technical demo, and answered questions about our backgrounds and affiliations.
At medium scale, many of the cloud-based software services our systems were built on were no longer eligible for free-tier usage. For one service, this meant that we would be unable to continue without signing a multiyear contract that extended beyond the length of the deployment. We found a workaround for this deployment by applying to a special program within the company, but in future deployments we would more carefully consider pricing models for ancillary services.
Visiting households and firms requires permission from the relevant local district assemblies. We wrote letters of introduction and visited these assemblies to receive permission. Receiving this permission also increased participant trust.
We also worked with the field officers to refine our survey design. During training activities, the field officers had the opportunity to react to questions and provide suggestions for improvement. We used this feedback to make the survey as culturally appropriate and in line with our research objectives as possible. As field officers entered the field, we received continuous feedback on ways to improve our survey and deployment procedures.
Finally, we learned that a uniform would be valuable for building participant trust. We provided DumsorWatch-branded shirts and backpacks for the field officers, so they would look official when approaching participants. These are shown in Fig. 6.4. Field officers also carried identification cards that they could show participants in case of any questions.
At medium scale, frequently visiting sensors for debugging was no longer feasible, so we prioritized sensor stability and remote failure detection and mitigation. This included developing a full custom embedded system for PowerWatch (shown in Fig. 6.3 B.1) with built-in mechanisms to reset the device on failure. Additionally, we spent considerable time implementing and testing more reliable firmware, incorporating error collection libraries, and building dashboards displaying the health of both PowerWatch and DumsorWatch. We assembled this version of PowerWatch over 3 days with the help of fellow graduate students.
Another technical challenge concerned mobile phone heterogeneity. We had little insight into what types of mobile phones and versions of Android were most common in Accra. Thus, we implemented DumsorWatch to be backwards compatible to 4.0.0, a version of Android no longer supported by Google (Google, 2018). Backward compatibility took considerable engineering effort and had side effects such as making DumsorWatch incompatible with many modern Google cloud services, including Google’s bug tracking tools, making app failures much harder to correct. Further, we chose to support older versions of Android at the expense of supporting the newest Android version at the time, Android 8.1, a decision that made those with the newest phones no longer eligible for participation. While this did not reject large numbers of participants during this deployment, as more people moved to newer devices, it would have more impact on recruitment, necessitating future engineering costs.
Finally, we experienced two challenges related to SIM card operations. First, we could not identify a way to test PowerWatch sensors in the United States using MTN postpaid SIM cards, which were not configured to allow PowerWatch to connect to cellular networks outside of Ghana. We therefore built a US-based testbed for sensor development that used US-based SIM Cards. However, to perform final assembly and quality assurance, steps that require PowerWatch to be connected to the MTN network, we needed to wait until the sensors were in Ghana for a deployment, compressing the timeline of these tasks and increasing risk if quality assurance failed. Second, MTN’s process for provisioning the SIM cards required additional oversight and took much longer than expected, which delayed deployment and made clear that a different partner would be required to manage large fleets of SIM cards assigned to a single customer.
These problems led us to continue exploring global SIM card options, and we tested a small number of Twilio SIM cards during this deployment. We found they had similar problems to the Particle SIMs previously evaluated. We contacted Twilio support and found their documented list of Ghanaian network operators was out of date, making unlisted providers unavailable on the Twilio network and leading to a drop in service quality. This theme of global solutions lacking local service quality is explored further in Sect. 4.2.
The operational challenges started with transporting our equipment to Ghana. We carried the PowerWatch sensors, power strips (handed out to participants as incentives), and equipment for field officers into Ghana in suitcases over multiple trips from the United States. PowerWatch sensors were carried on the plane whenever possible to minimize their chance of being lost. This method of transportation worked but led to multiple questions from airport security in the United States and customs in Ghana. We were able to overcome these hurdles by creating documentation about our project and providing this along with letters of invitation from MiDA, but even still this transportation method depended on our team being persistent and prepared with documentation, unwrapping all equipment from its packaging to make it look less likely to be resold, labeling all equipment with tags indicating it was property of the university and not for resale and only traveling with a few suitcases at a time. More generally, ensuring safe and timely transport of nonconsumer technology across borders will likely require additional measures depending on the local context.
Implementation of our site selection methodology required GIS maps of the electric grid. We worked with stakeholders to determine where the best maps of the grid were maintained, a task made more complicated by maps being held by multiple subcontractors of the Ghana Power Compact. With MiDA’s support we were given access to maps that, while not perfect, included enough detail for our site selection procedures. At this medium scale, which was also relatively concentrated geographically, it was feasible for a member of the research team to study the GIS maps visually, identify the proposed treatment sites, manually identify control sites quasi-randomly to match treatment sites based on observable characteristics such as grid characteristics and satellite imagery, and produce site maps that field officers could use to identify potential respondents at each site. This process met the requirements for this scale deployment but would prove exceedingly complicated for a larger deployment.
At medium scale we felt it was not feasible to transfer recurring incentives to participants by hand. We had anticipated this problem and designed an incentive-management system to support this goal. The system was designed to capture user behavior (e.g., whether a participant completed a survey, installed DumsorWatch, kept DumsorWatch installed, etc.) and to transfer airtime automatically. The actual transfer of airtime took place through a third-party API. We developed and tested the incentive transfer system alongside our deployment activities (Klugman et al., 2019).
Finally, at medium scale, the data collected were significant enough that they became valuable to stakeholders in the region. Because many of these stakeholders would be responsible for helping the project achieve further scale, we made an effort to develop and share anonymized visualizations and summary statistics.
3.5.5 Failures and Missteps
One class of failures experienced at medium scale is attributable to simple technical immaturity. For example, we found (and are still finding today) bugs both in our automated incentive-transfer system and in the third-party payment API used to incentivize participants. This API is provided by a small company, but we believed it to be the best option for transferring airtime in Ghana. Both technologies should have been more aggressively tested prior to launch. There is a clear need for a fleet of testing phones in Ghana for continuous integration and automated testing of incentive transfers. However, as with most hardware-based testing systems, this is difficult to implement in practice. As a result, most participants experienced late payments, which we hypothesize caused the significant number of DumsorWatch uninstalls shown in Fig. 6.5.
More fundamental were issues with effectively recording, connecting, and correcting critical deployment metadata, such as the location and status of each device and payment, which we collectively refer to as the state of the deployment. We had not anticipated the complexity of managing data about participants, devices, and app installs, each of which was collected by a different system and some of which informed each other.
This led to an ad hoc sharing of information through our encrypted shared drive. The field team uploaded surveys containing participant- and deployment-placement information on a daily basis. The research team downloaded and cleaned these periodically and provided the resulting CSV files to the individual engineer handling either sensor management or the payment system. Errors in the surveys (common due to typos in long unique IDs) were communicated back to the field team via phone calls and emails, and the resultant corrections in the field were not always communicated back to the research team. This process was ineffective while we were in Ghana and completely collapsed after we returned to the United States and could not focus full time on deployment upkeep. As devices moved, we received multiple, conflicting reports about their current location. As a result, we permanently lost the state of some devices; five devices are still unaccounted for. These issues continue to make data analysis, sensor debugging, and correlation of problems with specific participants difficult for the devices in this deployment.
3.6 Adaptation from Medium-Scale Deployment and Evaluation of Large-Scale Deployment
Beginning in February 2019, we built upon our medium-scale deployment and added 292 new PowerWatch devices and 1419 new app downloads in 3 districts of Accra, resulting in a combined large-scale deployment of 457 PowerWatch devices and 3400 DumsorWatch apps.
3.6.1 Organizational and Cultural
The organizational and cultural challenges did not change from the medium-scale deployment. Existing service contracts were sufficient or easily renegotiated, and the field team scaled linearly with the size of deployment.
The increased number and technical complexity of the new PowerWatch sensors constructed for the large-scale deployment precluded relying on other graduate students to help assemble devices as we did with the medium-scale deployment; however, the scale was still too small to be cost- or time-effective for contracted assembly. Our solution was to build our own assembly line and hire 10 undergraduates to assemble devices. This required us to develop discrete assembly steps, a training guide, and quality assurance techniques. The PowerWatch assembly line can be seen in Fig. 6.6. This assembly line produced the 295 PowerWatch sensors over 4 weeks and 110 person-hours of total work, with a 2.4% error rate, which was far below what we were anticipating. Although this activity was successful, difficulties in recruiting and paying students hourly, and challenges with the academic schedule, mean this model would not scale much beyond 400 units.
The larger number of sites meant site selection was no longer easy to do manually. This led us to develop a GIS-based site selection system, which generates sites based on our site selection rules, labels these sites, and creates site location images for the field officers. This system requires cleaning the GIS maps of the grid collected from the utility and was designed and maintained by a dedicated graduate student.
We continued exploring global SIM card options, using Aeris SIM cards for a subset of this deployment. We found that due to Aeris’ focus on global IoT connectivity and the number of customers they have in sub-Saharan Africa, their SIM cards work significantly better in Ghana than Particle or Twilio SIMs.
The largest operational change was addressing the issues described in Sect. 3.5 with our custom deployment management software, described further in Sect. 4.1.
3.7 Adaptation from Large-Scale Deployment to Sustainable Large-Scale Deployment
After the completion of our large-scale deployment, our team was asked by MiDA to scale the deployment again, this time to over 1200 PowerWatch sensors, with the continued goal of estimating SAIDI and SAIFI and providing this data to multiple impact evaluations of the Ghana Power Compact.
Before scaling up further, a subset of our team’s engineers created a company. This was not an easy decision. The academic research lab context afforded space, materials, and access to other researchers. Building a new organization has overhead, potentially taking resources away from solving deeper problems. Working across disciplines lets our team address an important data gap for economists with a new technology provided by engineers, and developing the technology as researchers allowed us to be slow, make mistakes, iterate, and get to know the stakeholders without the pressures placed on a subcontractor delivering a product. Most importantly, freedoms enjoyed within the academy around transparency, knowledge transfer, and independence contribute greatly to our personal drive to perform this work, and the financial obligations that come with forming a company often deprioritize these goals.
We recognized, however, that this project was no longer a good fit for the goals of an academic research lab. We anticipated fewer innovations related to the sensor and deployment methodology and thus less research suitable for academic publication, while also anticipating the need to spend more time supporting the expanding deployments. Further, we recognized the need to free the project from institutional dependencies that had been too slow for the rapid pace of field work, a pace that would only increase with scale.
Since starting the company, the overhead of establishing an organization – including hiring employees, establishing accounting systems, navigating conflicts of interest, and trying to package our measurements as a product – has been significant. However, we have been more nimble and responsive when faced with challenges, have successfully scaled our deployment again, and, most importantly, have a structure in place to allow this work to exist for longer than just the length of a PhD thesis.
Pivot: Mobile Phone Sensing to Plug-in Technologies?
One original research question tested during this project was whether we could repurpose daily-use smartphones, already deployed in the hands of billions of people, as low-resolution grid reliability sensors. Leveraging the widespread ownership of cellphone devices in developing countries would allow cash-constrained utilities to improve the accuracy of reliability measurements at lower cost than widespread smart meter deployment. In fact, a primary objective of the deployment of PowerWatch, our plug-in sensor, was to provide ground-truth outage data against which the DumsorWatch app’s measurements could be compared! However, as we scaled up both technologies, we saw PowerWatch succeed, and DumsorWatch underperform in two unexpected ways, one due to unanticipated participant behavior and the second due to a changing technical context. The combination of these limitations led us to prioritize PowerWatch and put DumsorWatch on the back-burner just as it was starting to return some positive results,
The first unanticipated result we observed is that DumsorWatch was uninstalled from participant phones at a higher rate than originally anticipated, even when participants were financially incentivized to keep the app installed. Most frequently, this happened automatically – for example, because the respondent reset, replaced, or lost their phone – and the respondent did not reinstall the app. Many participants also uninstalled their apps intentionally, for example, to save space on their phones or because they had privacy concerns from having an unknown app installed on their private device. As a result, just 3 months after the original deployment, DumsorWatch was only detecting around 10% of the outages that it was at the start. On the flipside, participants also expressed PowerWatch was less invasive than an app on the phone, potentially explaining why it regularly remained plugged in over long periods of time.
The second unanticipated action that impacted the performance of DumsorWatch was that the Android operating system changed to limit long-running background services. DumsorWatch depended on a background service that would wake the app up whenever the phone was plugged in or unplugged. This function was eliminated mid-deployment by Google in an effort to improve the user experience, since some long running apps have an outsized impact on battery life or quietly collect large amounts of data on users, burning data allocations and leaking privacy. We tried to get around this by limiting the versions of Android we recruited participants for to those that still supported background services. However, the new OS limitations ensured the sunset of DumsorWatch as an application layer technology, which led to us to reprioritize our engineering efforts to improve the PowerWatch system.
It is worth noting that our implementation of DumsorWatch successfully detected power outages! For as long as it was operational, participant phones running DumsorWatch demonstrated that uncoordinated smartphones experiencing charge state changes at the same time correlate with ground truth grid reliability measurements as provided by PowerWatch. However, based on both factors explained above, we believe the best path forward for the ideas captured in DumsorWatch is for these types of measurements to be taken as an OS level service in Android and to be aggregated as a primary crowd-sourced measurement (similar to how Google Maps captures and exposes traffic data (Google, 2009)). This pivot was made easier by the presence of PowerWatch, which could answer many of our remaining questions. While it remained a hard choice, it lets us better prioritize our team’s and our funders’ resources.