Keywords

1 The Development Challenge

As India has developed, so has her demand for energy. This demand has largely been met with abundant, inexpensive, and highly polluting fossil fuels. These choices have had fundamental environmental consequences, imposing costs that significantly threaten the country’s economic prospects. Perhaps the most significant of these is the dangerously rapid deterioration in the quality of India’s air, with satellite data suggesting an increase of over 70% in the concentration of particulate matter 2.5 (PM 2.5) between 1998 and 2016.

Air pollution now poses one of the most severe public health challenges to the country (Balakrishnan et al., 2019). Study after study has pointed to the general health risks of air pollution, in terms of life expectancy for example, and additional new research suggests that the effects of poor air quality may even extend to reduced crop yields, lower labor productivity, and decreased cognitive skills (Chang et al., 2019; Bharadwaj et al., 2017; Burney & Ramanathan, 2014). In other words, the apparent trade-off between environmental protection and economic growth is something of a Hobson’s choice—the Indian growth story cannot continue without cleaning up the air.

Notwithstanding the poor quality of its environment, India has a fairly strong and wide-ranging set of environmental laws. Nevertheless, these have not been sufficient to effectively control industrial air pollution. Empirical evidence from the highly industrialized Indian states of Maharashtra and Gujarat highlight the degree to which factories have been found to be violating pollution norms. In the cities of Surat and Ahmedabad in Gujarat, for instance, Duflo et al. (2013) collected data from hundreds of industrial plants and found that about 35% were polluting above the legally-prescribed limits. In Maharashtra, Greenstone et al. (2018) digitized over 13,200 regulatory pollution tests, spanning the period from September 2012 to February 2018, and found over half to reveal exceedances in the regulatory limits.

One reason why Indian manufacturing emissions pose a particularly thorny challenge to understaffed environmental regulators is that it is difficult to monitor a large number of highly polluting but relatively small factories.Footnote 1 In the state of Maharashtra, for instance, data collected between 2017 and 2020 showed that even when considering only the largest plants, the frequency of pollution testing remained well below once per year, on average.

The development challenge therefore has two parts. First, how can we reduce industrial air pollution in an expansive and rapidly developing country like India? And second, how can we improve the quality of air pollution data available to regulators, and can these types of improvements help reduce air pollution?

One promising solution to this challenge is the use of Continuous Emissions Monitoring Systems (CEMS). These instruments are installed in the smokestacks of factories where they continuously measure the concentration (or mass) of the air pollution that is being emitted. The data collected are then transmitted in real time to a remotely located computer server, vastly improving the quality of information available to the environmental regulators.

This case study discusses a project that aimed to evaluate the impacts of CEMS in a large-scale randomized control trial (RCT) conducted in partnership with the Gujarat Pollution Control Board (GPCB), the environmental regulator in the state of Gujarat in India. The goal of the project was to see whether the “big data” generated through CEMS could spur improved regulatory actions, thus lowering air pollution. While CEMS is not a new technology per se, its deployment in the Indian setting was novel. As we learned from our experience, technology acts through the interaction of human beings and hardware. As such, its effectiveness cannot be divorced from the incentives and capabilities of the people using it. Consequently, a central theme running through this chapter is the importance of evaluating a technology within the specific institutional and economic context in which it will be used.

The remainder of this chapter is structured as follows. In Sect. 2, we provide background information on the setting for the project, including how it was conceptualized and implemented, as well as findings from related research. In Sect. 3, we describe the CEMS technology in more detail and how the pilot implementation was designed to enable rigorous measurement of its benefits. In Sect. 4, we provide some novel results on the impacts of CEMS on industrial air pollution in India. In addition, we discuss what it means for a technology intervention to “work” or to be “successful” in a developing country context. Section 5 concludes.

2 Context

There is no single cause for the problem of excessive industrial air pollution in India. As is mentioned above, the performance of India’s existing regulatory framework has been far from perfect. Low institutional capacity and expertise, high transaction costs in taking legal action, corruption, high compliance costs for industry, and poor data have all contributed to this problem in varying degrees.

The starting point for this project is the ground-breaking RCT by Duflo et al. (2013, 2018), which studied the role of information and monitoring in regulating industrial air pollution in Gujarat. When Duflo et al. began this work in 2010, air pollution levels were rising across industrial cities in Gujarat. When third-party laboratories were used to audit the emissions of these factories, however, there appeared to be widespread compliance with environmental standards. This was a puzzling result. How could these test results be reconciled with the common sight of black air rising out of hundreds of small chimneys in and around Gujarati cities?

To answer this question, Duflo et al. ran a 2-year RCT in partnership with the Gujarat Pollution Control Board, the state environmental regulator. First, the study shed light on how the regulator was crippled by a persistent culture of data falsification. It documented collusion between the industrial plants and the auditors that were supposed to report on their performance. Next, the study evaluated an intervention designed to resolve this problem of collusion by severing the conflicts of interest, namely, by shifting the auditor hiring and payment decisions away from the individual plants and towards the government regulator.

Although the researchers were able to provide a partial solution to the monitoring problem (by making the audit reports more accurate), manual testing remained an infrequent and expensive method of gathering data. The regulators would still be unaware of how much the industries were emitting on a regular basis. In addition, the status quo emissions audits could not say anything about what the factories were doing when they were not being actively tested. In consequence, they were perhaps best viewed as simply an assessment of how industrial plants performed when on their best behavior. Overall, Duflo et al. (2013, 2018) suggest that high-quality, continuous information might be necessary to know which plants were polluting the most and that these monitoring improvements could potentially reduce emissions.

2.1 How Was the Project Location Chosen?

Large field experiments involving partnerships between researchers and the government are often opportunistic, and a function of the initial interests of individual politicians and bureaucrats. In this case, India’s Minister of Environment, Forest, and Climate Change felt that CEMS might not only solve the information problem but could also be a first step towards implementing market-based methods of environmental regulation. The project described here was thus the first attempt in India to carry out a systematic field evaluation of the effectiveness of CEMS. It also became the precursor to India’s first cap-and-trade market for industrial particulate pollution, which was launched in Surat in 2019.

The location of the CEMS pilot itself shines some light on the myriad forces that determine how and where a policy innovation is first implemented. The project initially involved the state governments of Gujarat, Maharashtra, and Tamil Nadu, three of the largest industrial states in India. However, requiring industrial plants to install CEMS proved to be politically and administratively difficult for a number of reasons. For example, the technology was untested, and several implementation challenges could be foreseen. For risk-averse regulators, there was a potential downside to being the “first mover” on a project that might not end well. In addition, the installation of CEMS devices would impose a significant cost on polluters—in the order of several thousand dollars—and this would surely result in pushback and reluctance on the part of the industrial factories.

Furthermore, up to that point, there was no precedent to using CEMS data as the basis for prosecuting industrial polluters. Indian environmental law is built upon criminal penalties. The basis upon which plants can be found to be in violation of pollution limits is enshrined in the laws. New methods of monitoring cannot be easily used as the legal basis for regulatory actions without evidence of their reliability and accuracy. In sum, requiring plants to install equipment that had not been granted legal status posed challenges that would understandably create industry resistance.

Eventually, the government of Gujarat was able to prevail upon the industry association of the city of Surat to support (or at least not oppose) the use of CEMS devices. Surat is a hub for the textile industry and is the site of a dense manufacturing cluster of hundreds of relatively small-scale plants that typically burn solid fuels.

One potential reason for their agreement is that the government informed them that CEMS would later be used to introduce emissions markets. This acted as a powerful incentive for the manufacturing plants. It changed the impression surrounding the equipment installation mandate from a pure negative (e.g., regulation would not change, while monitoring would increase and expensive equipment would have to be purchased) to a potential opportunity. Market-based regulation had been repeatedly recommended in India as a potential solution to the inflexibility, uncertainty, and the high costs associated with status-quo regulation.

2.2 Identifying the Experimental Sample

Having identified Surat as a project site, a decision needed to be made on how to select the initial sample of plants that would be transitioned to the new technology. This filtering process was driven by the potential of a plant to harm people through air pollution, as well as by the limitations of the technology itself. As a starting point, the sample of eligible plants was first restricted to factories located within 30 km of the city center, in order to focus attention to the plants that were located in the most densely populated areas. For a sense of scale, the metropolitan population of Surat was about 6.5 million, with a density of over 4000 people per square kilometer.

Conditional on being located within this radius, a factory needed to be burning solid or liquid fossil fuels and needed to have a stack (i.e., chimney) large enough to accommodate a CEMS device. This led to a sample of 373 plants. Most of these plants were operating in the textile sector (over 94%) and were burning coal (37%) or lignite (27%). Over the course of the multiyear project, 42 plants closed down due to an economic downturn in Gujarat, which correspondingly reduced the pilot sample size.

In the case of the other two states, Maharashtra and Tamil Nadu, there was no suitable set of plants that could be transitioned to CEMS, despite multiple years of efforts on our part. Thus, the geographic scope of our overall CEMS research effort shrank from three states to one.

The point of recounting this history is to underscore that the location of technology pilots, as well as the population involved, emerge through multiple layers of deliberate or accidental selection. This means that it is worth paying attention to the external validity concerns for any given technology deployment. In this case, the impacts of rolling out CEMS in Gujarat may not apply in Maharashtra or Tamil Nadu. For instance, if the plants that are successfully able to resist CEMS also happen to be politically well-connected and more difficult to regulate, then what we learn about the impacts of CEMS on our pilot population would not apply to other populations.

2.3 How Does CEMS Work?

Continuous Emissions Monitoring Systems (CEMS) for particulate emissions consist of a network of hardware devices and software programs that link monitored industrial plants to the environmental regulator in a manner that allows emissions data to be securely transmitted at regular intervals. The CEMS hardware components required at each industry site consist of the following:

  1. 1.

    Particulate matter (PM) CEMS analyzer and flow meters to measure the mass of pollutants emitted.

  2. 2.

    Data logger unit for saving records on-site, in case of Internet failure.

  3. 3.

    Data acquisition system (DAS), normally consisting of an on-site computer and a server at the site of the regulator.

  4. 4.

    Software to visualize and analyze CEMS emissions data.

The CEMS analyzer for particulates is a device that relates the physical properties of emissions from a factory chimney to the concentration (or mass flowrate) of suspended particles in the air. For example, optical devices measure the attenuation in the intensity of a laser beam sent through smoke. An alternative technology exploits the so called “triboelectric” effect and relies on measuring the electric charge induced by the movement of particles near a probe.

These approaches to measurement are “indirect” since they measure a property of the gas that is used as a proxy for the presence of solid particles. As a consequence, these devices must first be calibrated against manual readings that directly measure the weight of particles in a specified volume of exhaust. In other words, an electrical signal generated by the analyzer must be mapped to a value of particulate emissions. Typically, this mapping is obtained by fitting a linear model relating a set of manual measurements taken at different levels of boiler loads (typically nine readings) to the corresponding measures of current produced by the analyzer. This delivers an equation y = a + bx where x is a current reading in amperes, y is an estimate of the concentration of pollutants, and a and b are parameters estimated during an initial calibration process.

3 Innovate, Evaluate, Adapt

There is a well-known management cliche that, “if you can’t measure it, you can’t manage it.” There is some truth to this when it comes to regulating industrial emissions. When the government is unaware of how plants are behaving most of the time, they have little ability to target regulatory actions, and may have no legal basis to penalize otherwise polluting factories who manage to pass the occasionally scheduled tests. Furthermore, from the point of view of human health—the ultimate motivation for environmental regulation in the first place—what matters is not how much pollution a plant emits at any given point in time but the cumulative mass released into the ambient air over a period of time. A highly polluting plant operating for only a few days in the year might fare the worst on a one-off manual test. But it may in fact be a lot less harmful than a “cleaner” plant that is operating 24 h a day, 365 days a year.

All of this suggests that if regulators are to do their job properly, they need information that identifies which plants are polluting. Unfortunately, the standard method for measuring factory air pollution, which is commonplace around the world including in India, involves a painstakingly manual process. To measure the concentration of suspended particulate emissions, a team of engineers must extract a sample of air from the smokestack of a factory while carefully following a prescribed protocol. This air sample must then be transferred to a laboratory where solid particles are dried to remove moisture and then weighed. The mass of these collected particles is then converted into a concentration measure after “normalizing” the volume of air extracted by standardizing it to a particular temperature and pressure. The whole procedure is labor- and time-intensive. The CEMS technology improves matters by transmitting emissions data in real time, directly from the chimney of a plant to the regulator. In comparison to the manual, point-in-time, method of measurement, the CEMS technology offers a potentially dramatic improvement.

There are a number of possible benefits from CEMS. Improving the information available to environmental regulators could allow existing rules to work better, for example, by allowing the government to target inspections and use calibrated penalties on the worst offending plants. CEMS could also open up the possibility of new forms of regulation that are made possible by higher quality and higher frequency data. For instance, emissions trading schemes or pollution taxes can be introduced only when there is an accurate count of the quantity being traded or taxed. These policy tools may have important advantages over existing “command and control” approaches to regulation but are feasible only if there is high-quality of data. Thus, on the surface at least, the CEMS technology would seem to provide everything needed to resolve the challenge of monitoring plants and, in so doing, to reduce the pollutants they release.

3.1 Setting Up an Evaluation

The goal of our pilot was to understand the impact of mandating CEMS on plant and regulator behavior. On the side of the plants, the primary outcomes of interest were pollution and reporting quality. On the side of the regulator, we were interested in whether the government interacted differently with these plants, enforcing more penalties, for instance. We were able to do this by integrating an RCT into the implementation of the pilot project.

The evaluation initially began with a small group of installations serving as a technical dry run. The purpose was to field-test the CEMS installation and calibration protocol stipulated in the Government of India’s Central Pollution Control Board (CPCB) specifications. We selected 11 industrial plants (which we refer to collectively as “Phase I”) from a cluster of factories in Surat. Next, four different vendors supplied these eleven plants with CEMS devices, consisting of nine DC triboelectric-based devices (measuring particulate matter (PM) mass flow), and two electrodynamic-based devices (measuring PM mass concentration). The 11 industries were chosen to represent the diversity of the full sample and to cover a range of boiler sizes and types of installed air pollution control devices (e.g., numbers of cyclons, bag filters, etc.).

The field tests were important because they showed that the error associated with a CEMS measurement taken at any point in time disappears as multiple observations are added together. These sorts of tests provided confirmation that even if a device noisily measured instantaneous levels of emissions, it could still provide valuable information to determine long-run aggregate or average levels of emissions.

For the remainder of the sample, we used an RCT design to rigorously quantify the impact of CEMS. The simplest possible design would have involved dividing plants into a treatment group and a control group and then mandating that factories in the former group install CEMS, while factories in the latter group are regulated and monitored as usual. But this design would have required an enforced pause in the rollout of CEMS, after the treatment group plants had installed their devices and begun sending the regulator real-time data.

In our context, an enforced pause was not feasible for several reasons. First, installing CEMS devices was a time-consuming affair because the devices were not readily available in a still nascent market. Saturating the treatment group of factories alone took several months due to delays in purchase orders being fulfilled, delays in calibrating devices, and data connectivity problems that appeared at scale (and were not encountered during implementation for the limited number of Phase I industries.) Losing many more months of time in order to enforce a pause was therefore unacceptable to the government. Also, it would have probably resulted in the small number of CEMS vendors operating in the market to exit due to lack of business, or declining to deploy the necessary staff to the field for required after-sales services.

Second, as we have mentioned, it was not easy to ensure that the industries in Surat would comply with the new requirements to install hardware. It was believed that the likelihood of smoothly saturating the entire population of industrial plants would be significantly reduced if there was a long period of time during which half of the plants in the sample were held off from placing new equipment orders.

Finally, as our work progressed, it became increasingly likely that the GPCB would want to launch an emissions trading scheme in Surat. Thus, the effort to set up these devices had the dual goals of understanding the impact of improved data on environmental outcomes, and eventually measuring the impact of a market-based approach, like cap-and-trade, to reducing industrial air pollution. All of these considerations led both the government and the research team to want to avoid long delays in implementation.

To meet these constraints and allow for a rigorous evaluation design, the remaining plants were randomly assigned into three groups (Phases II, III, and IV). Each group was mandated to install a CEMS and begin sending data to the regulator in a sequential manner. In this type of staggered, phased-in research design, the plants that were mandated to install CEMS in the later phases could serve as a comparison group for the plants that had installed CEMS earlier. For example, once Phase II installations had begun sending data, and while Phase III plants were installing and calibrating their devices, the Phase IV plants would still be continuing business as usual. Thus, the outcomes for Phase II and Phase IV plants could be compared against one another in order to estimate the causal effects of CEMS. This was possible due to random assignment to each group.

3.2 Implementation Challenges in the Field

On paper, the experimental design was fairly straightforward. In practice, implementation was anything but simple. Figure 7.1 provides a timeline for the pilot as a whole. The pilot began in early 2014 and ran until late 2018. From an experimental research standpoint, the period between the second baseline survey (or midline survey, carried out in 2016) and the endline survey (carried out in 2018) constitutes the effective duration of the treatment. This is just a fraction of the overall duration of the project. The 2-year period between the first baseline survey and the second baseline survey represents a period of time in which a mature technology—that had been readily deployed in large plants in developed countries—was effectively unusable in Surat.

Fig. 7.1
figure 1

Timeline of CEMS rollout and data collection

Notes: Each survey was 3 months long

Full size image

Why was this the case? The primary challenge was the interaction between the technology and the institutions and systems around it. The most notable unexpected challenges appeared in the process of installation and calibration of the CEMS devices, which I now detail below.

3.2.1 Problems Encountered While Installing CEMS

The first step in setting up a functioning CEMS network is for plants to equip themselves with the necessary hardware. This proved to be a rocky experience, in part because no well-functioning market in CEMS devices was in place when the pilot started. This was not surprising, given that there was no demand for CEMS before the equipment mandate. This posed several challenges to pilot implementation.

The most important of these was the near complete absence of after-sales service and trained personnel. Initially, the goal of the equipment vendors was to make a one-time sale, under the assumption that the pilot was likely to be an isolated, one-off experiment. Investing in trained staff to be based in Surat likely made little business sense for many of these companies. High-quality service contracts were likewise unavailable. The consequence was a host of equipment problems that could not be properly addressed and industry dissatisfaction with the technology, creating a fragile ecosystem prone to breakdowns. Only after several months of persuasion and pressure by the government on the manufacturers and vendors of CEMS devices did the situation improve.

A second area which proved to have several teething problems was the implementation of a robust data acquisition system to transfer emissions information from industrial sites to the state regulator. Two aspects of the prevailing ecosystem made this hard. First, different CEMS vendors used varying, proprietary data storage formats, making it difficult to create a common platform that might be able to read and transfer data. Second, industries in Surat frequently had poor Internet connections, old computers, no dedicated IT staff, and faced occasional power outages that would force a complete reset of the system. These issues are commonplace in developing countries but are not necessarily the focus of manufacturers of CEMS technology. Creating IT and data storage systems that were robust to these weaknesses proved to be challenging and time-consuming.

Finally, since the equipment mandate merely required industrial polluters to install a CEMS device, many industries chose the lowest price and lowest quality option. This type of equipment choice enabled plants to initially meet the regulatory requirement without incurring a large upfront payment but heightened the costs associated with the lack of after-sales service. Thus, several months after initial installation, some of these plants found their devices to be dysfunctional and new purchases became necessary, which set back the overall timeline for the experiment.

3.2.2 Problems Encountered While Calibrating CEMS

CEMS devices also need to be calibrated before being used. If the initial calibration parameters are misreported, the subsequent measurement of emissions will be biased. The calibration step requires plants to work with equipment vendors and environmental auditors to make manual measurements and then correlate these with the electrical output from the CEMS analyzers. The calibration requirement is crucial because it represents an important mechanism through which data can be falsified, in a similar way to the corruption noted in Duflo et al. (2013).

Calibration will not cause problems in a laboratory environment. However, it is exactly the sort of weakness that can render a technology unreliable in the field. Importantly, the calibration process requires information from outside the CEMS itself, since CEMS must be used in conjunction with data from a different method of measurement. This means that the issues that can affect the manual data collection process can also affect CEMS. In sum, we cannot evaluate CEMS without considering the underlying economic incentives of the calibration agents and the institutions that influence them.

Early on in the project, there were several instances of CEMS transmitting unreliable data, whether deliberately falsified or not. Figure 7.2 provides examples of data that can be identified as unreliable through visual inspection or a series of automated data checks. Mitigating these problems in the field was not easy and required a combination of measures including: regular data validation; statistical tests on the emissions time series; double-blind calibration, where the lab involved in testing manual samples is separated from those collecting samples; encryption of transmitted data; and regular spot-checks following calibration. All of these measures needed to be developed from the ground up, significantly reducing the degree to which these devices could be treated as a “plug-and-play” solution to the information problem.

Fig. 7.2
figure 2

Examples of potentially invalid CEMS data

Notes: Suspicious data transmitted by four CEMS devices during the month of December 2019. Clockwise from top left. (1) Zero values indicate device disconnection in a Type-1 device; (2) flatline values can result either from voltage top-censoring or a probe which has not been cleaned; (3) abnormally low variance suggests wrongly calibrated device or voltage amplification settings; (4) sudden decreases in overall emission levels are suspicious, and sudden changes in variance suggest tampering of voltage amplification settings

Full size image

3.2.3 Linking Data to Regulatory Practice

The installation of a CEMS analyzer in a factory chimney is unlikely to be useful in isolation. It is not uncommon for governments to accumulate large quantities of data without using them to guide policy. In order for the CEMS technology to achieve its potential, the environmental regulator needs to determine how the resulting data will be analyzed and used. Defining data use protocols in advance would seem to be an essential factor contributing to the success of this technology.

The GPCB chose to institute—but did not perfectly implement—a detailed Action Matrix linking the data they received to regulatory action. Plants were penalized if they did not transmit data regularly and if their pollution levels exceeded the specified thresholds. Table 7.1 summarizes the key features of this protocol.

Table 7.1 Regulatory actions tied to emission measurements with CEMS
Full size table

4 Did CEMS Work?

In a narrow sense, we might regard a technology as “working” if it delivers in the field a performance that is very close to the design specifications determined in a lab or via controlled engineering trials. This definition is not very useful from the perspective of a policymaker. Thus, a slightly broader definition is to define CEMS as “working” if it delivers useful data. The extent to which CEMS data is useful depends on both its information content and its potential value to regulators.

Yet even this broader definition may also be inadequate. Even if CEMS delivered valuable information, it might not result in lower levels of pollution. Changes in the behavior of industrial plants are likely to occur only if the regulator uses the new information to change the incentives for plants to pollute. Institutional weaknesses, rent-seeking, political patronage, and other implementation failures could lead to regulators ignoring to act upon these improved sources of information. In this section, I report on some of the descriptive and experimental results from our deployment, in order to assess the technical performance, usefulness, and impact of CEMS in Gujarat.

4.1 Assessing the Technical Performance of CEMS

Suppose we define a working CEMS device as an instrument that reliably delivers a signal that accurately represents plant emissions. An important insight from our experiment was that the CEMS devices did not automatically work until they were coupled with regulatory measures that appropriately aligned the incentives of plants and vendors with those of the regulator. As discussed, the mandate to install CEMS devices was followed by a host of implementation challenges. Some of these reflected difficulties that are widespread in developing countries, including poor infrastructure or the absence of trained staff in factories. Others, such as the paucity of after-sales service, arose out of market weaknesses that were a consequence of mandating purchases in a very nascent market. Still others arose from the opportunity for data falsification created by the calibration requirement. Although improper calibration does not reflect an engineering failure, the ease with which regulatory devices can be gamed is quite fundamental to an assessment of the technology’s effectiveness.

However, although the new monitoring system was not technically perfect at the outset, it improved steadily and significantly until it reached a point where a meaningful experimental evaluation could be conducted. Some of this improvement occurred after the environmental regulator introduced the Action Matrix, shown in Table 7.1. This is because this protocol created the first real set of consequences for plants that did not maintain their devices, failed calibration audits, or did not transmit data regularly. Statistical checks were later developed to automatically identify plants whose data suggested the possibility of miscalibration or tampering, further improving matters. Figure 7.3 shows that after the final calibration, a strong relationship existed between CEMS readings and concurrently measured manual measurements.

Fig. 7.3
figure 3

Accuracy of CEMS data

Notes: Correlation between CEMS reading and PM sample. The figure shows data the data from calibration of CEMS devices. The figure shows the normalized values of the PM samples and CEMS readings per industry, and the red line shows the fitted values

Full size image

Figure 7.4 shows the history of CEMS uptime rates for 66 industrial plants, with the regulatory measures taken by the Gujarat Pollution Control Board marked with vertical lines. The performance of CEMS appears to improve with the introduction of each measure. Eventually, a high level of data availability is achieved. This evidence has important implications for how a new technology is introduced and used. It seems intuitive that new regulation should be introduced only after the underlying technology is working very well. However, our experience suggests that the presence or absence of regulation cannot be disconnected from the performance of the enabling technology. In other words, in order to achieve an equilibrium involving a well-functioning CEMS network, it may be necessary to introduce regulation that commits to using the data generated by CEMS, even before all of the technical problems are resolved.

Fig. 7.4
figure 4

Average CEMS data availability at 66 industrial plants

Notes: Average percentage data availability from CEMS devices over time where periods of no transmission from a plant are set as zero. The introduction of the action matrix in February 2018 was followed by a significant improvement in performance. The introduction of an emissions trading scheme in July 2019 produced similar results

Full size image

4.2 Assessing the Usefulness of CEMS

CEMS are intended to augment an existing testing protocol in which the main problem is that information is only collected at a single point in time. The case for CEMS is based in part on the notion that continuous data is more useful as a measure of environmental performance than a single snapshot.

To investigate this, we examined periods of time in which we could compare readings from CEMS devices to manual samples collected by the GPCB. Recall that these manual samples formed the basis of regulation in the status quo. There are two potential sources of bias. First, there is the falsification of data identified in Duflo et al. (2013). Second, there is possible bias stemming from the fact that plants may behave differently when they are being tested to when they are operating normally.

Figure 7.5 shows a comparison between manual and continuous data for a plant in Surat. For this plant, it is striking how the period of time in which the manual sampling takes place is entirely unrepresentative of pollution at all other times. This suggests that relying on manual tests alone could result in significantly underestimating how much a plant is contributing to air pollution. Furthermore, the variation in emissions levels over time suggests that even if there were no systematic bias associated with pollution performance during testing periods, it is unlikely that extrapolations based on a single observation could reasonably approximate the true levels of pollution released by the plant.

Fig. 7.5
figure 5

CEMS data transmitted before, during, and after an in-person emission inspection

Notes: This plant appears to be strategically reducing emissions while being tested

Full size image

4.3 Estimating the Impact of CEMS on Pollution Emissions

The third measure of success relates to whether or not CEMS reduced pollution emissions, which is the ultimate goal of the policy intervention. According to this measure, we found limited evidence that the technology worked.

The direct effect of requiring plants to install CEMS on subsequent pollution levels can be estimated due to the RCT design we implemented. To rigorously quantify the effects of CEMS, we need to not only measure what happened to factories after installing these devices but also identify a counterfactual describing what outcomes would have occurred absent CEMS. The phased rollout of the pilot, combined with random assignment to different groups, meant that at the start of the experiment, the plants that were asked to install CEMS early on (Phase 2) were on average identical to those that were asked to install CEMS later on (Phase 4). For this reason, any differences in pollution emissions that are observed between the two groups after CEMS data is sent to the regulator can be attributed to the installation of CEMS. All other factors unrelated to the technology mandate would affect both groups equally, and thus would not bias any comparisons between the two groups.

Table 7.2 Differences in industrial plant characteristics at baseline
Full size table

Table 7.2 presents a comparison of baseline characteristics of treatment and control plants in the study sample. The randomization check shows that plant characteristics are balanced across treatment groups. Out of all of the baseline measures reported, there are just two variables that show a statistically significant difference between treatment groups, at the 10% level (these are the cost of recent modifications and the number of inspections performed by the GPCB in 2014). In Panel A we examine various costs associated with purchasing or using air pollution control equipment. The variable costs from operations and maintenance associated with using this equipment are significant, roughly 40% of capital costs on average. This becomes important because the presence of equipment is easily observed and enforceable when granting permissions for plants to begin manufacturing. Accordingly, all these factories in our sample do possess pollution control devices (Panel B), with cyclones being the most common. The high levels of pollution that we observe are nevertheless influenced by the costs of operating equipment. Panel C reports statistics on inputs and revenue from these plants and boiler and thermopack capacities (which influence fuel use and pollution). Panel D shows that pollution levels at baseline are remarkably high with mean levels of pollution concentration above 300 mg/Nm3. We also measure levels of regulatory engagement in terms of the number of inspections conducted and legal actions required in the year of the survey.

Table 7.3 reports the treatment effects of CEMS at endline, based on estimating the following regression equation:

$$ {y}_{i1}=\alpha +{\beta y}_{i0}+{\gamma T}_i+{s}_{it} $$
Table 7.3 Impact of CEMS on pollution and compliance
Full size table

Here y i1 is the outcome variable for plant i measured during the endline survey, y i0 is the baseline value, and T is a dummy indicating membership of the treatment group. We use this model to examine changes in three outcomes: PM readings from a manual sample taken at the plant; the logged value of the plant test result; and a binary measure of compliance, indicating whether the test resulted in a PM concentration reading below 150 mg/Nm3. This value is the fixed regulatory limit for the concentration of particulate emissions from an industry smokestack. Note that since the treatment itself was the installation of CEMS, there was no continuous data available for control plants.

In Column 4 of Table 7.3, we report changes in a fourth outcome variable, the results of Ringelmann tests that we carried out at periodic intervals throughout the experiment. A Ringelmann test is a simple visual measure of the color of smoke in the chimney stack, which is a crude proxy for the level of particulates in emissions. These tests were repeated between the baseline and endline, providing a panel dataset. To estimate the impacts of the treatment on this outcome, we estimate a similar regression model to the one above, but with additional weather controls and with multiple observations for each plant over time, rather than values captured only at baseline and endline.

Across all specifications, we find no strong evidence that plants in the treatment group polluted less than their counterparts in the control group. Between baseline and endline, there was an overall reduction in both the level and variance of the manual spot measurements of pollution. The average PM concentration at endline was 185 mg/Nm3 with a standard deviation of 211. At baseline, the average concentration was 338 mg/Nm3 with a standard deviation of 374. However, this reduction occurred in both factories with and without CEMS making it difficult to attribute the change to the new technology. This finding is a cautionary tale underscoring why the use of a treatment and control group is so important. Had we only looked at pollution changes in factories with CEMS, we might have noticed they became cleaner and then credited this to the technological intervention. As it happens, unless plants in later phases pre-emptively cut pollution in anticipation of a future mandate (possible, but unverifiable), it may be the case that better monitoring alone is not sufficient to solve the development challenge at hand.

Why was factory pollution unresponsive to the installation of CEMS? One answer might lie in how the data was actually used, or not used, by the regulator. Recall Table 7.1, which describes the protocol that the GPCB stated would determine how it acted upon the continuous pollution data it received. The protocol envisages actions of steadily increasing severity, rising to a legal notice. However, upon examining the history of regulatory interactions at the end of the pilot, we found that the regulators’ bark was worse than its bite. Although the first few actions in the schedule (e.g., SMS messages, in-person meetings, and letters) were executed, there was no significant difference between the treatment and control groups in the probability that a plant received a legal notice or even an in-person pollution test. In other words, when it came to taking actions that involved more than “cheap talk,” the regulator blinked.

Interestingly, this lack of follow-through action in response to high CEMS readings was not perfectly mirrored in the other data source available to the regulator, namely, the results of traditional manual tests. This difference is clearly visible in Fig. 7.6. The conditional probability of plants receiving a legal notice significantly increases when their last manual test reveals very high readings, but no such pattern exists when examining CEMS data. In other words, although the continuous monitoring systems generate useful, even superior, information, the regulator chose not to use it for targeting enforcement. If the plants understood that this would be the case, it is unlikely that CEMS would have had an observable effect on behavior.

Fig. 7.6
figure 6

Probability of legal action

Notes: Conditional probability of legal action by the Gujarat Pollution Control Board as a function of pollution measured by CEMS instruments and manual samples expressed as multiples of regulatory standard. Dots represent mean values in bins and bars are 95% confidence intervals

Full size image

5 Conclusion

The experiences recounted in this case study underscore the complexity involved in translating technology from a developed country to a developing country context. The Surat experience with CEMS demonstrates how technical solutions to development problems cannot be divorced from the institutional context in which they are used. It is often impossible to identify the effect of these interactions either in the lab or through iterative design processes in the field, because they are only apparent in applications at scale.

In the present example, the technical pilots that preceded the RCT provided an example of small-scale field trials based on which several technical tweaks were made. Although this step was necessary for implementation, it was far from sufficient. Other challenges such as calibration problems or staff capacity constraints became apparent only when several hundred devices were deployed at once.

Development engineering requires a blend of the social sciences and engineering. The Surat CEMS deployment was evaluated using an RCT design, drawing upon the expertise of a team of economists. However, without continuous technical innovations designed in situ, there would have been no intervention to test. The eventual results were mixed, placing the spotlight on additional questions, some of which have to do with the design of regulatory institutions and the legal framework underlying environmental regulation in India.

In 2019, the Gujarat Pollution Control Board launched its Emissions Trading Scheme (ETS) across a cluster of industrial plants in Surat. The Surat ETS is the first cap-and-trade market in particulate emissions across in the world. The data generated by the Continuous Emissions Monitoring Systems that were set up as a part of the project we have described here went on to form the data foundation for the ETS.

In the past, India’s environmental regulations have been criticized for being blunt and inflexible, proving both costly for industry and difficult for the government to implement and enforce. Market-based instruments, such as emissions trading schemes, provide an alternative that could meet the dual challenge of economic growth and environmental safety. Thus, the Surat ETS represents a paradigm shift in Indian regulation, stemming in part from the recognition that no matter how well these instruments worked, if they were used within an archaic regulatory framework, pollution was unlikely to be reduced. In a sense, the most important impact of these devices may have been to open up the option of using modern market-based regulation in India, something that would have been completely impossible without a transformation in how pollution from industry was monitored.

6 Discussion Questions

  1. 1.

    As a development engineer, what criteria would you apply before declaring a technological solution a success? What types of disciplinary skills would a research team need to possess to give them the best chance of meeting these targets? What are the differences between the criteria for success that an R&D lab, a venture capitalist in a startup, or a policymaker might apply when evaluating a technology solution to a development problem?

  2. 2.

    The adoption of technology designed to solve development problems often requires government support through tax-incentives, regulatory mandates, or subsidies. On the one hand, if such policy assistance is to be entertained, then rigorous evidence on effectiveness seems essential. On the other hand, it may be difficult for any technology to improve without learning-by-doing and the opportunity to iterate and improve given results in the field. Choose a technology other than CEMS (perhaps from another chapter in the book) and discuss how you would approach this balancing act? As policymakers, what strategies would move us most quickly towards effective, cheap, reliable technology solutions, and what do you think a phrase like “data-driven policy-making” should mean in this context?

  3. 3.

    Can you think of ways to make it hard for plants to falsify data coming from a CEMS device, holding the technology constant? Is there a new rule the government might impose or additional data that could be used to cross-check the validity of data? How would you judge the cost-effectiveness and feasibility of these solutions in a developing country context?