Skip to main content

Advertisement

Log in

Using experimental manipulation of questionnaire design and a Kenyan panel to test for the reliability of reported perceptions of climate change and adaptation

  • Published:
Climatic Change Aims and scope Submit manuscript

Abstract

While the use of surveys to understand perception of climate change and adaptation is common in research on agriculture, the reliability of some aspects of the methodology is largely untested. In particular, there is limited evidence on (i) the degree to which measures of perception are sensitive to questionnaire design, (ii) the accuracy of recall methods for climate change, and (iii) the degree to which measures of adaptation based on recall from one-time surveys match the historical record. Using an established panel of farmers from across Kenya and a split sample method, I test both the sensitivity of stated perceptions of climate change to question format and the accuracy of recalled adaptations. In one treatment, farmers face open-ended questions about temperature and rainfall changes while in the other treatment, farmers are offered closed-end questions. Both approaches are common in the voluminous literature on climate change adaptation. Responses are highly sensitive to question format, both in the degree of perceived change and in the types of changes. Stated adaptations are not so sensitive to question format, but still diverge. Stated adaptations do not correspond well to the historical record of farming practices over the 15 years of the panel. Overall, the evidence suggests that researchers and policy-makers should be highly cautious in their use of subjective perceptions of climate change and the use of adaptation measures based on recall data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. I also compare the perceived change in climate with satellite records, and thereby contribute to the literature on the relationship between subjective perceptions of climate change and more objective measures provided by satellite data. However, since this literature is already extensive (e.g. Hansen et al. 2012.), I do not focus on the results here.

  2. Marlon et al. (2018) contains a useful summary on the strength of the relationship between perception and reality of climate change. See also Zaval et al. (2014), Salerno et al. (2019) or Rao et al. (2011) for evidence that the accuracy of beliefs is often extremely limited.

  3. A third important issue, one that is not central to the main point of this paper is whether perception is a necessary condition for adaptation. In many papers cited, only farmers who indicate some form of climate change are asked about adaptation. This creates a potential problem of selection bias when the relationship between climate change and adaptation is to be estimated. At a deeper level, it is not clear that perception is necessary for adaptation. Farmer A, for example, may perceive change while B does not. Farmer A adapts and B notices that farmer A is producing higher yields or earning a higher income and then copies farmer A. Such transmission of technical knowledge through social networks is typical. I would argue that B also adapts to climate change, but in many surveys this adaptation would not be recorded.

  4. We can also test whether the stated perceptions of climate change match the available data on climate change. While speculation about each of these claims is common, it is this hypothesis that has received most attention in prior work (e.g. Simelton et al. 2013 ). The evidence seems to be that within relatively shorter time periods, the match between perception and meteorological data is poor, but that is slightly better over longer periods (Fosu-Mensah et al. 2012; Li et al. 2014; Budhathoki and Zander 2019). Since it is well-researched issue, I don’t pursue it here, but see Munro (2020).

  5. Inevitably there is some attrition and in fact all the panels have been refreshed at some point. In the data analysis below, when focusing on historical changes in land use, I only use households who are present throughout the panel.

  6. Open-ended questions may also elicit comments about extreme temperatures for example. In fact, we do not see many such comments, but the key issue for this study is whether the responses about means, rainfall seasons and droughts in the close-ended questions are reflected in the responses to the open-ended format.

  7. These data were obtained from the NASA Langley Research Center (LaRC) POWER Project funded through the NASA Earth Science/Applied Science Program.

  8. Kenya has 47 counties. The number of sub-counties per county varies, but is typically 10-15 so that the within-sub-county climate variation is small. I use fixed effects to absorb the myriad of influences on adaptation that are common to a locality. I do not use fixed effects at the village level since for about half the sample there is only one household per village. For many other villages there are only two or three households in the sample.Footnote 9 Overall I have 43 sub-counties in the sample.

  9. There are also no enumerator dummies in the equations. Kenya has many local languages and therefore a large team (35) of enumerators was used, most of whom worked in only one or two sub-counties. Models with enumerator and sub-county fixed effects do not typically converge. On the other hand, models with enumerator fixed effects only and no sub-county dummies give substantially the same results as those presented here.

  10. The omitted adaptations are “To non-farming” “Other”, and “No adaptation”. The first of these is omitted because only a few participants respond that they have switched away from farming and this produces a lack of variation within sub-counties that produces a non-convergent model. There is no evidence from the mean responses that this variable is sensitive to the version of the questionnaire.

  11. The impact of the questionnaire format may also affect the probability indirectly through its influence on the reporting of particular climate changes. Thus the coefficient on the version dummy may under- or overestimate the overall influence of the survey format on measures of adaptation. Leaving out the subjective reports of climate change, however, does not change the basic pattern of results reported in this section.

  12. This absence may be due to the use of the fixed effects model, which means that the climate change variables only pick-up the the impact of differences between perceptions and the satellite data. Footnote 13

  13. text

  14. These adaptations are as follows: Different planting dates/Earlier planting/Later planting, Shading and sheltering/tree planting, Used insurance or weather derivatives, Prayer or ritual offering. Possibly the first of these is the most important, but the dates of planting were not recorded in the RePEAT panel and it is not possible to reliably infer them from the database.

  15. I do not use any personal characteristics of the farms or farmers in these Tables, mainly because actual surveys of farmer perceptions will vary in the degree to which such information is available. Perhaps not surprisingly, it turns out that including some personal information can, for at least some equations, be predictive for changes in land management, though it seems to vary by question. However, the important point is that including other farm and farmer information does not alter the basic conclusion that stated adaptation is typically not predictive for historical change.

  16. Some care has to be taken over the meaning of the word “predictive” here. For example, reported intercropping is predictive of the historical record of change. However, according to the panel, intercropping has diminished over the sample period, whereas according to the climate change model, farmers report actually higher use of intercropping. The positive correlation occurs because farmers who have reduced intercropping more are less likely to report that they have increased intercropping.

  17. All the variables take on either two levels (positive, negative or no change and increase) or three (up, no change, down).

References

  • Assaad R, Krafft C, Yassin S (2018) Comparing retrospective and panel data collection methods to assess labor market dynamics. IZA Journal of Development and Migration 8(1):17

    Article  Google Scholar 

  • Battaglini A, Barbeau G, Bindi M, Badeck FW (2009) European winegrowers’ perceptions of climate change impact and options for adaptation. Reg Environ Chang 9(2):61–73

    Article  Google Scholar 

  • Beegle K, Carletto C, Himelein K (2012) Reliability of recall in agricultural data. J Dev Econ 1(98):34–41

    Article  Google Scholar 

  • Bryan E, Ringler C, Okoba B, Roncoli C, Silvestri S, Herrero M (2013) Adapting agriculture to climate change in Kenya: household strategies and determinants. Journal of Environmental Management 114:26–35

    Article  Google Scholar 

  • Budhathoki NK, Zander KK (2019) Nepalese farmers’ climate change perceptions, reality and farming strategies. Climate and Development, pp 1–12

  • Choi BC, Pak AW (2005) Peer reviewed: a catalog of biases in questionnaires. Preventing Chronic Disease 2(1)

  • Conley TG, Udry CR (2010) Learning about a new technology: pineapple in Ghana. American Economic Review 100(1):35–69

    Article  Google Scholar 

  • De Jalón SG, Silvestri S, Granados A, Iglesias A (2015) Behavioural barriers in response to climate change in agricultural communities: an example from Kenya. Reg Environ Chang 15(5):851–865

    Article  Google Scholar 

  • Deininger K, Carletto C, Savastano S, Muwonge J (2012) Can diaries help in improving agricultural production statistics? Evidence from Uganda. J Dev Econ 98 (1):42–50

    Article  Google Scholar 

  • Deressa TT, Hassan RM, Ringler C, Alemu T, Yesuf M (2009) Determinants of farmers’ choice of adaptation methods to climate change in the Nile Basin of Ethiopia. Global Environmental Change 19(2):248–255

    Article  Google Scholar 

  • Desiere S, Jolliffe D (2018) Land productivity and plot size: is measurement error driving the inverse relationship? Journal of Development Economics 130:84–98

    Article  Google Scholar 

  • Fosu-Mensah BY, Vlek PL, MacCarthy DS (2012) Farmers’ perception and adaptation to climate change: a case study of Sekyedumase district in Ghana. Environ Dev Sustain 14(4):495–505

    Article  Google Scholar 

  • Gbetibouo GA (2009) Understanding farmers’ perceptions and adaptations to climate change and variability: the case of the Limpopo Basin, South Africa, vol 849. Intl Food Policy Res Inst

  • Geer JG (1991) Do open-ended questions measure salient issues? Public Opinion Quarterly 55(3):360–370

    Article  Google Scholar 

  • Gray CL (2011) Soil quality and human migration in Kenya and uganda. Glob Environ Chang 21(2):421–430

    Article  Google Scholar 

  • Habiba U, Shaw R, Takeuchi Y (2012) Farmer’s perception and adaptation practices to cope with drought: perspectives from Northwestern Bangladesh. International Journal of Disaster Risk Reduction 1:72–84

    Article  Google Scholar 

  • Habtemariam LT, Gandorfer M, Kassa GA, Heissenhuber A (2016) Factors influencing smallholder farmers’ climate change perceptions: a study from farmers in Ethiopia. Environ Manag 58(2):343–358

    Article  Google Scholar 

  • Hansen J, Sato M, Ruedy R (2012) Perception of climate change. Proceedings of the National Academy of Sciences 109(37):E2415–E2423

    Article  Google Scholar 

  • Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S (1997) A comparison of goodness-of-fit tests for the logistic regression model. Statistics in Medicine 16(9):965–980

    Article  Google Scholar 

  • IPCC (2014) Climate change 2014–impacts, adaptation and vulnerability: regional aspects. Intergovernmental Panel on Climate Change, Cambridge University Press, Cambridge

  • Kahneman D (2003) A perspective on judgment and choice: mapping bounded rationality. American Psychologist 58(9):697

    Article  Google Scholar 

  • Kaplowitz MD, Hoehn JP (2001) Do focus groups and individual interviews reveal the same information for natural resource valuation? Ecol Econ 36(2):237–247

    Article  Google Scholar 

  • Krosnick JA (1999) Survey research. Annual Review of Psychology 50(1):537–567

    Article  Google Scholar 

  • Kurukulasuriya P, Mendelsohn R, Hassan R, Benhin J, Deressa T, Diop M, Eid HM, Fosu KY, Gbetibouo G, Jain S et al (2006) Will African agriculture survive climate change? The World Bank Economic Review 20(3):367–388

    Article  Google Scholar 

  • Le Dang H, Li E, Nuberg I, Bruwer J (2014) Understanding farmers’ adaptation intention to climate change: a structural equation modelling study in the Mekong Delta, Vietnam. Environmental Science & Policy 41:11–22

    Article  Google Scholar 

  • Li X, Wang Z, Hou X, Liu Z, Yin Y, Ding Y, Hu J, et al. (2014) Herders’ perception of climate change does not always fit with actual climate change. The Rangeland Journal 36(6):557–564

    Article  Google Scholar 

  • Lobell DB, Azzari G, Burke M, Gourlay S, Jin Z, Kilic T, Murray S (2018) Eyes in the sky, boots on the ground: assessing satellite-and ground-based approaches to crop yield measurement and analysis in Uganda. The World Bank

  • Maddison D (2007) The perception of and adaptation to climate change in Africa, vol 4308. World Bank Publications

  • Marlon JR, van der Linden S, Howe PD, Leiserowitz A, Woo SHL, Broad K (2018) Detecting local environmental change: the role of experience in shaping risk judgments about global warming. Journal of Risk Research 0(0):1–15

    Google Scholar 

  • Marx SM, Weber EU, Orlove BS, Leiserowitz A, Krantz DH, Roncoli C, Phillips J (2007) Communication and mental processes: experiential and analytic processing of uncertain climate information. Glob Environ Chang 17(1):47–58

    Article  Google Scholar 

  • Munro A (2020) Using experimental manipulation of questionnaire design and a kenyan panel to test for the reliability of reported perceptions of climate change and adaptation. GRIPS Discussion Papers

  • Muto M, Yamano T (2009) The impact of mobile phone coverage expansion on market participation: panel data evidence from Uganda. World development 37 (12):1887–1896

    Article  Google Scholar 

  • Mwesigye F, Matsumoto T (2016) The effect of population pressure and internal migration on land conflicts: implications for agricultural productivity in Uganda. World Dev 79:25–39

    Article  Google Scholar 

  • Orlove B, Roncoli C, Kabugo M, Majugu A (2010) Indigenous climate knowledge in southern Uganda: the multiple components of a dynamic regional system. Clim Chang 100(2):243–265

    Article  Google Scholar 

  • Rao K, Ndegwa WG, Kizito K, Oyoo A (2011) Climate variability and change: farmer perceptions and understanding of intra-seasonal variability in rainfall and associated risk in semi-arid Kenya. Experimental Agriculture 47(2):267–291

    Article  Google Scholar 

  • Roco L, Engler A, Bravo-Ureta BE, Jara-Rojas R (2015) Farmers’ perception of climate change in mediterranean Chile. Reg Environ Chang 15(5):867–879

    Article  Google Scholar 

  • Salerno J, Diem JE, Konecky BL, Hartter J (2019) Recent intensification of the seasonal rainfall cycle in equatorial Africa revealed by farmer perceptions, satellite-based estimates, and ground-based station measurements. Clim Chang 153(1-2):123–139

    Article  Google Scholar 

  • Schuman H, Presser S (1996) Questions and answers in attitude surveys: experiments on question form, wording, and context. Sage

  • Schwarz N (1999) Self-reports: how the questions shape the answers. American Psychologist 54(2):93

    Article  Google Scholar 

  • Silvestri S, Bryan E, Ringler C, Herrero M, Okoba B (2012) Climate change perception and adaptation of agro-pastoral communities in Kenya. Reg Environ Chang 12(4):791–802

    Article  Google Scholar 

  • Simelton E, Quinn CH, Batisani N, Dougill AJ, Dyer JC, Fraser ED, Mkwambisi D, Sallu S, Stringer LC (2013) Is rainfall really changing? farmers’ perceptions, meteorological data, and policy implications. Climate and Development 5 (2):123–138

    Article  Google Scholar 

  • Singh NP, Anand B, Khan MA (2018) Micro-level perception to climate change and adaptation issues: a prelude to mainstreaming climate adaptation into developmental landscape in India. Nat Hazards 92(3):1287–1304

    Article  Google Scholar 

  • Tanaka Y, Munro A (2014) Regional variation in risk and time preferences: evidence from a large-scale field experiment in rural Uganda. J Afr Econ 23(1):151–187

    Article  Google Scholar 

  • Waha K, Zipf B, Kurukulasuriya P, Hassan RM (2016) An agricultural survey for more than 9,500 African households. Scientific Data, 3

  • Weber EU (2010) What shapes perceptions of climate change? Wiley Interdiscip Rev Clim Chang 1(3):332–342

    Article  Google Scholar 

  • Yamano T, Sserunkuuma D, Otsuka K, Omiat G, Ainembabazi JH, Shimamura Y (2004) The 2003 REPEAT survey in Uganda: results. FASID Development Database, 09–01

  • Yamauchi C, Manang FE (2019) The impact of access to health facilities on maternal care use, travel patterns and health status: evidence from longitudinal data from Uganda. Economic Development and Cultural Change

  • Zampaligré N, Dossa LH, Schlecht E (2014) Climate change and variability: perception and adaptation strategies of pastoralists and agro-pastoralists across different zones of burkina faso. Reg Environ Chang 14(2):769–783

    Article  Google Scholar 

  • Zaval L, Keenan EA, Johnson EJ, Weber EU (2014) How warm days increase belief in global warming. Nat Clim Chang 4(2):143

    Article  Google Scholar 

Download references

Acknowledgements

Thanks are due to Yoko Kijima, Tomoya Matsumoto, Chikako Yamauchi, and the REPEAT survey team led by Bbosa Davis for data collection and assistance on task design. Thu Trang Vu provided capable research assistance. I gratefully acknowledge the useful feedback from EAERE conference attendees and funding from the National Graduate Institute for Policy Studies for piloting work and from JSPS Kakenhi Grant Number 25101002 for the main field work.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alistair Munro.

Ethics declarations

Conflict of interest

The author declares that he has no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1. Key survey questions

1.1 Version 1

  1. 1.

    Thinking back to 2003 (when the first Repeat survey was done) have you noticed any long term changes in the average temperature on your farm? (please explain). (record key phrases. Do not prompt subject)

  2. 2.

    Thinking back to 2003 (when the first Repeat survey was done) have you noticed any long term changes in the average rainfall on your farm? (please explain) (record key phrases. Do not prompt subject)

1.2 Version 2

  1. 1.

    Thinking back to 2003 (when the first Repeat survey was done), have you noticed any of the following long term changes in the average temperature on your farm?

    1. (a)

      Increased average temperatures 1=Yes, 2=No, 99 = Don’t know

    2. (b)

      Decreased average temperatures 1=Yes, 2=No, 99 = Don’t know

    3. (c)

      Altered Temperature range (the gap between lowest and highest temperatures has got bigger/small) 1=bigger, 2=smaller, 3=No change, 99 = Don’t know

  2. 2.

    Thinking back to 2003 (when the first Repeat survey was done), have you noticed any of the following long term changes in the average rainfall on your farm? (1=Yes, 2=No, 99 = Don’t know)

    1. (a)

      Increased rainfall /an extended rainy season

    2. (b)

      Decreased rainfall / a shortened rainy season

    3. (c)

      The rains come earlier

    4. (d)

      The rains come later

    5. (e)

      There are more periods of drought

    6. (f)

      There are fewer periods of drought

1.3 Both versions

  1. 1.

    What adjustments in your farming have you made to these long term shifts

    1. (a)

      Planted different varieties

    2. (b)

      Planted different crops

    3. (c)

      Crop diversification

    4. (d)

      Introduced inter-cropping

    5. (e)

      Different planting dates

    6. (f)

      Earlier planting

    7. (g)

      Later planting

    8. (h)

      Changing quantity of land under cultivation

    9. (i)

      Change from crops to livestock

    10. (j)

      Change from livestock to crops

    11. (k)

      Made adjustments to livestock management

    12. (l)

      Change from farming to non-farming activity

    13. (m)

      Change from non-farming to farming activity

    14. (n)

      Increased use of irrigation /groundwater / watering

    15. (o)

      Decreased use of irrigation / groundwater / watering

    16. (p)

      Changed use of chemicals and fertilizers

    17. (q)

      Increased use of water conservation techniques

    18. (r)

      Decreased use of water conservation techniques

    19. (s)

      Used soil conservation techniques

    20. (t)

      Shading and sheltering / tree planting

    21. (u)

      Used insurance or weather derivatives

    22. (v)

      Prayer or ritual offering

    23. (w)

      Other [specify]

    24. (x)

      Made no adaptation

Appendix 2. Additional tables

Table 10 Stated adaptation and measured change: all respondents versus only change noticers
Table 11 Stated adaptation and change noticers
Table 12 Stated adaptation and measured change 2012–2018

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Munro, A. Using experimental manipulation of questionnaire design and a Kenyan panel to test for the reliability of reported perceptions of climate change and adaptation. Climatic Change 162, 1081–1105 (2020). https://doi.org/10.1007/s10584-020-02709-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10584-020-02709-2

Keywords

Navigation