Is a local sample internationally representative? Reproducibility of four cognitive tests in family dogs across testing sites and breeds
A fundamental precept of the scientific method is reproducibility of methods and results, and there is growing concern over the failure to reproduce significant results. Family dogs have become a favoured species in comparative cognition research, but they may be subject to cognitive differences arising from genetic (breeding lines) or cultural differences (e.g. preferred training methods). Such variation is of concern as it affects the validity and generalisability of experimental results. Despite its importance, this problem has not been specifically addressed to date. Therefore, we aimed to test the influence of three factors on reproducibility: testing site (proximal environment), breed and sex (phenotype). The same experimenter tested cognitive performance by more than 200 dogs in four experiments. Additionally, dogs’ performance was tested in an obedience task administered by the owner. Breed of dog and testing site were found to influence the level of performance only mildly, and only in a means-end experiment and the obedience task. Our findings demonstrate that by applying the same test protocols on sufficiently large samples, the reported phenomena in these cognitive tests can be reproduced, but slight differences in performance levels can occur between different samples. Accordingly, we recommend the utilisation of well-described protocols supported by video examples of the whole experimental procedure. Findings should focus on the main outcome variables of the experiments, rather than speculating about the general importance of small or secondary performance outcomes which are more susceptible to random or local noise.
KeywordsBehaviour tests Reproducibility Replicability Dog Cross-country comparison Breed comparison
This research was supported by the European Union and the State of Hungary, co-financed by ESF Research Networking Programme “CompCog”: The Evolution of Social Cognition (www.compcog.org) (06-RNP-020), and the European Social Fund in the framework of TÁMOP 4.2.4. A/1-11-1-2012-0001 ‘National Excellence Program’. Dóra Szabó also received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant Agreement No. 680040), Ádám Miklósi also received support from the Hungarian Academy of Sciences (MTA 01 031), and Friederike Range received financial support from the FWF (Project Number: P24840-B16). We would like to thank Judit Abdai, Camille Hansart, Beáta Korcsok, Krisztina Kovács and Flóra Szánthó for their help. We are also grateful to all owners and their dogs for their participation in the study.
- Asendorpf JB, Conner M, De Fruyt F, De Houwer J, Denissen JJA, Fiedler K, Fiedler S, Funder DC, Kliegl R, Nosek BA, Perugini M, Roberts BW, Schmitt M, Van Aken MAG, Weber H, Wicherts JM (2013) Recommendations for increasing replicability in psychology. Eur J Pers 27:108–119. doi: 10.1002/per.1919 CrossRefGoogle Scholar
- Baldini S, Restani L, Baroncelli L, Coltelli M, Franco R, Cenni MC, Maffei L, Berardi N (2013) Enriched early life experiences reduce adult anxiety-like behavior in rats: a role for insulin-like growth factor 1. J Neurosci 33:11715–11723. doi: 10.1523/JNEUROSCI.3541-12.2013 CrossRefPubMedGoogle Scholar
- Brúder I (2010) Development of a test battery for evaluating dog personality and investigating the genetic background of personality traits. Eötvös Loránd University, BudapestGoogle Scholar
- Kampis G, Miklosi A, Viranyi Z, Gulyas L (2010) Video deep tagging and data archiving in the comparative mind database. In: Spink AJ, Grieco F, Krips OE, Loijens L, Noldus L, Zimmerman P (eds) Proceedings of the 7th international conference on methods and techniques in behavioral research, Eindhoven. Noldus, Wageningen, NL, pp 185–188Google Scholar
- Klein RA, Ratliff KA, Vianello M, Adams RB, Bahník Š, Bernstein MJ, Bocian K, Brandt MJ, Brooks B, Brumbaugh CC, Cemalcilar Z, Chandler J, Cheong W, Davis WE, Devos T, Eisner M, Frankowska N, Furrow D, Galliani EM, Hasselman F, Hicks JA, Hovermale JF, Hunt SJ, Huntsinger JR, Ijzerman H, John MS, Joy-Gaba JA, Kappes HB, Krueger LE, Kurtz J, Levitan CA, Mallett RK, Morris WL, Nelson AJ, Nier JA, Packard G, Pilati R, Rutchick AM, Schmidt K, Skorinko JL, Smith R, Steiner TG, Storbeck J, Van Swol LM, Thompson D, Van’T Veer AE, Vaughn LA, Vranka M, Wichman AL, Woodzicka JA, Nosek BA (2014) Investigating variation in replicability: a “many labs” replication project. Soc Psychol 45:142–152. doi: 10.1027/1864-9335/a000178 CrossRefGoogle Scholar
- Sidman M (1960) Tactics of scientific research: Evaluating experimental data in psychology, vol 5. Basic Books, New YorkGoogle Scholar
- Yang H, Harrington CA, Vartanian K, Coldren CD, Hall R, Churchill GA, Richter SH, Garner JP, Zipser B, Lewejohann L, Sachser N, Schindler B, Chourbaji S, Brandwein C, Gass P, Van Stipdonk N, Wolfer DP, Wu H (2008) Randomization in laboratory procedure is key to obtaining reproducible microarray results. PLoS ONE 3:e3724. doi: 10.1371/journal.pone.0003724 CrossRefPubMedPubMedCentralGoogle Scholar