Abstract
Rationale
Mathematical principles of reinforcement (MPR) provide the theoretical basis for a family of models of schedule-controlled behaviour. A model of fixed-ratio schedule performance that was applied to behaviour on progressive ratio (PR) schedules showed systematic departures from the data.
Objective
This study aims to derive a new model from MPR that will account for overall and running response rates in the component ratios of PR schedules, and their decline toward 0, the breakpoint.
Method
The role of pausing is represented in a real-time model containing four parameters: T 0 and k are the intercept and slope of the linear relation between post-reinforcement pause duration and the prior inter-reinforcer interval; a (specific activation) measures the incentive value of the reinforcer; δ (response time) sets biomechanical limits on response rate. Running rate is predicted to decrease with negative acceleration as ratio size increments, overall rate to increase and then decrease. Differences due to type of progression are explained as hysteresis in the control by reinforcement rates. Re-analysis of extant data focuses on the effects of acute treatment with antipsychotic drugs, lesions of the nucleus accumbens core, and destruction of orexinergic neurones of the lateral hypothalamus.
Results
The new model resolves some anomalies evident in earlier analyses, and provides new insights to the results of these interventions.
Conclusions
Because they can render biologically relevant parameters, mathematical models can provide greater power in interpreting the effects of interventions on the processes underlying schedule-controlled behaviour than is possible for first-order data such as the breakpoint.
Similar content being viewed by others
References
Aberman JE, Ward SJ, Salamone JD (1998) Effects of dopamine antagonists and accumbens dopamine depletions on time-constrained progressive-ratio performance. Pharmacol Biochem Behav 61:341–348
Arnold JM, Roberts DCS (1997) A critique of fixed and progressive ratio schedules used to examine the neural substrates of drug reinforcement. Pharmacol Biochem Behav 57:441–447
Aston-Jones G, Smith RJ, Moorman DE, Richardson KA (2009) Role of lateral hypothalamic orexin neurons in reward processing and addiction. Neuropharmacology 56:112–121
Baldo B, Kelley AE (2007) Discrete neurochemical coding of distinguishable motivational processes: insights from nucleus accumbens control of feeding. Psychopharmacology 91:439–459
Baron A, Mikorski J, Schlund M (1992) Reinforcement magnitude and pausing on progressive-ratio schedules. J Exp Anal Behav 58:377–388
Barr AM, Phillips AG (1999) Withdrawal following repeated exposure to d-amphetamine decreases responding for a sucrose solution as measured by a progressive ratio schedule of reinforcement. Psychopharmacology 141:99–106
Berridge CW, Espana RA, Vittoz NM (2010) Hypocretin/orexin in arousal and stress. Brain Res 1314:91–102
Bezzina G, Cheung THC, Asgari K, Hampson CL, Body S, Bradshaw CM, Szabadi E, Deakin JFW, Anderson IM (2007) Effects of quinolinic acid-induced lesions of the nucleus accumbens core on inter-temporal choice: a quantitative analysis. Psychopharmacology 195:71–84
Bezzina G, Body S, Cheung THC, Hampson CL, Deakin JFW, Anderson IM, Szabadi E, Bradshaw CM (2008) Effect of quinolinic acid-induced lesions of the nucleus accumbens core on performance on a progressive ratio schedule of reinforcement: implications for inter-temporal choice. Psychopharmacology 197:339–350
Bittar EG, Del-Claro K, Bittar LG, da Silva MCP (2012) Towards a mathematical model of within-session operant responding. J Exp Psychol: Anim Behav Proc. doi:10.1037/a0029086
Bizo LA, Killeen PR (1997) Models of ratio schedule performance. J Exp Psychol: Anim Behav Process 23:351–367
Bowman EM, Brown VJ (1998) Effects of excitotoxic lesions of the rat ventral striatum on the perception of reward cost. Exp Brain Res 123:439–448
Carlezon WA, Thomas MJ (2009) Biological substrates of reward and aversion: a nucleus accumbens activity hypothesis. Neuropharmacology 56(1):122–132
Cheeta S, Brooks S, Willner P (1995) Effects of reinforcer sweetness and the D2/D3 antagonist raclopride on progressive ratio operant performance. Behav Pharmacol 6:127–132
Corrigan PW, Reinke RR, Landsberger SA, Charate A, Toombs GA (2003) The effects of atypical antipsychotic medications on psychosocial outcomes. Schizophrenia Res 63:97–101
Covarrubias P, Aparicio CF (2008) Effects of reinforcer quality and step size on rats' performance under progressive ratio schedules. Behav Processes 78:246–252
Cunningham Owens DG (1999) A guide to the extrapyramidal side-effects of antipsychotic drugs. Cambridge University Press, Cambridge
Czachowski CL, Samson HH (1999) Breakpoint determination and ethanol self-administration using an across-session progressive ratio procedure in the rat. Alcoholism: Clin Exp Res 23:1580
Date Y, Ueta Y, Yamashita H, Yamaguchi H, Matsukura S, Kangawa K, Sakurai T, Yanagisawa M, Nakazato M (1999) Orexins, orexigenic hypothalamic peptides, interact with autonomic, neuroendocrine and neuroregulatory systems. Proc Natl Acad Sci USA 96:748–753
de Lecea L, Kilduff TS, Peyron C, Gao X, Foye PE, Danielson PE, Fukuhara C, Battenberg EL, Gautvik VT, Bartlett FS, Frankel WN, van den Pol AN, Bloom FE, Gautvik KM, Sutcliffe JG (1998) The hypocretins: hypothalamus-specific peptides with neuroexcitatory activity. Proc Natl Acad Sci USA 95:322–327
den Boon FS, Body S., Hampson CL, Bradshaw CM, Szabadi E, de Bruin N (2012) Effects of amisulpride and aripiprazole on progressive-ratio schedule performance: comparison with clozapine and haloperidol. J Psychopharmac. doi:10.1177/0269881111421974
Ferguson SA, Paule MG (1997) Progressive ratio performance varies with body weight in rats. Behav Processes 40:177–182
Ferster CB, Skinner BF (1957) Schedules of reinforcement. B. F. Skinner Foundation, Cambridge
Gancarz AM, Kausch MA, Lloyd DR, Richards JB (2012) Between-session progressive ratio performance in rats responding for cocaine and water reinforcers. Psychopharmacology. doi:10.1007/s00213-012-2637-9
Goudie AJ, Cooper GD, Cole JC, Sumnall HR (2007) Cyproheptadine resembels clozapine in vivo folowing both acute and chronic administration in rats. J Psychopharmac 21:179–190
Graham SJ, Langley RW, Bradshaw CM, Szabadi E (2001) Effects of haloperidol and clozapine on prepulse inhibition of the acoustic startle response and the N1/P2 auditory evoked potential in man. J Psychopharmac 15:243–250
Harris GC, Aston-Jones G (2006) Arousal and reward: a dichotomy in orexin function. Trends Neurosci 29:571–577
Hartfield A, Moore N, Clifton P (2003) Serotonergic and histaminergic mechanisms involved in intralipid drinking? Pharmacol Biochem Behav 76:251–258
Herrnstein RJ (1970) On the law of effect. J Exp Anal Behav 13:243–266
Herrnstein RJ (1974) Formal properties of the matching law. J Exp Anal Behav 21:159–164
Herrnstein RJ, Rachlin H, Laibson DI (1997) The matching law. Harvard University Press, Cambridge
Ho MY, Body S, Kheramin S, Bradshaw CM, Szabadi E (2003) Effects of 8-OH-DPAT and WAY-100635 on performance on a time-constrained progressive ratio schedule. Psychopharmacology (Berl) 167:137–144
Hodos W (1961) Progressive ratio as a measure of reward strength. Science 134:943–944
Hodos W, Kalman G (1963) Effects of increment size and reinforcer volume on progressive ratio performance. J Exp Anal Behav 6:387
Kheramin S, Body S, Herrera FM, Bradshaw CM, Szabadi E, Deakin JF, Anderson IM (2005) The effect of orbital prefrontal cortex lesions on performance on a progressive ratio schedule: implications for models of inter-temporal choice. Behav Brain Res 156:145–152
Killeen PR (1978) Superstition: a matter of bias, not detectability. Science 199:88–90
Killeen PR (1979) Arousal: Its genesis, modulation, and extinction. In: Zeiler MD, Harzem P (eds) Advances in analysis of behavior, vol 1, Reinforcement and the organization of behavior. Wiley, Chichester, pp 31–78
Killeen PR (1994) Mathematical principles of reinforcement. Behav Brain Sci 17:105–172
Killeen PR (1998) The first principle of reinforcement. In: Wynne CDL, Staddon JER (eds) Models of action: mechanisms for adaptive behavior. Erlbaum, Mahwah, pp 127–156
Killeen PR, Sitomer MT (2003) MPR. Behav Processes 62:49–64
Killeen PR, Hanson SJ, Osborne SR (1978) Arousal: its genesis and manifestation as response rate. Psychol Rev 85:571–581
Killeen PR, Posadas-Sanchez D, Johansen EB, Thrailkill EA (2009) Progressive ratio schedules of reinforcement. J Exp Psychol: Anim Behav Proc 35:35–50
Li N, He S, Parrish C, Delich J, Grasing K (2003) Differences in morphine and cocaine reinforcement under fixed and progressive ratio schedules; effects of extinction, reacquisition and schedule design. Behav Pharmac 14:619–630
Markou A, Weiss F, Gold LH, Caine SB, Schulteis G, Koob GF (1993) Animal models of drug craving. Psychopharmacology (Berl) 112:163–182
Meltzer H, Perry E, Jayathilake K (2003) Clozapine-induced weight gain predicts improvement in psychopathology. Schizophrenia Res 59:19–27
Mobini S, Chiang T-J, Ho M-Y, Bradshaw CM, Szabadi E (2000) Comparison of the effects of clozapine, haloperidol, chlorpromazine and d-amphetamine on performance on a time-constrained progressive ratio schedule and on locomotor behaviour in the rat. Psychopharmacology 152:47–54
Müller-Spahn F (2002) Current use of atypical antipsychotics. Eur Psychiat 17(suppl 4):377–384
Olarte Sánchez CM, Valencia Torres L, Body S, Cassaday HJ, Bradshaw CM, Szabadi E, Goudie AJ (2012a) A clozapine-like effect of cyproheptadine on progressive-ratio schedule performance. J Psychopharm 26:857–870
Olarte Sánchez CM, Valencia Torres L, Body S, Cassaday HJ, Bradshaw CM, Szabadi E (2012b) Effectof orexin-B-saporin induced lesions of the lateral hypothalamus on a progressive-ratio schedule. J Psychopharm 26:871–886
Perone M, Courtney K (1992) Fixed-ratio pausing: joint effects of past reinforcer magnitude and stimuli correlated with upcoming magnitude. J Exp Anal Behav 57:33–46
Peyron C, Tighe DK, van den Pol AN, de Lecea L, Heller HC, Sutcliffe JG, Kilduff TS (1998) Neurons containing hypocretin (orexin) project to multiple neuronal systems. J Neurosci 18:9996–10015
Ping-Teng C, Lee ES, Konz SA, Richardson NR, Roberts DCS (1996) Progressive ratio schedules in drug self-administration studies in rats: a method to evaluate reinforcing efficacy. J Neurosci Meth 66:1–11
Reilly MP (2003) Extending mathematical principles of reinforcement into the domain of behavioral pharmacology. Behav Processes 62:75–88
Richardson NR, Roberts DCS (1996) Progressive ratio schedules in drug self-administration studies in rats—a method to evaluate reinforcing efficacy. J Neurosci Methods 66:1–11
Rickard JF, Body S, Zhang Z, Bradshaw CM, Szabadi E (2009) Effect of reinforcer magnitude on performance maintained by progressive-ratio schedules. J Exp Anal Behav 91:75–87
Roberts DCS, Richardson NR (1992) Self-administration of psychostimulants using progressive ratio schedules of reinforcement. In: Boulton A, Baker G, Wu PH (eds) Neuromethods vol 24: animal models of drug addiction. Humana, New York, pp 233–269
Sakurai T, Amemiya A, Ishii M, Matsuzaki I, Chemelli RM, Tanaka H, Williams SC, Richarson JA, Kozlowski GP, Wilson S, Arch JR, Buckingham RE, Haynes AC, Carr SA, Annan RS, McNulty DE, Liu WS, Terrett JA, Elshourbagy NA, Bergsma DJ, Yanagisawa M (1998) Orexins and orexin receptors: a family of hypothalamic neuropeptides and G protein-coupled receptors that regulate feeding behavior. Cell 92:573–585, and page following p. 696
Salamone JD, Correa M, Farrar A, Mingote SM (2007) Effort-related functions of nucleus accumbens dopamine and associated forebrain circuits. Psychopharmacology 191:461–482
Schmelzeis MC, Mittleman G (1996) The hippocampus and reward: effects of hippocampal lesions on progressive-ratio responding. Behav Neurosci 110:1049–1066
Schneider BA (1969) A two-state analysis of fixed-interval responding in the pigeon. J Exp Anal Behav 12:677–687
Siegel JM (2004) Hypocretin (orexin): role in normal behavior and neuropathology. Ann Rev Psychol 55:125–148
Siegel JM (2005) Hypocretin/orexin and motor function. In: Nishino S, Sakurai T (eds) The orexin/hypocretin system: physiology and pathophysiology. Humana Press, New York
Skinner BF (1948) Superstition in the pigeon. J Exp Psychol 38:168–172
Skjoldager P, Pierre PJ, Mittleman G (1993) Reinforcer magnitude and progressive ratio responding in the rat: effects of increased effort, prefeeding, and extinction. Learn Motiv 24:303–343
Stafford D, Branch MN (1998) Effects of step size and break-point criterion on progressive-ratio performance. J Exp Anal Behav 70:123–138
Stafford D, LeSage MG, Glowa JR (1998) Progressive-ratio schedules of drug delivery in the analysis of drug self-administration: a review. Psychopharmacology (Berl) 139:169–184
Sutton RS, Barto AG (1990) Time-derivative models of Pavlovian reinforcement. In: Gabriel M, Moore J (eds) Learning and computational neuroscience: foundations of adaptive networks. MIT Press, Cambridge, pp 497–537
Timberlake W, Lucas GA (1985) The basis of superstitious behavior: chance contingency, stimulus substitution, or appetitive behavior? J Exp Anal Behav 44:279–299
Timberlake W, Lucas GA (1989) Behavior systems and learning: from misbehavior to general principles. In: Klein SB, Mowrer RR (eds) Contemporary learning theories: instrumental conditioning theory and the impact of constraints on learning. Erlbaum, Hillsdale, pp 237–275
Trivedi P, Yu H, MacNeil DJ, Van der Ploeg LH, Guan XM (1998) Distribution of orexin receptor mRNA in the rat brain. FEBS Lett 438:71–75
Valencia-Torres L, Olarte-Sánchez CM, da Costa AS, Body S, Bradshaw CM, Szabadi E (2012) Nucleus accumbens and delay discounting in rats: evidence from a new quantitative protocol for analysing inter-temporal choice. Psychopharmacology 219:271–283
Wise RA (1982) Neuroleptics and operant behavior: the anhedonia hypothesis. Brain Behav Sci 5:39–87
Wise RA (2006) Role of brain dopamine in food reward and reinforcement. Phil Trans Roy Soc Lond B: Biol Sci 361:1149–1158
Wynne CDL, Staddon JER, Delius JD (1996) Dynamics of waiting in pigeons. J Exp Anal Behav 65:603–618
Zhang Z, Rickard JF, Asgari K, Body S, Bradshaw CM, Szabadi E (2005a) Quantitative analysis of the effects of some "atypical" and "conventional" antipsychotics on progressive ratio schedule performance. Psychopharmacology (Berl) 179:489–497
Zhang Z, Rickard JF, Body S, Asgari K, Bradshaw CM, Szabadi E (2005b) Comparison of the effects of clozapine and 8-hydroxy-2-(di-n-propylamino) tetralin (8-OH-DPAT) on progressive ratio schedule performance: evidence against the involvement of 5-HT 1A receptors in the behavioural effects of clozapine. Psychopharmacology (Berl) 181:381–391
Conflicts of interest
None.
Author information
Authors and Affiliations
Corresponding authors
Additional information
The order of authorship is alphabetical. Correspondence may be addressed to either author.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Appendix
Appendix
A1. Coupling
Not all responding that is excited by incentives results in proper, on-target responding (Killeen 1978; Skinner 1948; Timberlake and Lucas 1985, 1989). The coupling coefficient tells us what proportion of stimulated behaviour will be measured on the lever. Because the reinforcer itself interrupts the stream of target response, it truncates the reach of those delay of reinforcement gradients. On PR schedules, however, animals have experience with long sequences of the target response on the longest ratios. It can be assumed that these drive coupling arbitrarily high. In the current model, it is set 1, to conserve that parameter: C PR ≈ 1.
A2. Reduction to FR model and prediction of de jure breakpoint
Rewrite Eq. 6c for the case where successive values of N and T TOT are equal:
Simplify by setting T 0 to 0, and solve:
Compute overall response rate:
Thus, the FR model.
Set to 0 to compute the de jure break point:
A3. Computer pseudocode*
*The standard values are initiated to provide some ‘burn-in’, with the assumption of a nominal 2 s T TOT under an FR1. SSs is an unweighted sum of SS RUN and SSOVERALL.
Rights and permissions
About this article
Cite this article
Bradshaw, C.M., Killeen, P.R. A theory of behaviour on progressive ratio schedules, with applications in behavioural pharmacology. Psychopharmacology 222, 549–564 (2012). https://doi.org/10.1007/s00213-012-2771-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00213-012-2771-4