Introduction

Serotonin (5-hydroxytryptamine, 5-HT) is believed to be involved in the control of a variety of personality traits, including impulsivity, risk taking behaviour (Coccaro 1989; Mehlman et al. 1994) and sensation seeking (Zuckerman 1993, 1996; Netter et al. 1996; Roberti 2004; Pascual et al. 2007). Sensation seeking describes a personality trait characterised by voluntary participation in activities involving personal risk (Zuckerman and Neeb 1979). In humans, individual differences in sensation seeking predict individual differences in vulnerability to psychiatric disorders like depression and schizophrenia (Dervaux et al. 2001; Laget et al. 2006) and to drug abuse (Patkar et al. 2004; Kelly et al. 2006). It has previously been reported that individual differences in the exploratory response to a novel environment in rats can predict their neurochemical and behavioural response to cocaine and amphetamine (Piazza et al. 1989; Hooks et al. 1991, 1992; Cools et al. 1997; Ranaldi et al. 2001; Chefer et al. 2003; Verheij et al. 2008). Moreover, rats that are marked by a differential exploration response to novelty have been found to differ in their behavioural response in animal models of schizophrenia (Ellenbroek et al. 1995; Ellenbroek and Cools 2002). On the basis of these and related studies, it has been suggested that enhanced novelty seeking in rats resembles sensation seeking in humans (Dellu et al. 1996; Cools and Ellenbroek 2002; Ballaz et al. 2007a, b, c).

In the present study, we used two types of rat that are known to differ in novelty seeking. Rats that are marked by a high level of exploration in a new environment are labelled high responders (HR) to novelty, whereas rats that are marked by a low level of exploration in a new environment are labelled low responders (LR) to novelty (Cools et al. 1990, 1993; Cools and Ellenbroek 1996, 2002; Cools and Gingras 1998). It has repeatedly been shown that environmental challenges like novelty or pharmacological challenges like cocaine increase accumbal dopamine levels more strongly in HR than in LR rats (Hooks et al. 1991, 1992; Saigusa et al. 1999; Verheij and Cools 2007, 2008; Verheij et al. 2008). Given that serotonergic agents are known to change the release of dopamine (Kilpatrick et al. 1996; De Deurwaerdere and Spampinato 1999; Ichikawa and Meltzer 2000), it is hypothesised that HR and LR rats differ in their sensitivity to these agents. Because individual differences in sensitivity to serotonergic agents have frequently been reported in humans (Lerer and Macciardi 2002; Pickar 2003; Bolonna et al. 2004; Lane et al. 2005; McMahon et al. 2006), rat studies are expected to contribute to our knowledge of the neuronal substrates that underlie these individual differences. In addition, rat studies may help to identify which subjects are sensitive to a particular serotonergic therapy and which subjects are not.

The aim of the present study was to investigate whether HR and LR rats differ in their behavioural response to serotonergic agents. These agents are known to alter the behavioural response on the elevated T-maze. The elevated T-maze consists of one enclosed arm perpendicular to two open arms (Viana et al. 1994). Rats are placed at the distal end of the enclosed arm, and the time to enter one of the open arms (with four paws) is scored. Drugs that mimic the action of serotonin increase the time to approach the open arms (Graeff et al. 1996a, b 1998; Mora et al. 1997; Viana et al. 1997), whereas drugs that block the action of serotonin exert opposite effects (Gargiulo et al. 1996; Mora et al. 1997; Graeff et al. 1998). Behavioural experiments have revealed that the avoidance of the open arms increases after a systemic injection of 5-HT releasers like d-fenfluramine (Graeff et al. 1996a, b, 1997), 5-HT3 agonists like SR77227A (Andrews and File 1992) and 5-HT2 agonists like DOI (Peng et al. 2004; Bull et al. 2004) and mCPP (Mora et al. 1997; Graeff et al. 1998; Zangrossi et al. 2001). Interestingly, 5-HT1A agonists like 8-OH-DPAT have been found to increase (Critchley and Handley 1987; Kshama et al. 1990; Treit et al. 1993) or to decrease (Dunn et al. 1989; Collinson and Dawson 1997) avoidance behaviour (see also “Discussion”). Additional experiments have also revealed that the avoidance of the open arms decreases after a systemic injection of 5-HT3 antagonists like ondansetron (Filip et al. 1992; Brioni et al. 1994; Sonavane et al. 2002) and 5HT2 antagonists like ritanserin (Critchley and Handley 1987; Mora et al. 1997; Graeff et al. 1998; Zangrossi et al. 2001). Finally, the 5-HT1A antagonist WAY100635 has been found to increase rather than to decrease the avoidance of the open arms (Collinson and Dawson 1997; Peng et al. 2004).

The present study showed that HR and LR rats indeed differed in their sensitivity to all of the serotonergic agents that are listed above.

Methods

Animals

Adult male Wistar rats (180–220 g at the start of the experiments) were used. These animals were reared in the central animal house of the Radboud University of Nijmegen, The Netherlands. Initially, rats were housed in groups of three per cage in a temperature-controlled environment (20 ± 2°C) under a 12/12 h light/dark cycle (lights on at 0730 hours). Food (Ssniff) and water were freely available at all time, except during the open-field and T-maze test (see below). Rats were isolated 3 or 4 days prior to the open-field selection (Cools et al. 1990). The experiments were performed in accordance with institutional, national and international policies on animal care and welfare. All procedures were in agreement with the NRC (National Research Council) 2003 guidelines for the care and use of mammals in neuroscience and behavioural research.

Open-field selection

Half an hour before the open-field selection, rats were placed in the experimental room to get accustomed to the new environment. Next, a rat was placed on a novel open field, which consisted of a black table (160 × 160 cm) made of Perspex (Cools et al. 1990). The table, which was 70 cm above the ground, was surrounded by a neutral environment. Light intensity was 170 lx at the middle of the open field. Once the rat was placed in the centre of the open field, its behaviour was recorded for 30 min using a computerised automated tracking system as described by Cools et al. (1990). Selection parameters were ambulation and habituation time. Ambulation was defined as the total distance travelled (cm) in 30 min. Habituation time was defined as the period (s) from the start of the selection until the moment the rat showed no locomotor activity for at least 90 s. Rats that habituated in less than 480 s and walked less than 4,800 cm were classified as LR (Cools et al. 1990, 1997). Rats that habituated in more than 840 s and travelled more than 6,000 cm were classified as HR (Cools et al. 1990, 1997). The selection procedure took place between 0900 and 1800 hours. The table was cleaned with 70% alcohol after each rat.

Elevated T-maze

Seven days after exposure to the open field, rats were tested on the elevated T-maze. The elevated T-maze was adapted from the elevated plus-maze (Viana et al. 1994) and the procedure to test our rats (see “Introduction”) was slightly modified from Graeff et al. (1996a, b). The arms of the T-maze were of equal dimensions (50 × 10 cm). The enclosed arm contained a wall of 40 cm, and the two open arms were surrounded by a rim of 1 cm. The three arms were connected by a central square (10 × 10 cm). Light intensity was 2.5 lx at the central square.

Twenty-five min before the first exposure to the T-maze, rats received a single injection of a particular serotonergic agent or a combination of serotonergic agents (volume, 1 ml/kg, i.p.), whereafter the rats were left undisturbed in their home cage. Different rats were used for each dose of every serotonergic agent or each combination of serotonergic agents. Twenty-five min after the injection, rats were placed at the distal end of the enclosed arm facing the central square, and the time to enter one of the open arms (with four paws) was recorded. The time to avoid the open arms was assessed in each of five consecutive trials (inter-trial interval, 10 min). All rats were removed from the elevated T-maze once they reentered the enclosed arm. A cut-off time of 600 s was used. The T-maze was cleaned with 70% alcohol after each trial. All experiments were performed between 0900 and 1400 hours (Griebel et al. 1993).

Drugs

The following drugs were used: 5-HT1A antagonist—WAY100635 (N-{2-[4-(2-methoxyphenyl)-1-piperazinyl] ethyl}-N-(2-pyridinyl)cyclo-hexanecarboxamide trihydrochloride); 5-HT1A agonist—8-OH-DPAT (8-hydroxy-2-(di-N-propylamino)-tetralin); 5-HT2A/C antagonist—ritanserin (6-[2-[bis(4-flourophenyl)methylene]-1-piperidenyl]ethyl]-7-methyl-5H-thiazolo[3,2-a]pyri-midin-5-one); 5-HT2(A)/C agonists—DOI (1-(2,5-dimethoxy-4-iodophenyl)-2-aminopropane) and mCPP (1-(3-chlorophenyl)-piperazine hydrochloride); 5-HT3 agonist—SR57227A (4-amino-1-(6-chloro-2-pyridyl)-piperidine hydrochloride); 5-HT releaser and reuptake inhibitor—d-fenfluramine ((+)-N-ethyl-α-methyl-m-[trifluoromethyl]-phenethylamine). All these drugs were purchased from Sigma. The 5-HT3 antagonist ondansetron (1,2,3,9-tetrahydro-9-methyl-3-[(2-methyl-1H-imidazol-1-yl)methyl]-4H-carbazol-4-one) was purchased from Glaxo. The drugs were chosen because they have been found to affect behaviour in the elevated plus-maze and T-maze (see “Introduction”). To our knowledge, no plus-maze or T-maze experiments have been performed with SR57227A, but it has been shown to produce other behavioural effects at the doses that were applied (Poncelet et al. 1995). All drugs were dissolved in saline. Ritanserin was dissolved in saline with a drop of acetic acid, and the pH was adjusted with sodium hydroxide (Schreiber et al. 1998; Cayetanot et al. 2001). In order to maintain the levels of stress as low as possible, rats received their injection only once before the first trial and not prior to every trial (Viana et al. 1994; Graeff et al. 1996a, b). In those cases where a particular agonist or antagonist changed avoidance behaviour, a cocktail of this drug with, respectively, an antagonist or agonist (volume, 1 ml/kg, i.p.) was also injected.

Statistical analysis

Data were analysed using a two-way ANCOVA with type of rat and treatment as independent factors and trials as a covariate. Where appropriate, this test was followed by a one-way ANCOVA and post hoc LSD analysis. A total number of 511 rats were exposed to the T-maze. Extreme values (values more than three times the interquartile range) were identified using the explore function of SPSS 12.0.1 and excluded from analysis (Tuinstra et al. 2000). Thirty-eight rats (7% of the total number of rats) were excluded at one or more trials. The n value (see Figs. 1, 2, 3, 4 and 5) represent the number of rats included on trial 5 (number of rats, 509 − 38 = 471). A probability level of 0.05 was considered to be statistically significant.

Fig. 1
figure 1

Effects of the 5-HT3 agonist SR57227A (a, b), the 5-HT3 antagonist ondansetron (c, d) and the combination of both (e, f) on the duration of avoidance of the open arms in LR (left) and HR (right) rats. Asterisks Significant change vs saline. Number signs Significant decrease vs SR57227A

Full size image
Fig. 2
figure 2

Effects of the 5-HT2C agonists DOI (a, b) and mCPP (c, d), the 5-HT2A/C antagonist ritanserin (e, f) and the combination of mCPP and ritanserin (g, h) on the duration of avoidance of the open arms in LR (left) and HR (right) rats. Asterisks Significant change vs saline. Number signs Significant increase/decrease vs mCPP

Full size image
Fig. 3
figure 3

Effects of the 5-HT1A agonist 8-OH-DPAT (a, b), the 5-HT1A antagonist WAY100635 (c, d) and the combination of both (e, f) on the duration of avoidance of the open arms in LR (left) and HR (right) rats. Asterisks Significant change vs saline. Number signs Significant decrease vs WAY100635

Full size image
Fig. 4
figure 4

Effects of the 5-HT reuptake inhibitor/5-HT releaser d-fenfluramine (a, b) and the combination of the 5-HT3 antagonist ondansetron and d-fenfluramine (c, d) on the duration of avoidance of the open arms in LR (left) and HR (right) rats. Asterisks Significant change vs saline. Number signs Significant decrease vs d-fenfluramine

Full size image
Fig. 5
figure 5

Effects of the 5-HT2A/C antagonist ritanserin (a, b) and the 5-HT3 antagonist ondansetron (c, d) on the WAY100635-induced increase of the duration avoidance behaviour in LR (left) and HR (right) rats. Asterisks Significant change vs saline. Number signs Significant decrease vs WAY100635

Full size image

Results

Open-field selection

The open-field selection procedure revealed 24% LR (n = 249) rats and 25% HR (n = 260) rats. The average distance travelled in 30 min (± SEM) was 3,624 ± 88 cm in LR rats and 8,316 ± 150 cm in HR rats. The average habituation time (±SEM) was 324 ± 16 s in LR rats and 1,310 ± 29 s in HR rats. All efforts were made to include the rats that did not fulfil the criteria (n = 531) in other studies (Verheij et al. 2007).

Saline

The duration of avoidance of the open arms was larger in saline-treated LR rats than in saline-treated HR rats (Figs. 1, 2, 3 and 4—ANCOVA: type effect, F (1,90) = 28.73, p < 0.001). The duration of avoidance behaviour was similar between the two types of rat during trials 1 and 2. During the remaining trials, LR rats were marked by more avoidance behaviour than HR rats. This larger increase of avoidance behaviour in LR rats compared to HR rats was also observed under naive conditions (data not shown).

5-HT3 drugs

Figure 1 shows that the 5-HT3 agonist SR57227A differentially affected the duration of avoidance behaviour in LR and HR rats (ANCOVA: type × treatment effect, F (3,341) = 13.01, p < 0.001). SR57227A increased avoidance behaviour in both LR (Fig. 1a—ANCOVA: treatment effect, F (3,172) = 11.56, p < 0.001) and HR rats (Fig. 1b—ANCOVA: treatment effect, F (3,168) = 19.80, p < 0.001). In LR rats, all doses of SR57227A increased avoidance behaviour (Fig. 1a—0 vs 1 mg/kg, p = 0.010; 0 vs 2 mg/kg, p < 0.001; 0 vs 3 mg/kg, p = 0.005), whereas only the highest dose of this drug increased avoidance behaviour in HR rats (Fig. 1b—0 vs 1 mg/kg, n.s.; 0 vs 2 mg/kg, n.s.; 0 vs 3 mg/kg, p < 0.001). In addition, SR57227A increased the duration of avoidance already at trials 1–4 in LR rats (Fig. 1a), but only at trials 3–5 in HR rats (Fig. 1b). Figure 1 also illustrates that the 5-HT3 antagonist ondansetron had differential effects in LR and HR rats (ANCOVA: type × treatment effect, F (2,286) = 6.82, p = 0.001). Ondansetron decreased the duration of avoidance behaviour in LR rats (Fig. 1c—ANCOVA: treatment effect, F (2,132) = 8.87, p < 0.001; 0 vs 0.01 mg/kg, p = 0.026; 0 vs 0.1 mg/kg, p < 0.001), but not in HR rats (Fig. 1d—ANCOVA: treatment effect, n.s.; 0 vs 0.01 mg/kg, n.s.; 0 vs 0.1 mg/kg, n.s.; and 0 vs 5.0 mg/kg, n.s.). Ondansetron attenuated the SR57227A-induced increase of avoidance in both LR rats (Fig. 1e—SR57227A vs ondansetron + SR57227A: ANCOVA, treatment effect, F (1,71) = 6.76, p = 0.011) and HR rats (Fig. 1f—SR57227A vs ondansetron + SR57227A: ANCOVA, treatment effect, F (1,75) = 6.05, p = 0.016). Finally, the duration of avoidance induced by the combination of ondansetron and SR57227A differed from the duration of avoidance induced by ondansetron alone in LR rats (Fig. 1e—ondansetron + SR57227A vs ondansetron: ANCOVA, treatment effect, F (1,65) = 13.87, p < 0.001) and in HR rats (Fig. 1f—ondansetron + SR57227A vs ondansetron: ANCOVA, treatment effect, F (1,62) = 6.38, p = 0.014).

5-HT2 drugs

Figure 2 illustrates that the 5-HT2A/C agonist DOI differentially affected the duration of avoidance behaviour in LR and HR rats (ANCOVA: type × treatment effect, F (2,294) = 7.69, p = 0.001). DOI decreased avoidance behaviour in LR rats (Fig. 2a—ANCOVA: treatment effect, F (2,156) = 3.30, p = 0.040; 0 vs 0.5 mg/kg, p = 0.039; 0 vs 1 mg/kg, p = 0.041), but increased avoidance behaviour in HR rats (Fig. 2b—ANCOVA: treatment effect, F (2,137) = 5.83, p = 0.004; 0 vs 0.5 mg/kg, n.s.; 0 vs 1 mg/kg, p = 0.001). Since a decrease in the duration of avoidance behaviour following a 5-HT2 agonist was unexpected (see “Introduction”), the effects of an additional 5-HT2 agonist were tested. The effects of the 5-HT2C agonist mCPP were similar to the effects of DOI. Figure 2 shows that mCPP had differential effects in LR and HR rats (ANCOVA: type × treatment effect, F (2,265) = 12.10, p < 0.001). Similar to DOI, mCPP decreased avoidance in LR (Fig. 2c—ANCOVA: treatment effect, F (2,131) = 3.72, p = 0.027; 0 vs 0.2 mg/kg, p = 0.026; 0 vs 0.4 mg/kg, p = 0.040), whereas mCPP increased avoidance in HR (Fig. 2d—ANCOVA: treatment effect, F (2,133) = 13.12, p < 0.001; 0 vs 0.2 mg/kg, n.s.; 0 vs 0.4 mg/kg, p < 0.001). Figure 2 also illustrates that the 5-HT2A/C antagonist ritanserin had differential effects in LR and HR rats (ANCOVA: type × treatment effect, F (3,428) = 3.59, p = 0.014). Ritanserin decreased the duration of avoidance behaviour in LR (Fig. 2e—ANCOVA: treatment effect, F (3,192) = 4.68, p = 0.004; 0 vs 0.1 mg/kg, n.s.; 0 vs 0.2 mg/kg, n.s.; 0 vs 0.3 mg/kg, p < 0.001), but not in HR (Fig. 2f—ANCOVA: treatment effect, n.s.; 0 vs 0.1 mg/kg, n.s.; 0 vs 0.2 mg/kg, n.s.; 0 vs 0.3 mg/kg, n.s.; 0 vs 0.5 mg/kg, n.s.). Both the mCPP-induced decrease of avoidance behaviour in LR rats and the mCPP-induced increase of avoidance behaviour in HR rats were attenuated by ritanserin (LR rats (Fig. 2g)—mCPP vs mCPP + ritanserin: ANCOVA, treatment effect, F (1,79) = 11.72, p = 0.001; HR rats (Fig. 2h)—mCPP vs mCPP + ritanserin: ANCOVA, treatment effect, F (1,73) = 4.87, p = 0.030). Finally, the duration of avoidance induced by the combination of ritanserin and mCPP differed from the duration of avoidance induced by ritanserin alone in LR (Fig. 2g—mCPP + ritanserin vs ritanserin: ANCOVA, treatment effect, F (1,92) = 5.68, p = 0.019), but not in HR rats (Fig. 2h—mCPP + ritanserin vs ritanserin: ANCOVA, treatment effect, n.s.).

5-HT1 drugs

The 5-HT1A agonist 8-OH-DPAT had differential effects on the duration of avoidance behaviour in LR and HR rats (ANCOVA: type × treatment effect, F (3,385) = 4.19, p = 0.006). Figure 3 illustrates that 8-OH-DPAT decreased avoidance behaviour in LR rats (Fig. 3a—ANCOVA: treatment effect, F (3,163) = 2.88, p = 0.038; 0 vs 0.1 mg/kg, p = 0.011; 0 vs 0.3 mg/kg, p = 0.023; 0 vs 0.5 mg/kg, p = 0.060), whereas it increased avoidance behaviour in HR rats (Fig. 3b—ANCOVA: treatment effect, F (3,221) = 8.07, p < 0.001; 0 vs 0.1 mg/kg, n.s.; 0 vs 0.3 mg/kg, p < 0.001; and 0 vs 0.5 mg/kg, n.s.). Figure 3 also shows that the 5-HT1A antagonist WAY100635 had differential effects in LR and HR rats (ANCOVA: type × treatment effect, F (3,332) = 7.89, p < 0.001). WAY100635 increased avoidance behaviour in both LR rats (Fig. 3c—ANCOVA: treatment effect, F (3,166) = 12.49, p < 0.001; 0 vs 0.01 mg/kg, p = 0.011; 0 vs 0.05 mg/kg, p = 0.052; 0 vs 0.1 mg/kg, p < 0.001) and HR rats (Fig. 3d—ANCOVA: treatment effect, F (3,165) = 11.76, p < 0.001; 0 vs 0.01 mg/kg, p < 0.001; 0 vs 0.05 mg/kg, p < 0.001; 0 vs 0.1 mg/kg, p = 0.001). However, WAY100635 increased the duration of avoidance already at trials 1–4 in LR rats (Fig. 3c), but only at trials 2–5 in HR rats (Fig. 3d). The WAY100635-induced increase of avoidance was counteracted by 8-OH-DPAT in both LR rats (Fig. 3e—WAY100635 vs 8-OH-DPAT + WAY100635: ANCOVA, treatment effect, F (1,79) = 33.02, p < 0.001) and HR rats (Fig. 3f—WAY100635 vs 8-OH-DPAT + WAY100635: ANCOVA, treatment effect, F (1,67) = 15.52, p < 0.001). Finally, the duration of avoidance following the combination of 8-OH-DPAT and WAY100635 was significantly different from the duration of avoidance following 8-OH-DPAT alone in LR rats (Fig. 3e—8-OH-DPAT + WAY100635 vs 8-OH-DPAT: ANCOVA, treatment effect, F (1,83) = 4.68, p = 0.034) and in HR rats (Fig. 3f—8-OH-DPAT + WAY100635 vs 8-OH-DPAT: ANCOVA, treatment effect, F (1,77) = 4.02, p = 0.049).

d-Fenfluramine

Figure 4 illustrates that the 5-HT releaser/reuptake blocker d-fenfluramine differentially affected the duration of avoidance behaviour in LR and HR rats (ANCOVA: type × treatment effect, F (2,285) = 3.75, p = 0.026). Although d-fenfluramine increased the duration of avoidance behaviour in both LR rats (Fig. 4a—ANCOVA: treatment effect, F (2,150) = 7.50, p = 0.001; 0 vs 0.1 mg/kg, p = 0.034; 0 vs 0.3 mg/kg, p < 0.001) and HR rats (Fig. 4b—ANCOVA: treatment effect, F (2,134) = 8.48, p < 0.001; 0 vs 0.1 mg/kg, p < 0.001; 0 vs 0.3 mg/kg, p = 0.006), post hoc analysis revealed that, similar to SR57227A, d-fenfluramine already increased avoidance behaviour at trials 1–3 in LR (Fig. 4a), but only at trials 3–5 in HR rats (Fig. 4b). Because our data suggest that the action of d-fenfluramine may be due to stimulation of 5-HT3 receptors (see “Discussion”), we also investigated the effects of the 5-HT3 antagonist ondansetron on the d-fenfluramine-induced changes of avoidance. Ondansetron was found to attenuate the d-fenfluramine-induced increase of avoidance in both LR rats (Fig. 4c—d-fenfluramine vs ondansetron + d-fenfluramine: ANCOVA, treatment effect, F (1,95) = 4.85, p = 0.030) and HR rats (Fig. 4d—d-fenfluramine vs ondansetron + d-fenfluramine: ANCOVA, treatment effect, F (1,77) = 11.29, p = 0.001). The duration of avoidance induced by the combination of ondansetron and d-fenfluramine was different from the duration of avoidance induced by ondansetron alone in LR rats (Fig. 4c—ondansetron + d-fenfluramine vs d-fenfluramine: ANCOVA, F (1,78) = 28.66, p < 0.001) and in HR rats (Fig. 4d—ondansetron + d-fenfluramine vs d-fenfluramine: ANCOVA, treatment effect, F (1,56) = 7.46, p = 0.008).

5-HT1 and 5-HT2/5-HT3 drugs

In order to investigate whether the behavioural effects of WAY100635, which is suggested to act at presynaptic 5-HT1A receptors (see “Discussion”), could be inhibited by an antagonist that is suggested to act at postsynaptic 5-HT2 or 5-HT3 receptors (see “Discussion”), rats were treated with a cocktail of WAY100635 and ritanserin or WAY100635 and ondansetron. The 5-HT2A/C antagonist ritanserin did not inhibit the WAY100635-induced increase of avoidance in either LR rats (Fig. 5a—WAY100635 vs ritanserin + WAY100635: ANCOVA, treatment effect, n.s.) or HR rats (Fig. 5b—WAY100635 vs ritanserin + WAY100635: ANCOVA, treatment effect, n.s.). The duration of avoidance induced by the combination of ritanserin and WAY100635 differed from the duration of avoidance induced by ritanserin alone in LR (Fig. 5a—ritanserin + WAY100635 vs ritanserin: ANCOVA, treatment effect, F (1,107) = 37.82, p < 0.001) and in HR (Fig. 5b—ritanserin + WAY100635 vs ritanserin: ANCOVA, treatment effect, F (1,83) = 15.42, p < 0.001). The 5-HT3 antagonist ondansetron inhibited the WAY100635-induced increase of avoidance in both LR rats (Fig. 5c—WAY100635 vs ondansetron + WAY100635: ANCOVA, treatment effect, F (1,73) = 40.48, p < 0.001) and HR rats (Fig. 5d—WAY100635 vs ondansetron + WAY100635: ANCOVA, treatment effect, F (1,74) = 6.51, p = 0.013). The duration of avoidance induced by the combination of ondansetron and WAY100635 differed from the duration of avoidance induced by ondansetron alone in LR rats (Fig. 5c—ondansetron + WAY100635 vs ondansetron: ANCOVA, treatment effect, F (1,63) = 13.11, p = 0.001) and in HR rats (Fig. 5d—ondansetron + WAY100635 vs ondansetron: ANCOVA, treatment effect, F (1,61) = 6.70, p = 0.012).

Discussion

Summary of the results

The aim of the present study was to assess individual differences in the sensitivity to serotonergic drugs between rats that were characterised as HR and LR to novelty. The 5-HT3 agonist SR57227A increased the duration of avoidance behaviour in both types of rat (Fig. 1a/b). However, higher doses of SR57227A were required to alter the duration of avoidance behaviour in HR than in LR rats (Fig. 1a/b). The 5-HT3 antagonist ondansetron decreased avoidance behaviour in LR rats, but had no effects in HR rats (Fig. 1c/d). Remarkably, the 5-HT2 agonists mCPP and DOI decreased the duration of avoidance behaviour in LR rats, but increased the duration of this behaviour in HR rats (Fig. 2a–d). The 5-HT2 antagonist ritanserin decreased avoidance behaviour in LR rats, but had no effects in HR rats (Fig. 2e/f). The 5-HT1A agonist 8-OH-DPAT decreased the duration of avoidance behaviour in LR rats, but increased the duration of this behaviour in HR rats (Fig. 3a/b). Both the 5-HT1A antagonist WAY100635 (Fig. 3c/d) and the serotonin releaser/reuptake inhibitor d-fenfluramine (Fig. 4a/b) increased avoidance behaviour in both types of rat. However, the onset of the effects of WAY100635 (Fig. 3c/d) and d-fenfluramine (Fig. 4a/b) was faster in LR than in HR rats. The same held true for the onset of the effects of SR57227A (Fig. 1a/b). Finally, the duration of avoidance behaviour was larger in control LR rats compared to control HR rats (Figs. 1, 2, 3 and 4). These data demonstrate that LR and HR rats differ in their sensitivity to all serotonergic drugs that were tested. Apart from the direction of the effects of DOI and mCPP in LR rats, the direction of the effects of the remaining serotonergic agents nicely fit in with the available literature (for reference, see “Introduction”).

Below, a model will be presented to explain our results. The model is based on the original finding by Graeff et al. that the duration of avoidance of the open arms becomes larger as synaptic serotonin levels increase (Graeff et al. 1996a, b, 1998; Mora et al. 1997; Viana et al. 1997). The finding that d-fenfluramine, a drug that is known to increase the levels of serotonin in the synapse (Consolo et al. 1979; Gundlah et al. 1997), increases the duration of avoidance behaviour in both LR (Fig. 4a) and HR (Fig. 4b) rats is nicely in line with the previously reported findings by Graeff et al.

Model explaining the results

5-HT3 drugs

The effects of SR57227A in HR rats were inhibited by a behaviourally silent dose (5.0 mg/kg) of the 5-HT3 antagonist ondansetron (Fig. 1f), suggesting that the SR57227A-induced changes in the duration of avoidance behaviour are mediated by 5-HT3 receptors. The fact that SR57227A resulted in an increase of avoidance (Fig. 1a/b) indicates that this agonist acts at postsynaptic 5-HT3 receptors. It has recently been shown that LR and HR rats do not differ in their 5-HT3 receptor expression (Ballaz et al. 2007a). Combining this previous finding with the present finding that higher doses of SR57227A were required to alter the duration of avoidance behaviour in HR than in LR (Fig. 1a/b) results in the notion that LR rats are characterised by postsynaptic 5-HT3 receptors that are more sensitive than those of HR rats (Fig. 6). This is supported by the finding that all doses of the 5-HT3 antagonist ondansetron decreased the duration of avoidance behaviour in LR rats (Fig. 1c), whereas none of the tested doses of this drug decreased the duration of avoidance behaviour in HR rats (Fig. 1d).

Fig. 6
figure 6

Putative serotonergic system of the brain of LR (upper row) and HR (lower row) rats. receptors that stimulate avoidance behaviour, receptors that reduce avoidance behaviour. > relatively sensitive receptors, < relatively insensitive receptors

Full size image

5-HT2 drugs

In HR rats, the effects of the 5-HT2C agonist mCPP were blocked by a behaviourally silent dose (0.3 mg/kg) of the 5-HT2A/C antagonist ritanserin (Fig. 2h). These data suggest that the mCPP-induced changes in the duration of avoidance behaviour are mediated by 5-HT2C receptors. The results of the 5-HT2A/C agonist DOI were similar to the results of mCPP, indicating that the effects of DOI were also mediated by 5-HT2C receptors. The fact that both mCPP and DOI resulted in an increase of avoidance in HR (Fig. 2b/d), but not in LR (Fig. 2a/c) indicates that the serotonergic projection regions of HR rats are marked by more or more sensitive postsynaptic 5-HT2C receptors than the serotonergic projection regions of LR rats (Fig. 6). The finding that the 5-HT2C agonists mCPP and DOI decreased the duration of avoidance behaviour in LR (Fig. 2a/c) suggest that these drugs reduced synaptic serotonin levels in these rats. This decrease of serotonergic activity cannot simply be explained by an action at presynaptic 5-HT2 autoreceptors because 5-HT2 receptors have found to be located postsynaptically only (Barnes and Sharp 1999). Recent studies, however, have revealed that the synaptic serotonergic activity is reduced after stimulation of postsynaptic 5-HT2 receptors that are located on inhibitory GABAergic interneurons in the dorsal raphe (Liu et al. 2000; Boothman et al. 2003; Serrats et al. 2005; Boothman and Sharp 2005). These GABAergic neurons are supposed to be part of a feedback loop. We, therefore, hypothesise that LR rats exhibit more or more sensitive 5-HT2C receptors that are located on this feedback loop than HR rats (Fig. 6). Ritanserin decreased the duration of avoidance behaviour in LR (Fig. 2e), but not in HR (Fig. 2f) rats. Given the lack of effects of mCPP at the postsynaptic 5-HT2C receptors of the serotonergic projection regions of LR rats (see above), these data can only be explained by an action of ritanserin in the serotonergic projection regions containing postsynaptic 5-HT2A receptors (Fig. 6). Our data indicate that LR rats are marked by more or more sensitive postsynaptic 5-HT2A receptors that belong to this feedforward loop than HR rats (Fig. 6). The fact that a behaviourally silent dose (0.1 mg/kg) of the 5-HT2A agent ritanserin inhibited the effects of the 5-HT2C agent mCPP in LR (Fig. 2g) strongly suggests that the postsynaptic 5-HT2A receptors that belong to the feedforward loop of these rats are located on the same pathway as the 5-HT2C receptors that belong to the feedback loop of these rats (Fig. 6).

5-HT1 drugs

The effects of WAY100635 during trials 1 and 2 in LR rats were inhibited by the 5-HT1A agonist 8-OH-DPAT (Fig. 3e), which in itself was not effective during these trials (Fig. 3a), suggesting that the WAY100635-induced changes in the duration of avoidance behaviour are mediated by 5-HT1A receptors. WAY100635 has been found to increase avoidance behaviour only when it acts presynaptically (Dos Santos et al. 2005; Dos-Santos et al. 2008). The finding that WAY100635 increases avoidance in LR (Fig. 3c) and in HR (Fig. 3d) indicates that inhibitory presynaptic 5-HT1A receptors are present in both types of rat (Fig. 6). The 5-HT1A agonist 8-OH-DPAT increased the duration of the avoidance behaviour in HR (Fig. 3b), but not in LR (Fig. 3a) rats. 8-OH-DPAT has been reported to result in an increase of avoidance behaviour only when its acts postsynaptically (Dos-Santos et al. 2008; Viana et al. 2008). Therefore, our data suggest the presence of more or more sensitive postsynaptic 5-HT1A receptors in HR than in LR rats (Fig. 6). The fact that a behaviourally silent dose (0.5 mg/kg) of the postsynaptic acting 5-HT1A agent 8-OH-DPAT inhibited the effects of the presynaptic acting 5-HT1A agent WAY100635 in HR (Fig. 3f) illustrates that the neurons that contain pre- and postsynaptic 5-HT1A receptors in these rats are located on the same pathway (Fig. 6). A presynaptic action of 8-OH-DPAT has repeatedly been reported to result in a decrease of avoidance (Sena et al. 2003; Dos Santos et al. 2005; Vicente et al. 2008). The finding that 8-OH-DPAT decreased the duration of avoidance in LR (Fig. 3a) can be explained by the already discussed presence of presynaptic 5-HT1A autoreceptors in these rats (Fig. 6).

Combination of 5-HT1, 5-HT2 and 5-HT3 drugs

The finding that the 5-HT2A/C antagonist ritanserin did not inhibit the WAY100635-induced increase in the duration of avoidance behaviour in either LR (Fig. 5a) or HR (Fig. 5b) rats suggests that the pathway that contains presynaptic 5-HT1A receptors does not contain postsynaptic 5-HT2A or 5-HT2C receptors (Fig. 6). The avoidance-increasing effects of WAY100635 were inhibited by the dose of 0.1 mg/kg of the 5-HT3 antagonist ondansetron in both LR (Fig. 5c) and HR (Fig. 5d) rats. This dose of ondansetron itself did not change avoidance behaviour during trials 1–2 in LR rats (Fig. 1c) and during trials 1–5 in HR rats (Fig. 1d). Our data, therefore, indicate that the presynaptic 5-HT1A receptors and the postsynaptic 5-HT3 receptors belong to the same pathway (Fig. 6).

d-Fenfluramine

The effects of d-fenfluramine appeared to be very similar to the effects of the 5-HT3 agonist SR57227A. More specifically, both drugs increased the duration of avoidance behaviour in HR and in LR rats (Figs. 1 and 4), whereas the 5-HT2C agonists (DOI and mCPP) and the 5-HT1A agonist (8-OH-DPAT) increased the duration of avoidance behaviour in HR rats, but decreased the duration of avoidance behaviour in LR rats (Figs. 2 and 3). Similar to the onset of the effects of SR57227A, the onset of the effects of d-fenfluramine was faster in LR (Fig. 1a vs Fig. 4a—trials 1–3) than in HR (Fig. 1b vs Fig. 4b—trials 3–5) rats. This striking similarity between the effects of the serotonin reuptake inhibitor/serotonin releaser d-fenfluramine and SR57227A suggests that the action of d-fenfluramine is mediated mainly via postsynaptic 5-HT3 receptors and not (or less) via postsynaptic 5-HT1A or 5-HT2A/C receptors (see also Consolo et al. 1994). This notion is supported by our finding that the avoidance-increasing effects of d-fenfluramine during trials 1–2 in LR rats and during trials 3–5 in HR rats were strongly inhibited by the dose of 0.1 mg/kg of the 5-HT3 antagonist ondansetron (Fig. 4c/d), which in itself was not effective at these trials (Fig. 1c/d).

Saline

Injection stress and exposure to a novel behavioural set-up have both been found to increase synaptic serotonin levels (Adell et al. 1997; Thomas et al. 2000). The finding that the duration of avoidance of the open arms increased more strongly in naive or saline-treated LR than in naive or saline-treated HR (Figs. 1, 2, 3 and 4) indicates that serotonin levels increase more strongly in challenged LR rats than in challenged HR rats. This behavioural finding nicely fits in with the neurochemical finding of Piazza et al. (1991b) that the brain levels of serotonin are relatively large in LR compared to HR rats. Although analysis over all trials clearly revealed individual differences in avoidance behaviour in saline-treated LR and HR rats, no individual differences were found between these two types of rat during the first trial of the elevated T-maze. This finding confirms the previously reported finding that a single exposure to the elevated plus-maze does not result in behavioural differences between LR and HR rats (Thiel et al. 1999; Ballaz et al. 2007b), whereas repeated exposure does (Ballaz et al. 2007b).

Personality traits explaining the results

Effects are not due to individual differences in anxiety

The elevated T-maze is generally used to measure anxiety related behaviour (Graeff et al. 1996a, b 1998). Rats bred for high anxiety behaviour are typically marked by a larger challenged-induced increase of corticosterone and ACTH compared to rats bred for low anxiety behaviour (Liebsch et al. 1998; Landgraf et al. 1999). The fact that challenged LR rats have been found to be marked by smaller levels of corticosterone and ACTH than challenged HR rats (Piazza et al. 1991a; Rots et al. 1995, 1996; Kabbaj et al. 2000, 2007) strongly indicates that the relatively large challenge-induced increase in the duration of avoidance behaviour in control LR rats (Figs. 1, 2, 3 and 4) is not due to a relatively large level of anxiety in these rats. Accordingly, alternative traits may have caused the serotonin-induced individual differences in the duration of avoidance behaviour in our rats.

Effects may be due to individual differences in habituation to a new environment

It has previously been hypothesised that the behavioural response during the first trial of a particular test may be of a different nature than the behavioural response during the remaining trials of this test (File et al. 1993; Ballaz et al. 2007b). In addition to SR57227A and d-fenfluramine, WAY100635 was found to result in a larger increase of avoidance behaviour during trial 1 in LR than in HR rats (Fig. 3). Remarkably, the remaining drugs (saline, ondansetron, DOI, mCPP, ritanserin and 8-OH-DPAT) did not result in individual differences in avoidance behaviour during the first trial. The fact that the presynaptic acting agent WAY100635 increased avoidance behaviour in the same trial as the postsynaptic acting agent SR57227A fits in with our above-mentioned notion that the presynaptic 5-HT1A receptors belong to the same pathway as postsynaptic 5-HT3 receptors (Fig. 6). On the basis of these considerations, it is suggested that the pathway containing postsynaptic 5-HT3 receptors (Fig. 6) is important in mediating the immediate response to a novel environment, whereas the pathways containing postsynaptic 5HT1A and 5-HT2 receptors (Fig. 6) may be more important in mediating the behavioural response after reexposure to this environment. Based on the criteria that were used to select HR and LR rats (see “Methods”), we hypothesise that the observed individual differences in the time to approach the open arms of the T-maze are due to individual differences in exploration of a novel environment (possible mediated by postsynaptic 5-HT3 receptors) and habituation to this environment (possibly mediated by postsynaptic 5-HT1A and 5-HT2 receptors).

Putative contribution of the serotonergic system to the effects of drugs of abuse

Serotonergic agents have been found to affect the neurochemical and behavioural response to drugs of abuse. These effects may well be the result of the fact that serotonergic agents can change the release of mesolimbic dopamine (Kilpatrick et al. 1996; De Deurwaerdere and Spampinato 1999; Ichikawa and Meltzer 2000). 5-HT3 antagonists are suggested to be effective in the treatment of alcohol abuse (Greenshaw 1993; Grant 1995), whereas 5-HT2A antagonists (McMahon and Cunningham 2001; Davidson et al. 2002; Filip et al. 2004; Bubar and Cunningham 2006) and 5-HT2C agonists (Fletcher et al. 2004; Filip et al. 2006; Bubar and Cunningham 2006; Neisewander and Acosta 2007) can reduce the behavioural effects of cocaine. Finally, systemic administration of 5-HT1A antagonists and agonists are known to, respectively, increase and reduce the accumbal dopamine response to cocaine and amphetamine (Ichikawa et al. 1995; Andrews et al. 2005). In case serotonergic agents can be used to treat drug abuse, personalised therapy is preferred. On the basis of our finding that rats being selected on a low and high response to novelty differ in their sensitivity to serotonergic agents, it is hypothesised that the optimal serotonergic therapy may differ between substance abusers with a low and a high sensation-seeking score.

Conclusions

HR and LR rats have been found to differ in the expression of 5-HT6 and 5-HT7 receptors (Ballaz et al. 2007a, c) as well as in the levels of serotonin in the brain (Piazza et al. 1991b). The present study provides the original finding that HR and LR rats also significantly differ in their sensitivity to serotonergic drugs that act at the level of 5-HT3, 5-HT2 and 5-HT1 receptors (see Fig. 6). Future studies using HR and LR rats are required to investigate in which brain structures these receptors are located. Knowledge about the serotonergic system of high and low responders to novelty rats may well contribute to our knowledge about the serotonergic system of high and low sensation seekers in humans (Dellu et al. 1996; Ballaz et al. 2007a, c). On the basis of our animal studies, it is hypothesised that the level of sensation seeking in humans can predict which subjects are vulnerable to a specific serotonergic therapy and which subjects are not.