Timing the Brain to Time the Mind: Critical Contributions of Time-Resolved Neuroimaging for Temporal Cognition
Time is a loaded concept, which cognitive neuroscientists have to consider from two major viewpoints simultaneously: a physicalist viewpoint consisting in providing refined descriptions and characterizations of the complex dynamical system, that is, the brain, and a psychological viewpoint consisting in understanding how different temporal phenomenologies (perceiving duration, ordering events in time, thinking about the past or the future, etc.) relate and map onto the described brain dynamics. In this chapter, we wish to emphasize the major conceptual differences between timing, seen as the inherent property of all neural processes dedicated to perception, action, and cognition, and time perception, or more generally temporal cognition, which specifically targets how the brain represents the temporal structure of events and of our environment, implicitly or explicitly. If techniques such as electroencephalography (EEG) and magnetoencephalography (MEG) have been systematically used for timing, there is a surprising paucity of studies specifically focusing on temporal cognition. Nevertheless, the field is in full bloom, providing an adequate momentum for researchers to embrace MEG and EEG as techniques of choice to address their research questions.
KeywordsOscillations Time Timing Cognition Duration Order Simultaneity Expectation Prediction Interval timing Temporal hazard MMN MMF CNV CMV EEG MEG Delta Theta Alpha Beta Gamma
A surge of interest in time research has sprang within and beyond the field of neurosciences. How the brain deals with the temporal dimension of our world is far from being understood, and how humans can feel, quantify, attend to, expect, reason about, or conceptualize time remain fascinating topics of investigation. Yet, and crucially, “[…] the field of neuroscience will have to further mature and embrace the fact that it will not be possible to understand the human mind without describing how the brain tells, represents, and conceptualizes time. […]” (Buonomano 2017). Indeed, the temporal structure of the environment makes time an indissociable property of all sensory inputs, just as time is inherent to the cognitive architecture.
We will primarily focus on human temporal cognition (with few references to the animal literature, but by no means an exhaustive review of it) for which noninvasive techniques are of prime importance and the sole means to quantitatively assess healthy brain activity, while participants perform controlled psychological tasks. The relevance of using magnetoencephalography (MEG) and electroencephalography (EEG) to assess temporal cognition in humans is perhaps best motivated by early opinions on the topic, which further highlight the importance of understanding the endogenous aspect of brain activity for mental activity: “At a neurophysiological level we should expect to find evidence of clock systems, especially if is accepted (a) that the cerebral cortex behaves in a calculator-like way and that “its function is to select, differentiate, condense and abstract rhythms or patterns of neuronal activity” (Gooddy and Reinhold 1954); and (b) that perception depends on the spatiotemporal arrangement of nervous activity. We should be able to deduce, further, that the characteristic features of the cortical neurophysiological clock should be a simplified rhythm, abstracted from the multitudes of nerve-cells, processes, channels, and impulses. We see, in the electroencephalograph, the exact illustration of this deduction […]” (Gooddy 1958). MEG and EEG provide exquisite time-resolved noninvasive recordings of human brain activity suited to address timing (Hari et al. 2010; Hari and Parkkonen 2015). Historically, the galvanometric recordings of gray matter revealed the presence of electrical brain activity (Caton 1875) soon followed by noninvasive EEG recordings by Berger in 1924. These techniques fed the search for endogenous activity selective to afferent inputs, i.e., activity which does not passively react to stimulation (Jasper 1937). Hence, the use of EEG was motivated by the search for central, endogenous, and spontaneous activity that goes beyond mere reflexive behavior, and this was a quite remarkable observation put into the historical context of neurosciences of cognition.
The millisecond resolution of MEG and EEG allows researchers to intentionally adapt the time scale at which processes of interest will be quantified (i.e., down-sampling and filtering being permitted!). These techniques also provide descriptors of local (population scale) and large-scale (network) activity. These neuroanatomical scales are presumably most relevant for sensorimotor timing, temporal cognition, and, more generally, coordinated adaptive behaviors within and across individuals. Methodologically, time-resolved neuroimaging is increasingly complemented by sophisticated methodological tools, which increase the flexibility of experimental designs and data quantification (from single-trial analyses to brain decoding techniques and encoding model-informed analysis) as will be illustrated with recent empirical work throughout the chapter. Simultaneous recordings of MEG and EEG further strengthen the quality of source reconstruction estimates by including complementary neural contributors when combined with anatomical MRI. All in all, MEG and EEG are methods of choice for the study of temporal scales that are relevant to human perception and cognition.
2 A Brief History of Psychological Time
2.1 General Taxonomies of Psychological Time Scales
A complementary and important approach to temporal cognition is to operationalize time not as a function of external clocking phenomena (aka the arbitrary experimenter’s real-time objective clock), but rather as a function of the neurophysiological, psychological, and behavioral attributes which we wish to explain. Such approach captures aspects of temporal cognition from at least two distinct perspectives: implicit and explicit timing (e.g., Coull and Nobre 2008). Hereby, implicit timing refers to real-world or experimental scenarios in which an individual is not aware of keeping track of their timing but does extract temporal regularities from the environment to perform a sensory or motor task efficiently, for example, when the detection of a weak sensory signal is aided by a temporal regularity in the signal. Explicit timing, on the other hand, refers to situations in which the observer actively engages in timing to produce an overt estimate of time (for instance, a duration estimate or tapping along with a rhythm).
General overview of existing timing models. This list is not intended to be exhaustive nor is it entirely dedicated to the processing of duration – although highly dominated by it. Several factors can be considered in attempting a taxonomy of time models. Here, we used as descriptors whether the implementation would be seen as dedicated to temporal processing or intrinsic to neural processes, as well as how neuroanatomically centralized or distributed temporal processes may be. The right column provides an intuitive classification essentially based on Marr’s levels of analysis (van Wassenhove 2009)
Level of description
Distributed and intrinsic
Centralized and dedicated
Centralized and dedicated
Centralized and dedicated
Centralized and dedicated
Bayesian models (Jazayeri and Shadlen 2010; Petzschner et al. 2015) capture typical characteristics of time and magnitude estimation. Possibly implicating parietal regions (e.g., Bueti and Walsh 2009)
Chronotopy in which the temporal tuning of sensory neurons intrinsically provides temporal information of sensory inputs (Holcombe 2013)
Centralized or intrinsic
Event-based computations in which durations are the outcome of distance computations between events stored in memory (Gallistel 1990)
Computational, predictions on implementation
Distributed and intrinsic
Implementation, computational predictions
2.2 Models of Time Perceptions
The perception of duration is a canonical aspect of subjective time awareness, one that is highly personal and labile, yet remarkably not entirely off with respect to clock time when tested in controlled environments. Numerous studies have apprehended, as a first approach, duration perception as the linear accumulation of “time units” in real time that would be predicted by the psychological internal clock models (Treisman 1963, 2013; van Wassenhove 2016). There exist four flavors of timing models to account for duration perception (Table 1): two classes distinguish how dedicated to temporal processing or intrinsic to neural processes (Ivry and Schlerf 2008) and how neuroanatomically centralized or distributed (Ivry and Spencer 2004; Ivry 1996) timing may be.
Timekeeping mechanisms also come with a number of first- and second-order principles (Allman et al. 2014): the first-order principles apply to durations and typically involve the accuracy and precision with which a duration is being timed. In other words, how fast the subjective clock is ticking, and how the duration is subjectively stored compared to the objective duration. The second-order principles are used to compare multiple durations with each other with respect to the scalar property in which the smallest detectable difference between two durations will scale linearly with the mean duration. Most EEG and MEG studies of time perception have by far focused on the first-order properties of timing mechanisms.
Although psychological internal clock models were not conceived to be literally implemented as such in the brain, the observation of endogenous oscillatory and rhythmic activity rapidly leads to the hypothesis that oscillations may actuate the pacemaker of the internal clock. As we will see, this direct implementation has not been successful from the point of view of a centralized internal clock. Despite clear evidence for the implication of oscillatory activity in many aspects of timing, it is unlikely that a single neural oscillation (or “beat frequency”) would be the master clock. Positing several oscillators would be needed, but the working hypothesis of oscillators as pacemakers has not received substantial and robust empirical evidence over a century worth of testing as will be reviewed in Sect. 3. A direct implementation of neural oscillations as pacemakers for time perception has also been deemed computationally intractable (Miall 1989). Alternatively, the internal clock may operate on the logic of coincidence detectors, which would read out the phase synchronization across multiple time scales (Gallistel 1990), or the activity level of oscillators set into activity by a timing task (Gu et al. 2015). This animal model is under active investigation.
A third viewpoint that has surprisingly attracted much less attention is the consideration of an internal clock that would serve computations of temporal distances on the basis of event counting and in the absence of duration storage (Gallistel 1990). The difficulty, as readily acknowledged by the author, is that the predictions of an event-based model for duration computations are virtually identical as for the internal clock that would be based on a duration storage system. The endogenous structuring of events in time lends itself to the notion of duration as the outcome of a computation process as opposed to a retrieval process. Alternative models to the internal clocks take a different perspective on time and heavily incorporate neural oscillations as a fundamental architecture for timing. For instance, Miall (1989) and Church and Broadbent (1990) proposed multiple oscillator models in which timing is implemented by neuronal ensembles oscillating at different frequencies. This seminal approach has been formalized in the Dynamic Attending in Time model (DAT: Jones 1976; Large and Jones 1999). Contrary to the internal clock model, which relies on the storage of durations, the DAT posits a hierarchy of time scales, which automatically drives the structuration of events in time. This formalization presupposes rhythmic structures as the backbone of temporal phenomenology, yielding an inherent structuring of time to any model incorporating neural oscillations as a functional mechanism. This approach to timing is radically different in that the temporal structure can be seen as a temporal map onto which sensory and mental events are tagged. This view also provides endogenous time metrics, which have been used successfully for human production ranging from musical rhythms to speech.
2.3 Psychological Paradigms
We provide below a summary of classical experimental paradigms used in the timing and time perception literature. These will be discussed throughout the text, and this table may serve as a quick reminder of what each task implies (Fig. 4).
3 An Endogenous Representation of Subjective Duration
Surprisingly very few studies have directly tackled duration estimation with MEG and/or EEG, perhaps due to the fallacious intuition that time in the brain linearly maps to time in the mind (Dennett and Kinsbourne 1992). It is widely acknowledged that the quantification of a time interval in the brain is complicated by the absence of time receptors and by a coding scheme that is not isomorphic to the arbitrary time metrics of external clocks (Gallistel 1990). Following the prediction of the internal clock model, we will first discuss the implications of oscillations for duration perception and then turn to additional seminal reports focused on several evoked responses.
3.1 Alpha Oscillations: Subjective Moments and Ticking Internal Clocks
Alpha oscillations (α; ∼8–12 Hz) are the dominant spontaneous brain rhythms seminally reported by Berger (1929) and explored by Adrian (1944). Jasper observed early on that α was sensitive to temporal conditioning (Fig. 5a). The peak of α oscillations can be detected with closed eyes, and the variability in α peak frequency across individuals is well-known (Werboff 1962; Surwillo 1966; Klimesch 1999; Haegens et al. 2014), but intraindividual variability is more controversial (Barlow and Brazier 1957). Seminal reports showed that α oscillations were indicative of conscious states in humans (Berger 1929), and Gooddy (1958) suggested that cortical rhythms were relevant for timing and clocking mental activities. The notion that α may be relevant for general information processing as a clocking process also emerged with computational theories of information processing and cybernetics during the cognitive revolution (Wiener 1961). Several researchers (Ellingson 1956; White 1963) suggested that α, as the dominant spontaneous oscillatory activity, may contribute to defining psychological moments with the hypothesis that one α cycle (∼100 ms) would form the unit of subjective time, i.e., the psychological moment.
This started a series of experimental work trying to link various parameters of α oscillations and temporal estimation. First, Werboff (1962) showed that the proportion of time during which α (8–13 Hz) was present during open-eyed resting state was relevant to time estimation: participants in which α was present less than 50% of the time underestimated a 2 s and an 8 s time interval as compared to the group in which α was present more than 50% of the time (Fig. 5b). In two experiments, Legg (1968) tested the implication of α in temporal perception by considering separately participants with clear, some, or no α: he then established an α index which he could link with the individuals’ time estimates in different time scales. Legg (1968) partially replicated the findings of Werboff but highlighted the very weak supporting evidence linking α to time estimations (Fig. 5b). In the same line of inquiry, Surwillo (1966) reported a lack of substantial evidence linking interindividual variability of α and time estimations. He then assessed, in a single-case study, intraindividual fluctuations of the α period, which he found to be very weakly associated with a prospective estimation of 10 s duration (Fig. 5d). Using a different approach, Anliker (1963) explored the link between α and the lengthening of inter-taps interval during a time production task of 3 s. He reported an increase of alpha amplitude with the lengthening of inter-tap intervals (Fig. 5e) although his results were admittedly confounded by numerous factors, such as arousal, drowsiness, and lapses of attention, all contributing to a major time-on-task effect over the 3–4 h EEG acquisition time. Cahoon (1969) reported a positive relationship between endogenous (but not induced) arousal, α oscillations, and subjective timing for verbal estimation and motor tapping but not for temporal productions; he concluded in favor of Hoagland’s chemical clock hypothesis.
In a more modern approach, Nelson et al. (1963) considered that given the same number of neural units, the amplitude of the response would become a function of the amount of synchronization among the units. They tested the effect of the rate of visual stimulus on the behavioral discrimination of duration and found that the maximal effect was for a rate of 10 Hz, thus close to α oscillations. Holubar also tested if flicker frequency decreased response intervals and concluded that α was the temporal pacemaker (1969, as reported by Treisman (1963)). In his PhD thesis, Ross (1968) followed up on this notion and tested using EEG the hypothesis that increased neural synchronization would increase subjective duration. Ross tested participants on a temporal production task of 10 s in four conditions, in the dark, in steady light, with sound, or with visual flickers, selected on the basis of an individual’s alpha peak frequency. Results showed (with clear limitation in the analysis and statistical techniques) that the amount of synchronization was involved in the perception of duration so that the more synchrony, the longer the perception of duration (Fig. 5f). Very recently, an EEG study confirmed these early observations by showing that an increased amplitude of entrained α oscillations (using visual flicker to elicit a visual steady state response) was paired with longer temporal reproductions (Hashimoto and Yotsumoto 2018).
In the internal clock proposed by Treisman (1963), one α cycle was assumed to represent one tick. Following this reasoning, the peak of the α rhythm should be linked to the subjective speed at which time is felt. The higher the individual’s α peak frequency, the smaller its period and the more ticks would be accumulated in the same amount of time, thus predicting a lengthening of subjective time; conversely, the lower the α peak (the larger the period), the less ticks accumulated in the internal clock, thereby predicting a shortening of subjective time. Surwillo’s experimental data failed to provide substantial evidence favoring this hypothesis (Surwillo 1966; Fig. 5c). Much later, Treisman also used EEG to test the hypothesis of α oscillations as a common pacemaker for the internal clock: through a series of systematic experimental manipulations, including drawing a direct link between the α peak frequency and time estimations (Fig. 5d), Treisman reached the firm conclusion that a common pacemaker could not underlie duration estimation. In a more recent time production study, Glicksohn et al. (2009) reported an intriguing hypothesis of mutual hemispheric suppression in an individual’s α peak frequency; to the best of our knowledge, this hypothesis has not received more support or further replications. Finally, other researchers had suggested the possibility of a phase relationship between α and temporal discrimination (Holubar 1960; cited by Nelson et al. 1963). In a recent EEG study, the α phase synchrony between auditory and visual responses was suggested to support the integration audiovisual durations (van Driel et al. 2014) although it is unclear how these results could fit with models of duration perception.
As of today, a direct and robust link between the perception of duration and alpha oscillations remains more speculative than empirical (Kononowicz and van Wassenhove 2016). Perhaps a major issue lies in the formalization of the possible implementations of a pacemaker for the internal clock (Treisman 1984; van Wassenhove 2016), which has so far been taken quite literally. If the implication of α oscillations in the temporal structuring of information processing is well established, it has not yet provided robust mechanistic insights on the explicit and conscious aspects of duration perception.
3.2 Is Beta (β, ∼14–30 Hz) Activity the New Tick in the Clock?
Beta oscillations traditionally associated with motor functions (discussed below) have started to emerge as a plausible candidate for a timekeeping mechanism. Several studies investigating β oscillations over a range of timing tasks and biological systems have started to draw a converging line of evidence. The contribution of β power to timing behavior has been demonstrated in monkeys trained to perform a synchronization-continuation task with short time intervals (Bartolo and Merchant 2015; Bartolo et al. 2014; also see Fujioka et al. 2012, 2015). First, the monkeys synchronized to the external rhythm for a couple of cycles, and then, they were required to maintain the rhythm in the absence of external sensory entrainment, i.e., to pursue the rhythm with continuation. The authors recorded local field potentials (LFPs) from the putamen while the monkeys were performing this task and demonstrated that β power was systematically present in the continuation, but also in the synchronization phase, of the paradigm. Moreover, the power of β oscillations was larger for longer durations. These results suggested that the power of β oscillations reflected the to-be-produced duration (Bartolo et al. 2014; Bartolo and Merchant 2015). As such, β power may support the timekeeping mechanisms or the guidance of internally driven motor sequences.
Given the physiological considerations that dynamic processes do not necessarily reflect evidence accumulation, an alternative hypothesis suggested it may signify a release from inhibition triggered at the interval onset (Kononowicz et al. Imprint). Consistent with this, when participants were asked to compare sub-second durations, their judgments were influenced by the pre-stimulus phase in the beta range (Milton and Pleydell-Pearce 2017), highlighting the relevance of the initial states of the system during temporal processing. In line with that conclusion, Wiener et al. (2018) used drift-diffusion modeling to show that β oscillations exclusively shift the starting point of the decision process in temporal bisection, an effect that may be tied to a change in the first-stage timekeeping process. Still, these results are difficult to reconcile with EEG and MEG patterns seen in time reproduction and production tasks (Kononowicz and Van Rijn 2015; Kononowicz et al. Imprint), as motor task requirements differ in time production and temporal bisection task.
In another perceptual timing task using MEG, Kulashekhar et al. (2016) asked participants to memorize the duration or the color of dynamically displayed stimuli and to compare them with a comparison interval. The authors found that duration judgments were associated with stronger β oscillations (14–30 Hz) as compared to color judgments, in both the standard and the comparison intervals (Fig. 6b). The increased β power was present in a number of frontoparietal regions, indicating that β oscillations may be specific to some aspects of timekeeping processes as opposed to other oscillatory components. The specialization of β oscillations for temporal processing recently gained further support with a study using transcranial alternating current stimulation (tACS) over fronto-central cortices, stimulating at α and β frequencies (Wiener et al. 2018). This study showed that β stimulation exclusively shifted the perception of time such that an increase in β yielded a reported lengthening of subjective duration; this was not seen for α stimulation (Fig.6b).
In summary, monkey neurophysiology, human MEG, EEG, and tACS studies converge toward the hypothesis that β oscillations play a special role in time perception. Future and upcoming studies will need to explore the links between β oscillations measured during perceptual timing and those reported in during motor timing. Traditionally, β oscillations have been associated with motor functions (Kilavik et al. 2013) and motor preparation: for instance, modulations of β power have been reported in a delay period ahead of motor execution (Praamstra et al. 2006). Action-related β oscillations might reflect sensorimotor updating and planning processes (Donner et al. 2009; Kilavik et al. 2013). As recent studies in motor control have questioned the popular view that β oscillations may coordinate higher-level variables in motor execution tasks (Tan et al. 2016; Tzagarakis et al. 2010), the results observed during perceptual timing suggest that β oscillations may reflect an important aspect of timing inherent to all motor actions. The implications of β oscillations in motor timing will be specifically extended in Sect. 5.
3.3 Entrainment and Duration
Neural entrainment , that is, neuronal rhythms aligning to external rhythmic inputs, can have both beneficial and detrimental consequences on temporal phenomenology from boosting temporal expectation to biasing duration and order perception. In the framework of neural oscillations representing an internal metric of time, entraining oscillations to different frequencies could affect the speed of the clock and hence perceived duration (Kanai et al. 2006; Herbst et al. 2013, 2015; Hashimoto and Yotsumoto 2018). While entraining stimuli like visual flicker clearly alter perceived duration (Kanai et al. 2006; Herbst et al. 2013), there is no evidence that the amplitude or the frequency of entrained oscillations is directly related to an acceleration or deceleration of an internal metric of time over a broad range of frequencies (Herbst et al. 2015). Nevertheless, a recent study by Hashimoto and Yotsumoto 2018 has shown that the amplitude of an entrained oscillation at 10 Hz correlates with perceived duration, which argues more for a specific role of single-frequency bands.
3.4 Evoked Responses and Time Perception
A primordial assumption when considering evoked responses as markers of time perception is that neural events unfold serially with respect to the sequence of external information being presented. As the time arrow is a mathematical simplification – or a particular case – of physical time, it would be important to elaborate further which theoretical stance a researcher in time perception takes when considering evoked responses as markers of time perception. This is outside the scope of this chapter, but an important consideration to bear in mind. We will briefly mention three main seminal examples of evoked responses taken as markers of time perception: sensory evoked responses (in line with a chronotopic view of timing; Table 1), contingent negative variation (CNV; in line with a centralized clock), and mismatch negativity/field (MMN/F; in line with predictive coding). We would also like to direct the reader to a recent review of EEG responses related to timing (Ng and Penney 2014), which covers in more depth each of the evoked responses discussed below.
3.4.1 Sensory Evoked Responses: Onset/Offset Latency Code
Taking an information-theoretic approach to time estimation has motivated a great majority of studies on the perceptual estimation of time, in which a participant estimates or discriminates the duration of sensory events (Fig. 4). When estimating the duration of a sensory interval, the precise encoding of the onset and of the offset sensory stimuli is a priori needed to derive a reliable internal representation of duration (Schlauch et al. 2001). In a seminal EEG study (Picton et al. 1978; Fig. 7a), an increased auditory sustained response with a marked offset response was seen when participants judged the duration, but not the intensity of a sound. Auditory sustained responses were hypothesized to reflect the subjective uncertainty about the timing of the offset of a (filled) stimulus, possibly reflecting a contingent negative variation or CNV (Järvilehto and Fruhstorfer 1973 cited in Picton et al. 1978), a seminally reported electrophysiological marker in time estimation. While auditory offset responses have previously been reported (Hari et al. 1987; Gutschalk et al. 2002), their functional implications for timing and time perception have not been fully explored, neither in human neuroimaging literature nor in animal studies (Kononowicz et al. Imprint).
In line with the known interaction between duration and attention (Block and Zakay 1997), the amplitude of the auditory sustained potential thus increased when participants were asked to detect a long duration sound as compared to detecting its intensity or its warbling (Picton et al. 1977) and the authors noted that paying attention to duration delayed the latency of the N2-P3. Fraisse (1988) observed that temporal errors in duration reproduction decreased for shorter intervals (in agreement with Vierordt’s law) and that the amplitude of the N1-P2 auditory evoked responses decreased with shorter intervals at the encoding stage. Although no significant correlations were found between the temporal reproduction and the amplitude of the sensory evoked responses, Fraisse speculated that the decrease in amplitude may correspond to a subjective error in the perception of duration. More recently, Bendixen et al. (2005) reported the existence of a sustained auditory response whose amplitude, given the same physical duration, varied as a function of perception being shorter or longer (Fig. 7b). The authors interpreted the sustained responses as being compatible with the notion of a CNV and the implication of attentional resources in the estimation of duration.
Interestingly, while the existence of cortical offset responses, a priori necessary for the perception of duration (Schlauch et al. 2001), has been shown in audition (Gutschalk et al. 2002; Hari et al. 1987; Takahashi et al. 2004) and in somatosensation (Yamashiro et al. 2011), they have not been systematically studied in the context of timing. Yet, the sources of slow sustained fields have different refractory properties and physiologically distinct generators (Pantev et al. 1994) suggesting distinct mechanisms contributing to the early sensory encoding stages of duration estimation.
3.4.2 Mismatch Negativity and Field (MMN/F) : Illusory Duration, Coding Efficiency, and Violation of Temporal Prediction
Subjective temporal dilation could result either from the temporal compression of predictable events (the standard stimuli in a sequence), or from an increased neural response elicited by surprise (the oddball event in a sequence of standard events), or from a combination of both mechanisms (Pariyadath and Eagleman 2012; Eagleman and Pariyadath 2009). This behavioral observation is well-suited for MEG/EEG investigations. The mismatch negativity and its magnetic equivalent the mismatch field (MMN and MMF, respectively) are elicited by the presentation of a surprising event in the context of repeated stimulation of the same event or mental category. Repeated presentation, by virtue of adaptation, yields the neural suppression of activity from a given neural population. MMN/F is classically considered an index of automatic change detection (Näätänen 1995), and the recent Bayesian “explaining away” (Gotts et al. 2012) suggests that repetition of the same information may refine an internal template subsequently used to generate an hypothesis as to the impending sensory evidence. The MMN/F is thus considered an index of predictive coding (Friston 2005; Kiebel et al. 2008; Garrido et al. 2009).
Whether duration is, itself, a property that can be predicted by a generative internal model is not fully clear. Early observations of duration mismatch made with EEG reported that the deviant tones longer than the standard tones elicited larger MMN amplitudes than those that were shorter (Catts et al. 1995; Jaramillo et al. 1999; Näätänen et al. 1989; Näätänen 1992; Joutsiniemi et al. 1998). One study (Colin et al. 2009) also reported larger MMN in response to shorter deviants. Another study (Jaramillo et al. 2000) suggested that the magnitude of the deviance, irrespective of its direction (shorter or longer), drove the amplitude of the mismatch response. Kononowicz and Van Rijn (2014) also investigated responses evoked by supra-second durations shorter and longer than a standard interval: the amplitudes of the evoked responses were a function of the deviance from the standard, with the N1/P2 amplitude forming a V-shaped pattern tracking the temporal distance of the target to the standard interval (cf. also Tarantino et al. 2010). This symmetric pattern of post-interval components indicates that the brain remains sensitive to temporal duration even after the standard interval has passed. This observation is consistent with other studies (i.e., Mento et al. 2013, 2015; van Wassenhove and Lecoutre 2015; Fig.7g).
This interpretation was consistent with the observation that the amplitude of the MMN elicited by a duration deviant is predictive of discriminability and duration estimation (Loveless 1986; Amenedo and Escera 2000; van Wassenhove and Lecoutre 2015). Hence, the elicitation of genuine MMN/F by duration oddballs (Loveless 1986; Kaukoranta et al. 1989; Joutsiniemi et al. 1998; Tervaniemi et al. 1999; Näätänen et al. 1989; Amenedo and Escera 2000; Akatsuka et al. 2005; Colin et al. 2009; van Wassenhove and Lecoutre 2015; Recasens and Uhlhaas 2017) suggests that the timing of information can be automatically encoded and thus that an internal template for duration is possible. When deviants are longer than standards, the distinction between two durations can only occur after the shorter event has terminated (i.e., the offset of the standard): hence, the latency of the MMN relative to the onset of the deviant was predicted to be a function of the representation of the standard duration (Näätänen et al. 2004). This was recently reported (van Wassenhove and Lecoutre 2015) with the report of a possible compression of duration information consistent with previous work (van Wassenhove and Lecoutre 2015; Yabe et al. 2005). It is also important to highlight that MMN/F have often been obtained during the presentation of duration oddballs typically generated by stimuli that were also physically different (Jacobsen and Schröger 2003): as such, the MMN/F can provide erroneous estimations of duration deviance if standard and oddball stimuli undergo different neural analyses.
3.4.3 Contingent Negative Variation (CNV) and Ramping Activity: Accumulation Toward Threshold?
Some of the motivations for the early studies on the electrophysiology of time perception focused on the warning effect (Treisman 1963; Lages and Treisman 2010; Walter et al. 1964; Gaillard and Näätänen 1973): when the delay between a warning signal (nowadays designated as a “cue”) and an imperative stimulus (nowadays designated as a “target”) is constant, a foreperiod effect is observed so that participants become faster at detecting the target (Woodrow 1914; Iemi and Näätänen 1981). In an early EEG study assessing this effect (Walter et al. 1964), a slow negative potential was observed to evolve during the foreperiod. The slow waveform associated with the impending stimulus was observed within (i.e., both cue and target were auditory or visual) and across sensory modalities (i.e., the cue was presented in a different sensory modality than the target). This waveform was called the contingent negative variation (CNV) because it occurred between two contingent events, possibly reflecting the expectancy, preparation, or temporal orientation elicited by the first event toward the future second event. Similar slow potentials had been described for motor preparation with MEG (Deecke et al. 1982) and were considered comparable to the “terminating” portion of the sensory CNV.
The first report of a magnetic CNV was observed when participants terminated a 100 ms target tone systematically presented to them 1 s after a standard 100 ms tone (Weinberg et al. 1984; Fig. 7c–f). The magnetic counterpart of the CNV was compared with EEG recordings and described in a series of studies (Elbert et al. 1994; Rockstroh et al. 1993). In a seminal MEG study (Fig. 4a), participants underwent a go/no-go task in which they were asked to press a button at the end of an auditory tone and refrain from pressing a button at the end of a different tone. Tones lasted 4 s, and their full duration was considered the “warning stimulus.” The authors reported the existence of a CMV likely generated by multiple sources encompassing temporal and frontal cortices. Consistent with the possible mixture of expectancy and preparation in the CNV, motor action (go) yielded larger CMV amplitudes than no action (no-go) (Elbert et al. 1994).
Much earlier, Macar (1977) had synthetically proposed that the CNV was an electrophysiological index of time estimation, following a thorough review of the then existing evidence. Since then, she dedicated a major part of her research in describing parametric changes of the CNV responses during various time paradigms. For instance, the amplitude of the CNV was considered a marker of neural accumulation that correlated with subjective duration (Macar et al. 1999) although replications have failed to comfort this interpretation (Kononowicz and van Rijn 2011; Ng et al. 2011; Tamm et al. 2014; but see Herbst et al. 2014). Instead, Kononowicz and van Rijn (2011) observed that the CNV amplitude was influenced by time-on-task, as the CNV amplitude decreased from the beginning to the end of the experimental session (see also Mento et al. 2018). Another prediction linking CNV and the accumulation process is that the CNV should continue to increase with the passing of time. However, studies have reported CNV profiles that contained longer plateau-like amplitude patterns (Ng et al. 2011; Kononowicz et al. Imprint). The time-on-task effect is thus in contrast with the assumption that the CNV reflects a stable accumulation process and is therefore not predicted by, but rather evidenced against, the proposal that the CNV reflects temporal accumulation. A recent EEG study showed the existence of a repetition enhancement of the amplitude of the CNV, which was interpreted as reflecting the updating of durations in memory or an indexing of the similarity between successive durations which were presented (Wiener and Thompson 2015). The CNV’s latency and resolution were also proposed to reflect the memory of target duration, but these results have also been disputed (Ng and Penney 2014). Participants recorded simultaneously with EEG and MEG, while performing a visual and auditory duration discrimination showed sustained activities in MEG and CNV in EEG (N’Diaye et al. 2004). The authors discussed the difficulty to infer common source generators to the two signals, suggesting that the sustained sensory responses were likely dominating in MEG although they did observe additional sources in parieto-frontal regions (N’Diaye et al. 2004). Similarly, a recent study pointed out to the lack of direct mapping between CNV and CMV using a temporal reproduction task (Kononowicz et al. 2015) with the suggestion that the CMV may track important decision processes related to timing. Both the CNV and the CMV decreased as a function of the reproduced interval so that temporal overestimation during reproduction was paired with lower amplitudes of the CNV or CMV. These observations were taken as evidence that CNV and CMV may reflect the implications of temporal expectancy and decision-making during timing. It is likely that the current mixed and contradictory CMV/CNV results in the timing literature arose due to various processes contributing to the slow ramping activity. Thus, the future challenge lies in an appropriate unmixing of these signals and a refined attribution of specific perceptual and cognitive processes to this activity pattern (Kononowicz and Penney 2016), something to which MEG could significantly contribute.
4 Event Timing
Order and simultaneity are two intricate sides of a perceptual conundrum, namely: how does the brain integrate or segregate sensory inputs as a function of whether they represent (or not) the same event in the world? The question of serial order is fundamental to system neurosciences (Lashley 1951) considering that one’s theoretical conception of serial order dissociates two information-theoretic views of brain processing: one view in which sensory events are serially dealt with, in the order with which they occurred in the external world, and the other, in which endogenous mechanisms represent information according to internal rules and causal relations between events, which will alter the original sequence in which sensory events occurred. These two conceptions result in very different considerations of what timing in brain recordings may mean, including the functional interpretations of latency, serial processing, sequence, phase relations, and causality, among others. Additionally, it influences how we comprehend the relations between the internal dynamics of the brain system and the temporal statistics of external events.
Although the existence of temporal moments was rejected by William James, it was evoked by Bergson (1889) and later studied in Gestalt psychology. As later on discussed by Stroud (1956), a moment in psychology can be seen as the amount of time within which temporal changes may not be discerned. Beyond the epistemological importance of moments, the notion that information may be integrated within a particular temporal moment or window of time has computational consequences for perception and cognition at large. A precise definition in neural systems (Theunissen and Miller 1995) is that encoding windows represent the minimal amount of time needed by the nervous system to categorize information reliably. In line with the proposal that neural oscillations structure information in time (Pöppel 1972, 1997), the range of naturally occurring brain rhythms (Buzsáki et al. 2013) may naturally provide different moments or integration windows, which can parse information into representational units for perception.
Event timing thus refers to a conception in which time is represented in the brain by including the temporal statistics of events (e.g., the duration, frequency, rate, etc.) and the temporal relations between events (e.g., order, distance, rhythm, etc.). These representations bear some veridicality to the common external reality that all of us experience, considering that “Within any world with which we can communicate, the direction of time is uniform” (Wiener 1961, p. 35). Yet, assessing the temporal sensitivity of participants to different temporal relations between sensory events is one way to characterize the resolution with which mental events can be automatically coded, structurally constrained, and consciously accessed.
4.1 Event Individuation, Simultaneity, and Integration of Information
Event individuation, more readily used in visual neurosciences, and parsing, more readily used in auditory neurosciences, correspond to the capacity to individuate or discretize sensory events from a stream of continuous information. Speech parsing is perhaps one of the best known illustrations of this problem in Humans. If we tackle the problem from a purely bottom-up viewpoint, we can simplify it by asking what the minimal temporal (or spatial) distance is so that one can perceptually separate two events in the sensory world. If we tackle the problem more realistically by incorporating the capacity of endogenous attentional orienting and integration constants in the brain to affect individuation and parsing, the question being asked is what the minimal requirements for the representation of an event in the brain are. This question is central to the notion of the nature of representations in cognitive neurosciences and at the heart of the questioning on the granularity of discrete representations and their neurobiological implementation (e.g., VanRullen and Koch 2003; Fingelkurts and Fingelkurts 2006; Poeppel et al. 2008). Brain oscillations (Wang 2010) provide a natural hierarchical structure in time, which may enable the parsing of information both at local and global spatiotemporal scales (e.g., Buzsáki 2006), thereby implementing implicitly and pre-semantically a temporal architecture for perception and cognition (Pöppel 1972, 1997, 2009). In this context, the frequency, the power, and the phase of neural oscillations are three major properties that may bear functional importance for the understanding of the temporal properties of internal representations.
Let’s consider a well-studied example linking spontaneous alpha oscillatory activity and visual perception. Individuation is a process by which an approximate representation of a visual object would be elicited prior to its identification: the parsing of information in a complex visual scene is presumed to be constrained both by the spatial and by the temporal resolution of the system (Wutz and Melcher 2014). α oscillations are a marker of functional inhibition: an increase of alpha power has classically been interpreted as an increased inhibition of network activity with a major hypothesis suggesting that sensory processing is gated with a periodicity of 100 ms (Jensen and Mazaheri 2010). In other words, the fluctuations of α activity implement a periodic temporal selectivity of information processing in the brain (Klimesch 2012) through the modulation of cortical excitability at particular points in time. As such, the phase of spontaneous α oscillations informs, and predicts, the perceptual detectability of sensory events (Busch et al. 2009; Mattewson et al. 2009). In a seminal EEG study (Varela et al. 1981; Gho and Varela 1988), the authors suggested that the phase of the occipital α at which a pair of visual events would be presented would determine whether participants would see two events (i.e., individuate events) or only one (i.e., integrate events due to their being perceived as simultaneous). Indeed, the authors showed that the probability of perceiving events as being simultaneous or successive in time was a function of when events were presented with respect to the phase of ongoing α oscillations.
Consistent with this seminal work, recent EEG work capitalizing on the flash-lag effect has shown a dependency on the phase of spontaneous theta (θ, 4–7 Hz)/α oscillations and the perceived temporal disparity between visual events (Chakravarthi and VanRullen 2012). The flash-lag effect (MacKay 1958; Nijhawan 1994) is a visual illusion in which a visual event that is briefly flashed while another visual object is moving is perceptually misplaced in time. The general intuition behind the flash-lag illusion is that the trajectory of moving objects is updated, when that of stationary objects need not to be; as a result, one perceives a temporal disparity between the two visual objects. The EEG results suggest that the updating process may be cyclic (in the θ or α range) and that this cycle contributes to the perceived temporal disparity between the moving and the stationary object (Chakravarthi and VanRullen 2012). Recently, the correct individuation of items within 100 ms was also reported to correlate with a stronger phase-resetting of α oscillations (Wutz et al. 2014), supporting the notion that a period of α oscillation may temporally limit the processing of visual information by regulating windows of opportunity for the encoding of sensory information.
A corollary prediction is that the peak frequency of oscillatory α may inform on the propensity of visual stimuli to be perceived as dissociated in time or fused together within the same temporal frame. It is perhaps noteworthy that this hypothesis has a long history: replicating previous work (Meili and Tobler 1931), Brenner (1957) had showed that visual apparent motion and simultaneity thresholds evolved with age so that younger children perceived apparent motion for slower flickers than adults did. Brenner then speculated a possible link with the shifts in alpha peak frequency that were reported during development (Walters 1950). Consistent with such intuitions, the peak frequency of α has been reported to negatively correlate with the amount of fusion between two visual events (Samaha and Postle 2015): specifically, the faster an individual’s α peak frequency, the more accurate the temporal resolution of visual perception. These results brought further support to the notion that alpha oscillations may constrain the temporal granularity of visual perception. Consistent findings were also reported showing a positive correlation between the individual α peak frequency and the size of the temporal window within which auditory and visual events yielded a visual illusion, i.e., the temporal resolution within which multisensory integration occurred (Cecere et al. 2015). The authors further demonstrated that modulating the peak frequency with tACS affected the temporal integration of these audiovisual events.
These few examples illustrate two fundamental ways of understanding temporal resolution. On the one hand, that the timing of sensory events is coded and represented refers to the fundamental issue that timing is not absolute. As such, temporal statistics (the onset, the offset, the duration, but also the order of events with respect to other events) need to be recoded and/or computed. On the other hand, the event itself is represented (i.e., the flash of light or the picture of a face) with a temporal granularity that is constrained by the timing of information processing. The former alludes to the representations of time that are foundational for the elaboration of temporal perception; the latter alludes to the representation of events needed for visual perception, but not time per se. Both are associated with neural timing, but the relations between the representation of event timing and the timing of event representation remain uncertain. Finally, the notion of temporal window suggests that integration over time takes place with a particular temporal granularity. This observation underlies hypotheses regarding which neural mechanisms may be appropriate for the representation of particular kinds of information.
The temporal resolution of events was posited as the outcome of temporal integration taking place over ∼30 ms windows (Pöppel 1972, 2009), thus operating in the gamma band (Singer 1999). The synchronization of neural populations in the gamma range (γ, ∼>40 Hz) is essential for binding (Singer 1999) and neural communication (Engel et al. 2001; Fries et al. 2007; Freeman 2000). A seminal MEG study asked what neurophysiological markers dissociated perceiving one versus two auditory events (Joliot et al. 1994). In this study, the authors showed that perceiving one event correlated with one gamma band response (GBR), whereas perceiving two events correlated with two GBRs. This suggested a potential correlation between GBR and fusion threshold but not with the temporal order threshold. Electroencephalographic (EEG) findings have nevertheless challenged these observations by showing that a complete GBR per auditory transient was observed only when events were at least 100 ms apart (Boemio 2003). Specifically, the temporal structuring of acoustic events and the GBRs depicted three distinct phenomenological zones in this study (Boemio 2003): when 0.1 ms clicks were separated by more than 100 ms, they elicited a single isolated GBR, and clicks were perceived as discrete events; when the time between clicks decreased from 100 to 10 ms, the GBRs increasingly summated over time and so did the individual clicks, which acquired a pitch-like quality to them; and finally, when clicks were presented with less than 10 ms of temporal distances, a single GBR was elicited by the first click only, and the individuation of single clicks was lost. These findings suggest that the temporal fine structure of acoustic events may be preserved in cortex although not consciously represented as such. Comparable results in vision showed that V1 neurons follow flicker frequencies below the perceptual resolution (Gur and Snodderly 1997; see also Herbst et al. 2015). Hence, the temporal granularities in early stages of neural processing may, to some extent, preserve fine-grained temporal information, which remains subliminal or implicit.
In speech processing, the concept of temporal integration windows has been linked to linguistic theory, with neural oscillations as parsers of speech information into linguistic units (Poeppel et al. 2008; Giraud and Poeppel 2012). For instance, seminal MEG evidence reported an increased phase-tracking of speech dynamics in the theta range with increased speech intelligibility (Luo and Poeppel 2007). More recently, Ding et al. (2016) showed that the pattern of endogenous oscillatory responses was indicative of linguistic structures or grammatical knowledge and not of the temporal statistics of the speech stimuli. These observations are highlighting the importance of the endogenous temporal structuring of information by neural oscillations, irrespective of the a priori temporal statistics in the environment. Selective parsing of speech information can also be realized by using a bistable speech stream, which participants can perceive in two ways; it was shown with MEG that an individual’s gamma latency was informative of the conscious speech report (Kösem et al. 2016; Kösem and Van Wassenhove 2017).
As briefly discussed above, alpha oscillations operating on a ∼100 ms time scale are considered a major perceptual metric for establishing the temporal unit of visual processing (for review, VanRullen 2016). What is unclear is to which extent this temporal constant may affect overt and conscious representations of event timing. Although evidence for the existence of temporal windows of integration is increasing, there is surprisingly much less evidence dedicated to the representation of time. In a recent EEG study focusing on pre-stimulus α oscillations and visual simultaneity perception (Milton and Pleydell-Pearce 2016), the authors reported an α phase dependency of the proportion of asynchronous judgements, in line with previous work (Kristofferson 1967; Varela et al. 1981; Gho and Varela 1988). Overall, these findings are consistent with the principle that when two stimuli fall within the same oscillatory cycle, they are integrated, whereas when they fall on two distinct cycles, they are segregated/individuated. Using MEG, Lange et al. (2011) focused on tactile simultaneity and described that pre-stimulus β activity, and in one condition α activity, was predictive of participants’ simultaneity perception, notably for the most ambiguous tactile delays. Franciotti et al. (2011) used audiovisual stimuli and assessed participants’ simultaneity perception to series of synchronous or asynchronous stimuli, while they were recorded with MEG. The authors notably reported that a later latency of auditory evoked responses when the audiovisual stimuli were synchronous as compared to asynchronous.
4.2 Temporal Order and Segregation of Information
Once events have been individuated or discretized in the brain, temporal relations can be drawn. Fundamental temporal relations are that of order and simultaneity, which are currently not accounted for by timing models (Table 1). The seminal effect of endogenous attention on the perception of temporal order is captured by the prior entry effect (Titchener 1908; Spence and Parise 2010) in which attended stimuli are perceived earlier than non-attended stimuli. In a timing framework relying on neural delays and processing latencies, attended stimuli should show earlier latencies than non-attended stimuli. One EEG study focusing on audiovisual prior entry reported an increased amplitude of the evoked responses, and this pattern was interpreted as a possible perceptual gain for the representation of the stimulus being perceived first (McDonald et al. 2005). In another EEG study focusing on visuotactile order, earlier visual latencies were found when the visual event was perceived first (∼4 ms for the N1, ∼14 ms for the P300) although it was not fully predictive of the observed ∼40 ms perceptual difference (Vibell et al. 2007). Nevertheless, the authors considered that serial timing in the brain linearly reflected perceptual latencies (Vibell et al. 2007). In another audiovisual temporal order study, unsystematic changes in evoked response latencies were also reported (e.g., Kaganovich and Schumaker 2016).
In an oscillatory framework implicating alpha and gating-by-inhibition (Klimesch 2012; Jensen and Mazaheri 2010), one prediction would be that a decrease of α activity should be seen in the regions that prioritize information processing. For instance, decreased α power has been reported in visual cortices when visual detectability (Ergenoglu et al. 2004; Hanslmayr et al. 2007) and discrimination (van Dijk et al. 2008) increase. Hence, following the prior entry hypothesis, the ongoing fluctuations of pre-stimulus α activity in sensory cortices may predict temporal order perception. This hypothesis was recently tested in an MEG study (Grabot et al. 2017), in which the authors presented ambiguous audiovisual stimuli whose temporal delays were at the individual’s perceptual threshold: for a given participant, the audiovisual pair could be perceived in one order (e.g., the sound preceded the visual event) or in the other (e.g., visual event preceded the sound) roughly half of the times. This design enabled contrasting for a given physical time order, what in the MEG response could predict the changes in participant’s temporal order perception. One working hypothesis was that an increase in auditory alpha power at the same time as a decreased visual α power would predict the perception of a visual stimulus being first. Conversely, high visual alpha power and low auditory α power would predict the perception of the auditory stimulus first. However, this is not what was observed. Rather, the authors reported a pattern of pre-stimulus oscillatory activity that was a function of the individual’s temporal order bias: the larger an individual bias, the larger the pre-stimulus auditory α power differences between the two perceptual outcomes. This suggested that endogenous changes in α power may regulate the weights attributed to incoming sensory evidence for or against an individual’s structural bias. Additionally, in a recent MEG study, the phenomenon of order reversal of tactile stimuli delivered to crossed hands was studied with the hypothesis that α oscillations may regulate the illusory order (Takahashi and Kitazawa 2017). The authors described and localized several alpha components during the order task, but only one located in the parieto-occipital sulcus showed a phase dependency with the illusory reversal of tactile perceptual order. The authors concluded that the phase of the parieto-occipital α may be important not only for ordering of visual events but also for tactile events.
The state of other ongoing oscillations may also be essential in predicting the order of sensory events. In a study focusing on tactile simultaneity, the authors reported an increased pre-stimulus β power in somatosensory cortex when participants reported perceiving simultaneity (Lange et al. 2018) and pre-stimulus β activity was found to predict correct order perception (Bernasconi et al. 2011). The phase coherence in the β band between auditory and visual cortices has also been shown to impact simultaneity perception (Kambe et al. 2015).
As previously discussed in Sect. 4.1, the timing of events with respect to the phase of ongoing neural oscillations is relevant for their individuation. Although events can be fully manipulated in the lab, information in ecological settings come in analogical streams to which neural oscillations could be entrained to (Rees et al. 1986; Regan 1966; Thut et al. 2011). Neural entrainment can precisely align cortical processing to the timing of sensory events (Schroeder and Lakatos 2009) and can be conceived as an exogenous control or temporal tuning of internal rhythms, which were previously posited by the dynamic attending theory (Jones 1976; Large and Jones 1999). Thus, neural entrainment may help the brain calibrate its timing to the external temporal regularities and help establish internal temporal references for information processing (see also Sect. 5).
In this context, psychological studies focusing on temporal order have shown that adaptation to particular asynchronies could result in the temporal recalibration of perceived audiovisual synchrony (Fujisaki et al. 2004; Vroomen et al. 2004; Di Luca et al. 2009; Heron et al. 2010; Roseboom and Arnold 2011). This signifies that if a participant is shown with a sound systematically preceding a visual flash for a few seconds to minutes (i.e., adapted to an audiovisual lag), the subsequent perception of audiovisual order will be shifted in the direction of the lag: a sound will have to be presented even earlier to be perceived as simultaneous with the visual stimulus. In an MEG study focusing on which neural mechanisms may support audiovisual temporal recalibration (Kösem et al. 2014), the authors reported that although rhythmic audiovisual stimuli entrained a 1 Hz neural oscillation, the phase response of the entrainment oscillation in auditory cortices varied between the beginning and the end of the adaptation period. In other words, the phase response showed some non-stationarities with respect to the external statistics of the rhythmic stimuli and these non-stationarities were telling of the participant’s perceptual timing: specifically, the direction and the magnitude of the oscillatory phase shifts linearly predicted an individual’s subsequent temporal order threshold (or point of subjective simultaneity). These results suggested that although entrainment to external stimuli may temper endogenous timing, the temporal framing of information processing is regulated by endogenous mechanism that is important for subjective time perception. This observation was consistent with the prior observation that the preferred phase of oscillatory entrainment is context-dependent (Besle et al. 2011; Gomez-Ramirez et al. 2011; Lakatos et al. 2008; Rees et al. 1986) and that neural entrainment may not be a fully passive neural response.
At this time, several EEG and MEG studies are highlighting to the implication of the phase of both spontaneous and entrained oscillations as possible contributors to the mapping of temporal order.
5 Implicit Timing
Dynamic sensory environments unfold in time and space, and the human perceptual system is tuned to extract inputs along these dimensions. In the last two decades, the view of the brain as a predictive organ rather than a passive receiver has become the prevalent one (Rao and Ballard 1999; Friston 2005; Bar 2007). However, the largest part of the work dedicated to predictive processing has focused on spatial, feature-based, or semantic predictions, namely, the what and where, but much less on the when. Temporal aspects of predictions have only recently become a focus of interest (Coull and Nobre 1998; Battelli et al. 2007; Arnal and Giraud 2012; Nobre and van Ede 2018) and are subject to a rapidly growing literature, with significant contributions from EEG and MEG experiments.
Temporal predictions allow for the orientation of attention in time (Nobre et al. 2007), resulting in increased response speed, perceptual detection, and discrimination performance for stimuli occurring at expected time points (Woodrow 1914; Niemi and Näätänen 1981; Jones et al. 2002; Doherty et al. 2005; Rolke and Hofmann 2007; Cravo et al. 2013; Morillon et al. 2014; Rohenkohl et al. 2014; Herbst and Obleser 2017). Temporal predictability also improves short-term memory performance (Wilsch et al. 2014, 2018). The neural representation of stimuli that occur at expected time points differs from those occurring at unexpected time points from the earliest sensory representations in subcortical to cortical regions as measured in animal electrophysiology (Ghose and Maunsell 2002; Jaramillo and Zador 2011) to cortical representations assessed as evoked potentials in humans (such as the P100 and N100; e.g., Miniussi et al. 1999; Correa et al. 2006; Rimmele et al. 2010; Schwartze et al. 2013; Hsu et al. 2014).
To form temporal predictions, endogenous dynamics presumably need to form an internal representation of the temporal statistics of the inputs. To explain how the temporal structure of external inputs can be internalized and endogenously represented, influential theories have capitalized on the observation that, in natural environments, temporal regularities often convey a rhythmic although not perfectly periodic structure. The DAT (dynamic attending theory; Jones 1976; Large and Jones 1999) notably proposes that oscillatory systems adjust their time scale to match environmental time scales, thereby providing a parsing mechanism. DAT and other theoretical frameworks relying on the use of periodic mathematical formulations can provide a basis for guiding MEG experimentation, as will be described in more details below. The rhythmic approach is appealing in that it mimics the rhythmic properties of brain dynamics. At the same time, this parallelism makes it impossible to fully disentangle the internal representation of the external temporal structure from the external temporal structure itself. A current means to bypass this issue is to use so-called foreperiod paradigms, which induce temporal predictions for single time intervals, allowing to investigate an endogenous representation of temporal predictions devoid of any external temporal input structure (see below).
5.1 Entrainment of Delta/Theta Band Oscillations to External Rhythms
Brain rhythms are thought to reflect large-scale fluctuations in neuronal excitability, that is, brain states that are more or less beneficial for processing sensory input. Several EEG studies have shown that visual and auditory perception fluctuate with the phase of low-frequency spontaneous oscillations from 1 to 12 Hz (Busch et al. 2009; Mathewson et al. 2009; Ng et al. 2012; Strauß et al. 2014; Henry et al. 2016). By retrospectively sorting trials according for behavioral performance, these studies have revealed that the phase angle at which a stimulus occurs determines the efficiency with which it is processed. Crucially, fluctuations of cortical excitability can be driven by exogenous temporal regularities which entrain neural oscillations (Lakatos et al. 2008; Schroeder and Lakatos 2009; Besle et al. 2011), resulting in similar dependencies between oscillatory phase and behavioral performance. These findings are taken as empirical evidence for DAT, which postulates rhythmic fluctuations of endogenous attention aligned with rhythmic sensory inputs (Jones 1976; Jones and Boltz 1989; Large and Jones 1999; Jones et al. 2002), thereby explaining the observed increases in behavioral performance by the alignment of states of enhanced attention to the external rhythm. Accordingly, a number of EEG studies have reported enhanced inter-trial phase coherence of slow oscillations (δ (1–3 Hz) and θ (4–7 Hz) bands) prior to temporally predictable events in rhythmic context and critically modulations of behavior by entrained oscillatory phase in sensory and motor regions (Breska and Deouell 2017; Cravo et al. 2013; Stefanics et al. 2010, Exp. 1; Henry and Obleser 2012; Henry et al. 2014; Arnal et al. 2015; Keil et al. 2016). For instance, in a recent MEG experiment (Herrmann et al. 2015), the authors directly tested the relation between temporal predictions and performance, by modeling the phase of an attentional oscillator at the onset of auditory events that were slightly jittered with respect to an entraining rhythm. Detection performance correlated with the modeled phase, but only in the presence of high-amplitude delta oscillations in auditory cortices.
In addition to entrainment effects found at the stimulation frequencies, some of the reported studies above report modulations of the amplitude of α oscillations (Rohenkohl and Nobre 2011; Samaha et al. 2015; Bidet-Caulet et al. 2012) prior to an expected stimulus. Others have also reported modulations of beta oscillations (Arnal et al. 2015; Keil et al. 2016; Morillon and Baillet 2017; see also Arnal 2012; Arnal and Giraud 2012) in the anticipation time window. If the fluctuations in α power most likely reflect anticipatory attention, the modulations of beta power may play a role in the internalization and tracking of rhythmic inputs, especially, but not only, with the engagement of the motor system.
5.2 Rhythmic Modulation of Beta Rhythms in Rhythm Perception
The brain can track the statistical structure of the environment, including rhythmic structures. However, only the human brain has the ability to synchronize movements to a metronome beat (Leow and Grahn 2014; Kotz et al. 2018, but see Patel et al. 2009). Synchronizing movements to the beat involves parsing the temporal structure of a sequence as well as the prediction of future events in the rhythmic beat sequences (Leow and Grahn 2015; Grahn and Rowe 2013). Synchronization studies have revealed several interesting effects. The capability to predict future events on the short time scales can be appreciated in finger tapping to the beat. When compared to reaction times to a stimulus, the interval between beat and tap is notably shorter than the fastest reaction times (Repp 2005; Repp and Su 2013). The parsing of temporal structure thus entails a structuring of temporal intervals. In turn, the percept of the rhythmic sequence leverages perception of temporal intervals such that temporal processing is improved when embedded in the beat sequence (Patel et al. 2005; Povel and Essens 1985).
Time-resolved neuroimaging critically contributed to the work on beat perception by allowing to observe “ups and downs” of β oscillations (15–30 Hz). Typically, power modulation of beta amplitude follows the tempo of sound stimulation in auditory areas (Snyder and Large 2005; Zanto et al. 2006). For example, Fujioka et al. (2012) found that modulations of β power follow the beat, such that the power of β ramps up in prediction of upcoming events, followed by a sharp decrease after the stimulus; β power reaches a minimum with a latency of around 200 ms and subsequently recovers with a shallow slope. That dynamic modulation of β power in response to beat is thought to support parsing and structuring of beat sequences (Fig. 9a). Moreover, it is suggested that beta may signify expectation by carrying a “predictive” or “expectation” signal. In line with that assertion, during the perception of rhythmic sequences, Snyder and Large (2005) found that β “fills in” missing beats at the time points when the sound did not occur. However, cautionary notes are present in the literature as well with respect to these observations. Meijer et al. (2016) used inter-β intervals between 1 and 2, thus exceeding typical stimulation range. Contrary to recent views, the β power reached a peak at a similar latency irrespective of presentation rates as opposed to peaking at a fixed interval before the next stimulus (Fig. 9b, upper panel). This demonstrates that, at interstimulus intervals between 1–2 s, β synchronization slopes are not modulated by timing mechanisms related to prediction of upcoming stimuli. The authors proposed that when shorter interval durations are used, as in most of beat studies, β resynchronization is interrupted by the presentation of a new stimulus (Fig. 9b, bottom panel). Consequently, it may seem as if beta power attains its peak prior to upcoming stimuli. However, the slope of the ramping β power depends on tempo, something that is not easily accounted for by Meijer et al.’s (2016) explanation. Future studies will have to investigate the still open notion of predictive timing of beta power in different temporal ranges.
Very few studies investigated the interplay between β and δ oscillations in rhythm perception (Doelling and Poeppel 2014; Morillon and Baillet 2017). A close link between auditory and motor circuits in the brain, possibly coordinated through β oscillations, could subserve beat perception (Doupe et al. 2005; Patel 2006). In support of this hypothesis, the modulation of β oscillations was reported in tasks in which participants were asked to passively listen to a beat (e.g., Fujioka et al. 2012) but also in which they were required to actively synchronize to a beat (Bartolo et al. 2015). In an elegant study (Morillon and Baillet 2017; see also Morillon et al. 2014), participants had to actively tap to the behaviorally relevant rhythm, which improved auditory task performance and revealed that in both conditions temporal predictions were reflected by a combination of β power modulation nested in the phase of the entrained delta oscillation. Together, these studies imply that β oscillations may work as a carrier for top-down processes enabling the modulation of auditory processing. Not surprisingly, beat perception can be improved by motor behavior (Zatorre et al. 2007), but interestingly, several MEG studies also suggested that motor regions may be involved in beat perception even when no overt movements were required. For example, the coherence in β band has been reported between auditory and motor regions such as the supplementary motor area, the cerebellum, and the pre- and postcentral gyri when subject passively listened to auditory beats (Fujioka et al. 2012). The coupling between auditory and motor areas has also been indicated by fMRI studies (Grahn and Rowe 2009). Generally, motor areas tend to be in involved during beat perception (Grahn and Brett 2007) confirming MEG results. These modulations are in line with theories postulating a predictive modulation of auditory areas by motor areas in beat perception (Patel and Iversen 2014; Repp 2005). Similar modulations according to beat and meter were observed to elicit sustained, periodic brain responses tuned to beat frequency (Nozaradan et al. 2011, 2012, 2016).
In addition to supporting temporal expectations by filling missing beats (Snyder and Large 2005) and predicting upcoming tones in a sequence (Fujioka et al. 2012), β oscillations may contribute to the generation of subjective accents. Recordings of β oscillations were instrumental in demonstrating that β was uniquely sensitive to manipulation of the endogenous sense of meter (Iversen et al. 2009): when participants were asked to impose different metrical interpretations on a rhythmic sequence, MEG recordings showed an enhancement of β power for subjectively enhanced beats (Fig. 9c). Similar results were shown for more complex metric structures such as in the “waltz” condition of the Fujioka et al. study (2015). Chang et al. (2018) also reported that the predictability of a pitch change can modulate the power of β oscillations immediately prior to a deviant pitch onset, such that when pitch deviants were predictable, they were preceded by a decrease in β power.
A critical question thus becomes whether oscillatory dynamics are fully dedicated to the encoding of exogenous temporal regularities, thereby depending on the structure of the input signal, or whether they are indicative of higher endogenous construals for incoming inputs, reflecting an internal representation of temporal prediction (Pöppel 1972; Varela 1999).
5.3 Temporal Predictions in the Absence of Rhythmic Context
To assess whether temporal predictions can be fully internalized and instantiated without a local temporal structure, paradigms relying on interval-based instead of rhythmic temporal predictions have been developed (Woodrow 1914; Niemi and Näätänen 1981). Here, temporal predictions are induced via the so-called foreperiod interval, that is, the time between a warning stimulus and a target stimulus. The foreperiod is either fixed over a block, or the warning stimulus is used as a cue that signals at which foreperiod the target will occur. Multiple EEG studies (Cravo et al. 2011; Samaha et al. 2015; ten Oever et al. 2015; Barne et al. 2017; Herbst and Obleser 2017, 2018; Solís-Vivanco et al. 2018) and at least one MEG study (Todorovic et al. 2015) have addressed how such single-interval temporal predictions are reflected in neural dynamics.
Alluding to the entrainment studies described above, they found that delta phase coherence increased before the most likely time point of target occurrence and that behavioral performance varied with the phase angle of the oscillation. Cravo et al. (2011; see also Fig. 8c) applied a probabilistic manipulation of three different foreperiods in a visual paradigm, leading to different target occurrence probabilities over time. They report increases in the phase-locking of theta oscillations, as well as an anticipatory β amplitude increase, that occurred prior to the most likely time point of target occurrence defined by the respective condition. These studies strongly suggested that the internalization of temporal predictions via slow neural oscillations found for rhythmic temporal predictability also generalizes to interval-based predictions. Yet, an unanswered question is whether the frequencies at which the effects are found reflect single-trial entrainment to the average interstimulus interval or a frequency band dedicated to this function. Another open question is, in such paradigms, how much phase-locking at slow oscillatory frequencies overlaps with the neural generators and dynamics of an “implicit CNV” (Praamstra et al. 2006; Mento 2013; Herbst et al. 2018).
Critically, the above studies required a speeded response to a target, and Stefanics et al. (2010) explicitly instructed their participants about the underlying foreperiod distributions and the cues, which probably engaged the use of explicit timing strategies. In two recent studies in which foreperiods were manipulated strictly implicitly (block, and trial-wise), and in which a perceptual discrimination task had to be performed, no phase coherence effects were observed in the delta band (Herbst and Obleser 2017, 2018). However, in Herbst and Obleser (2018), delta phase angles measured shortly after the implicit temporal cue were found to vary between a temporally predictive and non-predictive condition. Importantly, the delta phase angles predicted individuals’ perceptual sensitivity for processing the target, which occurred on average 1.8 s later. These results are in line with a study by Barne et al. (2017), who found that the recalibration of temporal predictions is reflected by a shift in delta phase angles and the general notion that phase shifts may regulate the temporal mapping of events (Kösem et al. 2014). In sum, interval-based temporal predictions seem to be internalized by the phase of low-frequency oscillations, but the observational patterns depend strongly on the experimental paradigm. Critically, even though all of the above studies did not use strictly periodic stimulation, it cannot be fully excluded that residual rhythmicity explains some of the described phase coherence effects (Obleser et al. 2017).
Endogenous brain dynamics represent temporal statistics of the environment that go beyond single intervals. For instance, foreperiod distributions can be transformed to the hazard function, which describes the probability of an event to occur at a certain point in time, given it has not yet occurred. Importantly, a uniform (i.e., flat) foreperiod distribution leads to a rising hazard over time, reflecting the rising expectation for an event to occur. Recordings from monkeys’ lateral intraparietal area have shown that different hazard functions are represented in neural activity (Janssen and Shadlen 2005), which has been confirmed by EEG experiments (Trillenberg et al. 2000; Cravo et al. 2011; Wilsch et al. 2015a; Herbst et al. 2018). Using a probabilistic and implicit manipulation of foreperiods, Herbst et al. (2018; see Fig. 8d) showed that human EEG activity tracks even subtle variations of temporal hazard and that this tracking signal emerges mainly from the SMA. Finally, temporal predictions have been studied in combination with spatial predictions (Auksztulewicz et al. 2017; Heideman et al. 2017, 2018; Solís-Vivanco et al. 2018), often surfacing as a lateralization of alpha oscillatory power occurring specifically before predicted stimulus onsets. Even though these different types of predictions interact, there is no doubt that temporal predictions alone are relevant to behavior and are represented endogenously.
Taken together, these studies show that temporal statistics of the environment are internalized in the temporal structure of brain dynamics, which in turn tune brain systems to the temporal properties of the environment. The brain is always timing, even when timing is not an explicit requirement of the task or useful to the situation at hand. A currently rather unexplored question is to what extend the endogenous representation of time that underlies temporal predictions in implicit timing situations relates to explicit representations of time such as the perception of duration.
Throughout the chapter, we have discussed the concept of time and its crucial relevance to human cognitive processing, providing the structure of dynamic sensory environments, and being reflected in a multitude of psychological phenomena, from implicit temporal predictions to the conscious apprehension of time intervals. Despite – or, perhaps, because of – the ubiquity of time, the cognitive mechanisms and neural architecture that provide us with the ability to time, and the subjective experiences associated to this function, are not well understood. MEG and EEG are today the most useful tools available to study these processes, because they provide the adequate temporal resolution for the phenomena under investigation and allow researchers to assess timing as it unfolds, for instance, during a to-be-timed interval. As outlined above, this research has not provided us with a unique substrate of psychological time, such as a dedicated pacemaker for an internal clock. Nevertheless, tremendous progress has been made in describing how endogenous brain dynamics implement aspects of temporal processing, reflected, for instance, in the CNV/CMV, delta/theta, and beta oscillations during critical time intervals and in evoked responses to events that define these intervals, such as the MMN/F. The study of “time” poses not only important conceptual but also methodological challenges when designing sensible experimental paradigms and data analysis pipelines. To give one example, varying time intervals, as often needed to study duration perception, pose a problem to the traditional averaging approaches applied in ERP/F but also time-frequency analyses, in which a considerable number of similarly long epochs are overlaid. Recently developed methods such as decoding or encoding models allow more flexibility in the input signals and thus represent promising new avenues for the field of timing research.
- Arnal LH (2012) Predicting ‘when’ using the motor system’s beta-band oscillations. Front Hum Neurosci 6. https://doi.org/10.3389/fnhum.2012.00225
- Barlow JS, Brazier MAB (1957) The pacing of EEG potentials of alpha frequency by low rates of repetitive flash in man. Electroencephalogr Clin Neurophysiol 9(1):161Google Scholar
- Bergson H (1889) 1913. Time and free will: an essay on the immediate data of conscious ness. TransGoogle Scholar
- Boemio AB (2003) The perceptual representation of acoustic temporal structure. Doctoral dissertation, University of Maryland, College ParkGoogle Scholar
- Buonomano D (2017) Your brain is a time machine: the neuroscience and physics of time. W. W. Norton & CompanyGoogle Scholar
- Caton R (1875) Electrical currents of the brain. J Nerv Ment Dis 2(4):610Google Scholar
- Deecke L, Weinberg H, Brickett P (1982) Magnetic fields of the human brain accompanying voluntary movement: Bereitschaftsmagnetfeld. Exp Brain Res 48(1):144–148Google Scholar
- Fraisse P (1988) Time perception and evoked potentials. Paris V, Lab de Psychologie Expérimentale, ParisGoogle Scholar
- Gallistel CR (1990) The organization of learning. The MIT Press, Cambridge, MAGoogle Scholar
- Gho M, Varela FJ (1988) A quantitative assessment of the dependency of the visual temporal frame upon the cortical rhythm. J Physiol 83(2):95–101Google Scholar
- Herbst SK, Obleser J (2017) Implicit variations of temporal predictability: shaping the neural oscillatory and behavioural response. Neuropsychologia 101(July):141–152. https://doi.org/10.1016/j.neuropsychologia.2017.05.019CrossRefGoogle Scholar
- Herbst SK, Obleser J (2018) Implicit temporal predictability biases slow oscillatory phase in auditory cortex and enhances pitch discrimination sensitivity. bioRxiv, 410274Google Scholar
- Herrmann B, Henry MJ, Haegens S, Obleser J (2015) Temporal expectations and neural amplitude fluctuations in auditory cortex interactively influence perception. NeuroImage. http://www.sciencedirect.com/science/article/pii/S1053811915008253
- Hoagland H (1935) Pacemakers in relation to aspects of behavior. Macmillan, New YorkGoogle Scholar
- Holcombe A (2013) The temporal organization of perception. In: Handbook of perceptual organization, OxfordGoogle Scholar
- Holubář J (1969) The sense of time: an electrophysiological study of its mechanisms in man. MIT Press, Cambridge, MAGoogle Scholar
- Hsu Y-F, Hamalainen J, Waszak F (2014) Both attention and prediction are necessary for adaptive neuronal tuning in sensory processing. Front Hum Neurosci 8. https://doi.org/10.3389/fnhum.2014.00152
- Kononowicz TW, Van Rijn H (2011) Slow potentials in time estimation: the role of temporal accumulation and habituation. Front Integr Neurosci 5:48Google Scholar
- Kononowicz TW, Roger C, van Wassenhove V (Imprint) Temporal metacognition as the decoding of self-generated brain dynamics. Cereb Cortex, https://doi.org/10.1093/cercor/bhy318
- Lashley KS (1951) The problem of serial order in behavior, vol 21. Bobbs-MerrillGoogle Scholar
- Leow LA, Grahn JA (2014) Neural mechanisms of rhythm perception: present findings and future directions. In: Neurobiology of interval timing. Springer, New York, pp 325–338Google Scholar
- Macar F (1977) Meaning of contingent negative variations with the temporal dimension of behavior. Annee Psychol.Google Scholar
- Meck WH (1983) Selective adjustment of the speed of internal clock and memory processes. J Exp Psychol Anim Behav Process 9(2):171Google Scholar
- Meili R, Tobler E (1931) Les mouvements stroboscopiques chez les enfants. W. Kundig, GenèveGoogle Scholar
- Mento G (2013) The passive CNV: carving out the contribution of task-related processes to expectancy. Front Hum Neurosci 7(December). https://doi.org/10.3389/fnhum.2013.00827
- Morillon B, Schroeder CE, Wyart V (2014) Motor contributions to the temporal precision of auditory attention. Nat Commun 5(October). https://doi.org/10.1038/ncomms6255
- Näätänen R (1992) Attention and brain function. Erlbaum, LondonGoogle Scholar
- Ng KK, Tobin S, Penney TB (2011) Temporal accumulation and decision processes in the duration bisection task revealed by contingent negative variation. Front Integr Neurosci 5:77Google Scholar
- Niemi P, Näätänen R (1981) Foreperiod and simple reaction time. Psychol Bull 89(1):133Google Scholar
- Nozaradan S, Peretz I, Keller PE (2016) Individual differences in rhythmic cortical entrainment correlate with predictive behavior in sensorimotor synchronization. Sci Rep 6:20612Google Scholar
- Patel AD, Iversen JR (2014) The evolutionary neuroscience of musical beat perception: the Action Simulation for Auditory Prediction (ASAP) hypothesis. Front Syst Neurosci 8:57Google Scholar
- Penney TB (2003) Modality differences in interval timing: attention, clock speed, and memory. In: Meck WH (ed) Functional and neural mechanisms of interval timing. CRC Press, Boca Raton, pp 209–233Google Scholar
- Picton TW, Woods DL, Baribeau-Braun J, Healey TM (1977) Evoked potential audiometry. J Otolaryngol 6(2):90–119Google Scholar
- Polti I, Martin B, van Wassenhove V (2018) The effect of attention and working memory on the estimation of elapsed time. Sci Rep 8, 1–11Google Scholar
- Rammsayer TH (1999) Neuropharmacological evidence for different timing mechanisms in humans. Q J Exp Psychol 52(3):273–286Google Scholar
- Ross DA (1968) Time perception and brain rhythms. Theses and dissertations. 2920. https://preserve.lehigh.edu/etd/2920
- Stroud JM (1956) The fine structure of psychological time. In: Quastler H (ed) Information theory in psychology: problems and methods. Free Press, New York, pp 174–207Google Scholar
- Van Dijk H, Schoffelen JM, Oostenveld R, Jensen O (2008) Prestimulus oscillatory activity in the alpha band predicts visual discrimination ability. J Neurosci 28(8):1816–1823Google Scholar
- van Wassenhove V, Wittmann M, Craig AD, Paulus MP (2011) Psychological and neural mechanisms of subjective time dilation. Frontiers in Neuroscience 5:56Google Scholar
- Varela FJ (1999) The specious present: a neurophenomenology of time consciousness. In: Naturalizing phenomenology: issues in contemporary phenomenology and cognitive science, vol 64. Stanford University Press, Stanford, pp 266–329Google Scholar
- Walter WG (1950) Normal rhythms – their development, distribution and significance. In: Hill D, Parr G (eds) Electroencephalography; a symposium on its various aspects. Macdonald, Oxford, UK, pp 203–227Google Scholar
- Wearden JH (2005) Origins and development of internal clock theories of time. Psychol Fr 50(1):7–25Google Scholar
- Wittmann M, Van Wassenhove V, Craig B, Paulus MP (2010) The neural substrates of subjective time dilation. Front Hum Neurosci 4:2Google Scholar
- Woodrow H (1914) The measurement of attention. The psychological monographs, 17(5). Psychological Review Company, PrincetonGoogle Scholar