Keywords

1 Introduction

Each of the presently available technologies for noninvasive electromagnetic or hemodynamic measurements of brain activity offers different spatiotemporal resolution and physiological sensitivity. Human functional MRI [6, 30] is temporally limited by the slow hemodynamic response (~ seconds) due to relative cerebral blood flow (CBF), cerebral blood volume (rCBV), and metabolism changes, which are indirect markers of neuronal signaling. Using the echo-planar imaging [36] technique, fMRI can typically provide a spatial sampling on a millimeter scale with homogeneous volumetric sensitivity and whole-brain coverage.

Magnetoencephalography (MEG) and electroencephalography (EEG) in turn detect extracranial magnetic fields and electric potential differences on scalp, which are both elicited by spatially clustered and temporally coherent postsynaptic neuronal currents [40]. Consequently, MEG/EEG can be used to study neuronal dynamics with millisecond resolution. Different from fMRI, where tomographic images are usually obtained, the spatial resolution of MEG/EEG is related to the capability to resolve intracranial current sources from extracranial measurements. To characterize the distribution of postsynaptic neuronal currents responsible for the macroscopically measured voltage and magnetic field, we have to solve an electromagnetic inverse problem, which admits no unique solution [26, 28]. With auxiliary mathematical and physiological assumptions, we can get reasonable estimates of the spatial distribution of the neuronal currents that generate the MEG (and EEG) measurements (see, Fig. 1).

Fig. 1
figure 1

(a) A simulated “point-like” current dipole shown on a simulated cortical surface obtained by shrinking the inner skull segmented from anatomical MRI by about 1 cm. The orientation of the dipole is tangential to the cortical surface. (b) The simulated MEG measurement shown with a realistic outer skin surface and sensor locations from an actual measurement. (c) The Minimum-Norm Estimate (MNE) vector field is displayed with black arrows, and the MNE amplitude is shown in color scale. (d) The full width at half maximum (FWHM) of the MNE point-spread function gives an estimate for the spatial resolution of the source localization method

Full size image

There are two main lines of approach to the MEG/EEG inverse problem (for a review, see [22]). The most traditional one assumes that the measurements are generated by a small number of equivalent current dipoles (ECDs). The main challenges with this approach are that i) the optimization problem for finding the best matching parameters for the ECDs is nonlinear and ii) the optimization becomes more and more difficult with increasing number of dipoles. The problem of handling the a priori unknown number of dipoles is also a nontrivial one, because it requires rather involved numerical methods [3, 29]. The Minimum-Norm Estimate (MNE) approach circumvents the nonlinearity of estimating current source location and orientation by assuming a discrete grid of source locations, each of which has three orthogonal dipole orientations. With respect to estimating dipole amplitudes, the inverse problem is linear. A least squares estimator (i.e., the MNE) can be calculated by standard numerical methods. The benefit is that the number of sources need not be known a priori, and a closed form solution is available in a computationally robust form. The natural drawback of this method is that the identity of each “localized source” becomes less clear-cut. In the case of Fig. 1, both the MNE and the single ECD explain the data equally well, which is a manifestation of the non-uniqueness of the inverse solution.

In principle, fMRI and EEG/MEG data can be integrated in order to achieve high spatiotemporal brain imaging (for a review, see, e.g., [44]). The basic rationale for such data integration is based on neurophysiological evidence: invasive studies in primates suggest that BOLD fMRI signal increases are closely related to the same postsynaptic neuronal activity [34] that generates MEG responses [22, 40]. Tight coupling between neuronal and vascular events has also been reported in the somatosensory system of rodents [12, 13]. Tentatively, these observations support the computational strategy of using fMRI, a vascular marker of neuronal events, as a physiological constraint for reducing the spatial ambiguity in the source localization of MEG/EEG. For example, the ECD fitting method in MEG/EEG source localization can be informed by fMRI [2, 20, 48]. The statistical maps derived from fMRI data can also be used as a spatial prior for the distributed source reconstruction [10, 32, 33]. A further study using simulations demonstrated the advantage of combined fMRI and EEG for a higher efficiency of cortical current density estimation at different signal-to-noise ratios (SNRs) with the presence of both fMRI-visible and fMRI-invisible sources [5]. MEG has a millisecond temporal resolution, ideal for studying cortical oscillations. It has been shown that integrating fMRI and MEG can also improve the localization of cortical sources of oscillatory activity [31].

In this article, we use the cortically constrained distributed source modeling framework to illustrate how fMRI information can be used to assist MEG/EEG localization and what are the potential benefits and pitfalls of this approach. We then briefly discuss the further modeling efforts and extensions of the fMRI-weighted MNE that have been presented in the literature. We also elaborate on the practical aspects of designing a successful MEG/EEG/fMRI experiment, data analysis, and interpretation of the results. The neurovascular coupling, technical challenges, and opportunities for further optimizing the integration are also described. As the MEG and EEG signals have a similar physiological origin, but their sensors have different sensitivity profiles, the combination of the two yields theoretically the best localization results. However, the EEG is substantially more sensitive to the volume conductor model: the poorly conducting skull distorts and smears the electric scalp potentials, whereas the currents in the skull and scalp make only a minor contribution to MEG [23]. In what follows, all models and methods could be formulated in terms of both MEG and EEG measurements, but we use MEG as our main example due to the less error-prone forward model. Ultimately, we expect to develop multimodal MEG/EEG/fMRI neuroimaging methodology for characterizing spatiotemporal functional connectivity in large-scale neural networks of the human brain with high sensitivity and accuracy.

2 Methods

2.1 Minimum-Norm Estimates

Under the quasi-static approximation of Maxwell’s equations [22], the measured MEG signals and the underlying current source are related by a linear transformation:

$$ \mathbf{Y}(t)=\mathbf{AX}(t)+\mathbf{N}(t), $$
(1)

where Y(t) is an m-dimensional vector containing measurements from m sensors at time instant t; X(t) is a 3n-dimensional vector denoting the unknown amplitudes of the three components of n current sources; A is the gain matrix representing the mapping from the unit dipole components to MEG sensors, i.e., the solution of the forward problem; and N(t) denotes noise in the measured data. For typical analysis of evoked responses, the measurement noise N(t) can be assumed to be Gaussian with zero mean and a time-independent spatial covariance matrix C, which can be estimated from the data. The number of sensors is some hundreds, and any realistically spaced grid covering the cortex requires thousands of source points. Thus, the inverse problem is severely underdetermined as the number of equations m (sensors) is an order of magnitude smaller than the number of unknowns 3n (source amplitudes). With the presently available accurate reconstructions of cortical surfaces [8, 15, 16], the locations of the sources can be constrained according to the individual anatomy. If we assume that apical dendrites of pyramidal cells, which are mainly oriented perpendicular to the cortical mantle, are the principal generators of the MEG signals [40], we can also fix the orientation of the sources and reduce the number of unknowns from 3n to n (see, Fig. 2).

Fig. 2
figure 2

(a) A surface model of the left cortical hemisphere gray-white matter boundary reconstructed by FreeSurfer. (b) A close-up view of the sensorimotor cortex, showing the dipole sources as red arrows, oriented perpendicular to the cortical surface

Full size image

If we further assume that the source amplitudes have a Gaussian a priori distribution with a time-independent covariance matrix R, we obtain the maximum a posteriori (MAP) estimate or the ℓ2 minimum-norm solution, which is linearly related to the measurements [9]:

$$ {\mathbf{X}}^{\mathrm{MNE}}(t)={\mathbf{RA}}^T{\left({\mathbf{ARA}}^T+{\lambda}^2\mathbf{C}\right)}^{-1}\mathbf{Y}(t)={\lambda}^{-2}{\mathbf{RA}}^T{\left({\lambda}^{-2}{\mathbf{ARA}}^T+\mathbf{C}\right)}^{-1}\mathbf{Y}(t)=\mathbf{WY}(t), $$
(2)

where λ2 is a regularization parameter, which is introduced to avoid noise amplification in the matrix inversion, and the superscript T indicates matrix transpose. The parameter λ2 can be estimated from the amplitude signal-to-noise ratio (SNR) of the whitened data: \( {\lambda}^2=\mathrm{tr}\left(\tilde{\mathbf{A}}\mathbf{R}{\tilde{\mathbf{A}}}^T\right)/\mathrm{tr}\left({\mathbf{I}}_{m\times m}\right)/{\overline{SNR}}^2 \). Here tr(·) denotes the trace of a matrix. The whitened forward operator is \( \tilde{\mathbf{A}}={\mathbf{C}}^{-1/2}\mathbf{A} \), and Im × m is the whitened (unit) noise covariance. \( \overline{SNR} \) is an estimate for the SNR of the data.

The solution XMNE(t) in Eq. (2) provides the values of the current amplitudes that best fit the MEG measurements in the least squares sense, with the additional constraint of having the minimal (Euclidean) ℓ2 norm. It may be desirable to further transform the resulting current distribution estimate into a statistical map that takes into account the spatial distribution of fluctuations in the source estimate caused by noise [10]. To this end, we need to consider the variance of the linear inverse estimates, when the data consists of noise only:

$$ {w}_k^2={\left({\mathbf{W}\mathbf{CW}}^T\right)}_{kk}={\left({\tilde{\mathbf{W}}\tilde{\mathbf{W}}}^T\right)}_{kk}. $$
(3)

For fixed-orientation sources, we now obtain the noise-normalized activity estimate for the kth dipole and tth time point as the ratio

$$ {X}_k^{dSPM}(t)=\frac{X_k^{\mathrm{MNE}}(t)}{w_k}. $$
(4)

The dSPM thus normalizes the actual MNE by the standard deviation of the fluctuation of the MNE that results from inverting data, which consists of noise only.

To incorporate the spatial information from fMRI, it has been suggested that MEG source locations coinciding with significant fMRI activity were given a higher variance in the a priori source covariance matrix R [10, 32]. Specifically, the source covariance matrix was assumed diagonal, and the fMRI weighting for source location k was encoded as:

$$ {R}_{kk}=\left\{\begin{array}{l}{\sigma}_1^2\kern1em \mathrm{if}\kern1em k\kern1em \mathrm{active}\kern0.5em \mathrm{in}\kern0.5em \mathrm{fMRI}\\ {}{\sigma}_0^2\kern1em \mathrm{otherwise}\end{array}\right. $$
(5)

A weighting ratio of 10:1 between active (\( {\sigma}_1^2 \)) and inactive cortical locations (\( {\sigma}_0^2 \)) has been suggested by a simulation study [32].

2.2 Example: MNE Analysis and the Effect of fMRI Weighting

Figure 3a shows a simulated source consisting of two patches of activated cortex, located approximately at motor and auditory areas. As dictated by physics of quasi-static magnetic fields generated by dipolar current sources in a nearly spherical volume conductor, MEG is mostly sensitive to sources in the sulcal walls. The auditory cortical source is located entirely in the wall of the Sylvian fissure, whereas the motor cortical source extends over the precentral gyrus, thus giving less optimal summation of the MEG fields. This is readily visible in Fig. 3b: the dipolar field pattern from the auditory cortex is much more prominent. This translates directly to the MNE of Fig. 3c: even though originally of similar amplitude, the motor cortical source estimate is weaker, and the gyral part of the source is missing. As smaller more superficial sources can produce similar MEG field as larger deep sources, the minimum-norm constraint has a tendency to push the estimates toward the more superficial parts of the brain surface. However, as the superficial parts are also more prone to noise fluctuations in the inverse estimates, the effect of the noise normalization of dSPM counteracts this and pushes the source maxima deeper (Fig. 3d).

Fig. 3
figure 3

(a) Simulated sources on the cortical surface with source orientations normal to the cortical sheet. (b) Simulated MEG measurements corresponding to the source configuration of (a). (c) The resulting cortically constrained MNE. (d) The noise sensitivity normalized MNE (dSPM)

Full size image

Continuing with the same simulated source, Fig. 4 demonstrates the effects of incorporating an fMRI weighting. The first row shows the case where the fMRI weighting matches closely to the true source. Consequently, both sources are recovered, and the extra ripples are suppressed. The second row corresponds to the case where the motor cortical activity is not visible in the fMRI. Then, the fMRI weighting also abolishes this source from the MEG inverse solution with the selected MNE threshold. For the last row, we demonstrate a case where an extra activation cluster is present in the fMRI, leading to some false-positive sources at the corresponding fMRI-weighted MNE.

Fig. 4
figure 4

First row: the fMRI-weighted MNE in case where the fMRI prior information is concordant with the MEG sources. Second row: the motor cortical source is missing from the fMRI map. Third row: a superfluous activation cluster is present in the fMRI map near the auditory cortical source

Full size image

Note that for all the cortical images, the threshold was set to be 30% of the maximum amplitude, and the full color scale was used to display the sources or estimates on the cortical surface. This selection is rather arbitrary and has an obvious effect on the visual appearance of the estimates to be, for example, apparently more focal.

3 Discussion

3.1 Developments of the fMRI-Weighted MNE

As shown by simulation examples, the main problem with the simple “fMRI-weighted MEG” is the relatively strong bias toward the fMRI data. Biophysically, we have reasons to expect that the “active” areas detected by MEG and fMRI may be only partially overlapping. On one hand, the temporal synchronization and summation of the neuronal activity on a millisecond scale is crucial for elicitation of a measurable MEG response. On the other hand, fMRI hemodynamic response allows the summation of activity over seconds. For instance, a sharp transient stimulus may produce a response clearly visible in MEG but may not be strong enough to push the vascular system to produce a robust hemodynamic response. On the contrary, weak but asynchronously sustained activity may temporally integrate to a measurable fMRI response but remain undetected by MEG. In addition to temporal summation, MEG and fMRI have different spatial sensitivities: fMRI has no spatial cancellation due to neuronal currents having incoherent orientations as MEG [1], and it has equal sensitivity for detecting activity in gyri and sulci.

Different from using fMRI as a spatial prior, Daunizeau and colleagues propose a symmetric approach for multimodal integration of fMRI and MEG/EEG data by constructing a model where the spatial activation profile in each anatomically defined parcel of the cortex is assumed to be similar in both modalities [11]. Activations, which are not present in both modalities, are modeled as Gaussian residuals, allowing for natural discrepancies between the different types of data. The model becomes computationally rather complex as variables between modalities become spatiotemporally entangled. Although the explicit modeling of coupled and uncoupled sources is appealing from a theoretical viewpoint, it is somewhat questionable how desirable it is to let MEG/EEG data substantially influence the estimated waveform of the fMRI hemodynamic response. Henson and colleagues maintain the asymmetry between modalities but consider more flexible fMRI priors while preserving the basic Gaussian model structure, which renders computations highly tractable [27]. The work expresses the prior covariance as multiple variance/covariance components Ri: R = ΣiλiRi, each with an adaptive weight parameter λi. The covariance components Ri are generated based on fMRI analysis, whereas appropriate values for the covariance component weights \( {\hat{\lambda}}_i \) are estimated from the MEG/EEG data with the parametric empirical Bayesian (PEB) method. Once the prior covariance R is fixed to the PEB estimate \( \hat{\mathbf{R}}={\Sigma}_i{\hat{\lambda}}_i{\mathbf{R}}_i \), the solution for the dipole amplitudes reduces to the fMRI-weighted MNE. The obvious question is then how to partition the fMRI activation map into different variance/covariance components Ri – the limiting cases are that all locations determined active form one diagonal variance component R1, and the inactive form another R0, in the spirit of the original fMRI-weighted MNE (see, Eq. (5)) or that each fMRI-activated location/cluster is assigned to an individual variance component. Moreover, off-diagonal covariance terms in the components can be also introduced: we may have an a priori reason to believe that, for instance, left and right primary auditory cortices should be activated in a similar fashion if identical stimulation is delivered to both auditory pathways. As the number of ways in which the fMRI data can be split into covariance components is rather large, the practical question of how to generate a reasonable prior structure of appropriate complexity remains an important challenge. The fMRI-Informed Regional Estimation (FIRE) method [41] combines elements of the symmetric approach and the automatic relevance determination (ARD) approaches [19, 39, 45, 50]. FIRE also utilizes the anatomical parcellation of the cortex [17] and assumes that both electromagnetic and hemodynamic activity have a common spatial profile at each parcel but independent temporal waveforms. The overall source variance is adaptively estimated for each region, the ARD structure allowing adequate variability of source strengths for each parcel and letting thus non-active regions to be suppressed. Different from the symmetric approach, the FIRE approach assumes that the hemodynamic responses are directly observed. Thus, FIRE is computationally tractable and conceptually simple. The result is that for those regions where clear fMRI responses are detected, the MEG source localization leans on the available fMRI spatial information, and if fMRI information is missing but MEG signals detected, the localization results are similar to the basic MNE.

3.2 Experimental Design, Model Comparison and Validation, and Neurovascular Coupling Models

As mentioned above, different physiological origins of the signals introduce a natural challenge in how to design the experiment such that multimodal data fusion is meaningful. In MEG/EEG, stimuli eliciting transient responses may be optimal for high temporal resolution MEG/EEG measurements, while delivering “continuous” trains of stimuli may be more appropriate to drive the vascular system into a steady state for high contrast-to-noise ratio (CNR) fMRI data. Accordingly, the integrative analysis of the data requires some special attention. The fMRI weighting prior for the MNE, by definition, should encode information that we have about the phenomenon before (a priori) we see the MEG/EEG data. Thus, determining the fMRI weighting for the MNE should be based solely on the fMRI data. Strictly speaking, it is incorrect to tweak the fMRI prior after seeing the results of the fMRI-weighted MNE. However, saying that the (fMRI) prior should not depend on the (MEG/EEG) data to be modeled does not mean that the prior cannot have unknown parameters that are estimated from the data, such as the prior covariance weights λi in the PEB approach [27]. The symmetrical approach [11, 27] and the FIRE approach [41] partially avoid this problem as the prior is mainly fixed by the cortical parcellation and other explicit modeling assumptions about the spatial concordance of the electromagnetic and hemodynamic responses.

If multiple fMRI-MEG/EEG integration methods are applied to a given dataset, what can be said about the validity or accuracy of these models? Can we compare the models and select the most likely one in some sense? There are criteria for Bayesian model comparison and selection such as the model evidence, as utilized in [27], which can be useful for evaluating the model complexity and guiding the model selection process. In principle, due to the MEG/EEG inverse problem, there is no “true” solution that can be singled out by any statistical test – no matter how much MEG/EEG data we collect, or how realistic we make the volume conductor (head) model, there will be multiple source configurations that fit equally well to a given set of MEG/EEG data. Even in the presence of almighty Bayes, the silent sources remain silent. If the hypothetical silent MEG/EEG source configuration is detected by fMRI and incorporated into the spatial prior, we do not gain further information of its electromagnetic characteristics – the product between an inverse operator and zero measurements (due to silent MEG/EEG source) still yields zero source estimates. Obviously poor models can be detected by comparing, e.g., the data fit of a given model against a standard model, such as the MNE, which operates on the minimal assumptions. However, it is unlikely that any model will yield a substantially better data fit for the MEG/EEG data, since the MNE is also an optimal estimator in least squares sense. Some forms of cross-validation might also be applied to test and compare prediction errors and to detect over-fitting.

3.3 Neurovascular Coupling: The Physiological Bases of Integrating fMRI and MEG Source Modeling

A better understanding of neurovascular coupling is of fundamental importance in integrating hemodynamic and neuronal activity data (for review, see [25, 35]). BOLD contrast fMRI has been suggested to be closely related to the input synaptic activity [34] and to neuronal output spiking [24, 37]. When postsynaptic neuronal signals are highly synchronous, they constitute the magnetic fields measured by MEG in guinea pigs [40, 43]. Invasive recordings can directly measure the neurovascular coupling and therefore offer more detailed information for further testing and validating the noninvasive models. Animal models and intracranial measurements in humans will be needed to provide a backbone for the development of noninvasive imaging approach, which will always rest on some weighty modeling assumptions due to the indirect nature of observations. Population-level models of the neurovascular coupling can also provide insights and predictions about the noninvasive data in various circumstances [4, 47].

A linear relationship between the strength of neuronal signal and hemodynamic signal has been suggested by studies in the human visual system [24, 38, 43, 46, 49] and the motor system [42]. However, using a rodent model, a nonlinear relationship between the strength of the local hemodynamic response and neuronal activity has also been reported [12]. Such hemodynamic output may be explained as the spatiotemporal convolution of local electrophysiological responses [13]. A more complicated correlation structure between MEG and BOLD fMRI responses was also found in the human auditory system: the same auditory clicks can elicit transient and sustained MEG responses with the transient response more closely related to the BOLD fMRI signal [21]. Taken together, the mechanism through which the MEG and fMRI signals become coupled, as well as the ensuing degree of observed correlation in the macroscopic responses, remains only partially elucidated.

From the MEG/EEG source modeling perspective, it is highly motivated to explore and exploit the fMRI data as a spatial prior to complement the non-unique nature of estimating neuronal current source distributions using extracranial recordings [26]. Such a data fusion technique has been supported by studies showing reliable correlations between hemodynamic responses and neuronal activity as discussed above: brain areas showing significant hemodynamic responses measured by fMRI are expected to be engaged in corresponding neuronal activity, the synchronous synaptic components of which are measured by MEG. This rationale is also supported by studies showing that MEG and fMRI can co-localize to the same cortical areas in the visual system [7] and motor system [18]. However, it should be noted that the spatial distribution between electrophysiological activity and hemodynamic responses is not in a complete agreement. In the somatosensory area, the distance between the center of fMRI map and the center of the electrophysiological maps can be separated by approximately 1 cm [14]. In summary, further investigation of the neurovascular coupling should provide support for developing the mathematical models of integrating fMRI and MEG data for spatiotemporally sensitive and functionally specific detection of human brain activation.

In conclusion, the field of multimodal integration of noninvasive imaging technologies such as MEG/EEG and fMRI is still in a rather early stage, and the methodology will continue to evolve as more data about the physiological origins of the signals are accumulated.