1. Generative Models of M/EEG:
Group inversion and MEG+EEG+fMRI multimodal integration
Rik Henson
(with much input from Karl Friston)

2. Overview A Generative Model of M/EEG
Group inversion (optimising priors across subjects)
Multimodal integration:
3.1 Symmetric integration (fusion) of MEG + EEG
3.2 Asymmetric integration of M/EEG + fMRI
3.3 Full fusion of M/EEG + fMRI?

3. 1. A PEB Framework for MEG/EEG (Bayesian inference)
(Linear) Forward Model for MEG/EEG (for one timepoint):
Y = Data n sensors
J = Sources p>>n sources
L = Leadfields n sensors x p sources
E = Error n sensors
(Gaussian) Likelihood:
C(e) = n x n Sensor (error) covariance
Prior:
C(j) = p x p Source (prior) covariance
Posterior:
Phillips et al (2005), Neuroimage

4. 1. A PEB Framework for MEG/EEG (Covariance Components/Priors)
Specifying (co)variance components (priors/regularisation):
C = Sensor/Source covariance
Q = Covariance components
λ = Hyper-parameters
1. Sensor components, (error):
# sensors
# sensors
“IID” (white noise):
Empty-room:
2. Source components, (priors/regularisation):
# sources
# sources
Multiple Sparse
Priors (MSP):
“IID” (min norm):
Friston et al (2008) Neuroimage

5. 1. A PEB Framework for MEG/EEG (Hyperpriors)
When multiple Q’s are correlated, estimation of hyperparameters λ can be difficult (eg local maxima), and they can become negative (improper for covariances)
Prestim Baseline
Anti-Averaging
Smoothness
Depth-Weighting
Sensor priors
projected to sensors
Source priors
To overcome this, one can:
1) impose positivity on hyperparameters:
2) impose weak, shrinkage hyperpriors:
uninformative priors are then “turned-off” (cf. “Automatic Relevance Detection”)
Henson et al (2007) Neuroimage

6. 1. A PEB Framework for MEG/EEG (Generative Model)
Source and sensor space
Fixed
Variable
Data
Friston et al (2008) Neuroimage

7. 1. A PEB Framework for MEG/EEG (Estimation/Inversion)
1. Obtain Restricted Maximum Likelihood (ReML) estimates of the hyperparameters (λ) by maximising the variational “free energy” (F):
2. Obtain Maximum A Posteriori (MAP) estimates of parameters (sources, J):
cf. Tikhonov
3. Maximal F approximates Bayesian (log) “model evidence” for a model, m:
Accuracy
Complexity
(…where and are the posterior mean and covariance of hyperparameters)
Friston et al (2002) Neuroimage

8. 1. A PEB Framework for MEG/EEG
Summary:
Automatically “regularises” in principled fashion…
…allows for multiple constraints (priors)…
…to the extent that multiple (100’s) of sparse priors possible…
…(or multiple error components or multiple fMRI priors)…
…furnishes estimates of model evidence, so can compare constraints

9. Overview A Generative Model of M/EEG
Group inversion (optimising priors across subjects)
Multimodal integration:
3.1 Symmetric integration (fusion) of MEG + EEG
3.2 Asymmetric integration of MEG + fMRI
3.3 Full fusion of MEG/EEG + fMRI?

10. 2. Group Inversion (Covariance Components)
Specifying (co)variance components (priors/regularisation):
C = Sensor/Source covariance
Q = Covariance components
λ = Hyper-parameters
1. Sensor components, (error):
# sensors
# sensors
“IID” (white noise):
Empty-room:
2. Source components, (priors/regularisation):
# sources
# sources
Multiple Sparse
Priors (MSP):
“IID” (min norm):
Friston et al (2008) Neuroimage

11. 2. Group Inversion (Covariance Components)
Specifying (co)variance components (priors/regularisation):
C = Sensor/Source covariance
Q = Covariance components
λ = Hyper-parameters
1. Sensor components, (error):
# sensors
# sensors
“IID” (white noise):
Empty-room:
2. Optimise Multiple Sparse Priors by pooling across participants
# sources
Litvak & Friston (2008) Neuroimage

12. 2. Group Inversion (one subject) (Generative Model)
Source and sensor space
Fixed
Variable
Data
Litvak & Friston (2008) Neuroimage

13. 2. Group Inversion (multiple subjects) (Generative Model)
Source and sensor space
Fixed
Variable
Data
Litvak & Friston (2008) Neuroimage

14. Litvak & Friston (2008) Neuroimage
2. Group Inversion (multiple subjects) (Re-referencing leadfield matrices)
Concatenate data across subjects
…having projected to an “average” leadfield matrix
Common source-level priors:
Subject-specific sensor-level priors:
Litvak & Friston (2008) Neuroimage

15. 2. Group Inversion (Generative Model)
MMN
MSP
MSP (Group)
Litvak & Friston (2008) Neuroimage

16. 2. Group Inversion (Generative Model)
MMN + 3 fMRI priors
MMN + 3 fMRI priors (Group)
Henson et al (submitted) Frontiers

17. Overview A Generative Model of M/EEG
Group inversion (optimising priors across subjects)
Multimodal integration:
3.1 Symmetric integration (fusion) of MEG + EEG
3.2 Asymmetric integration of MEG + fMRI
3.3 Full fusion of MEG/EEG + fMRI?

18. 3. Types of Multimodal Integration
“Neural”
Activity
Causes (hidden):
(inversion)
Generative (Forward)
Models:
Balloon
Model
Head
Model
Head
Model ?
Data:
fMRI
MEG
EEG
? (future)
Henson (2010) Biomag

19. 3. Types of Multimodal Integration
“Neural”
Activity
Causes (hidden):
Symmetric
Integration
(Fusion)
Generative (Forward)
Models:
Balloon
Model
Head
Model
Head
Model ?
Data:
fMRI
MEG
EEG
? (future)
Asymmetric
Integration
Daunizeau et al (2007), Neuroimage

20. 3.1 Fusion of MEG+EEG (Sensor Components)
Specifying (co)variance components (priors/regularisation):
C = Sensor/Source covariance
Q = Covariance components
λ = Hyper-parameters
1. Sensor components, (error):
# sensors
# sensors
“IID” (white noise):
Empty-room:
2. Source components, (priors/regularisation):
# sources
# sources
Multiple Sparse
Priors (MSP):
“IID” (min norm):
Friston et al (2008) Neuroimage

21. 3.1 Fusion of MEG+EEG (Sensor Components)
Specifying (co)variance components (priors/regularisation):
Ci(e) = Sensor error covariance for ith modality
Qij = jth component for ith modality
λij = Hyper-parameters
1. Sensor components, (error):
# sensors
# sensors
E.g, white noise for 2 modalities:
2. Source components, (priors/regularisation):
# sources
# sources
Multiple Sparse
Priors (MSP):
“IID” (min norm):
Henson et al (2009) Neuroimage

22. 3.1 Basic Model for MEG or EEG (Generative Model)
Source and sensor space
Fixed
Variable
Data
Henson et al (2009) Neuroimage

23. 3.1 Fusion of MEG+EEG (Generative Model)
Source and sensor space
Fixed
Variable
Data
Henson et al (2009) Neuroimage

24. 3.1 Fusion of MEG+EEG (Theory)
Stack data and leadfields for d modalities:
(note: common sources and source priors, but separate error components)
Where data / leadfields scaled to have same average / predicted variance:
mi = Number of spatial modes
(e.g, channels)
Henson et al (2009) Neuroimage

25. 3.1 Fusion of MEG+EEG (Application)
ERs from 12 subjects for 3 simultaneously-acquired Neuromag sensor-types:
Magnetometers
(MEG, 102)
(Planar) Gradiometers
(MEG, 204)
Electrodes
(EEG, 70) fT μV
RMS fT/m
Faces
Scrambled ms ms ms
Faces - Scrambled ms
Henson et al (2009) Neuroimage

26. 3.1 Fusion of MEG+EEG Henson et al (2009) Neuroimage
MEG mags
MEG grads
Faces
Scrambled
Faces – Scrambled, ms
EEG
FUSED
IID noise for each modality; common MSP for sources
Henson et al (2009) Neuroimage
(fixed number of spatial+temporal modes)

27. 3.1 Fusion of MEG+EEG (Conclusions)
Fusing magnetometers, gradiometers and EEG increased the conditional precision of the source estimates relative to inverting any one modality alone
(when equating number of spatial+temporal modes)
The maximal sources recovered from fusion were a plausible combination of the ventral temporal sources recovered by MEG and the lateral temporal sources recovered by EEG
(Simulations show the relative scaling of mags and grads agrees with empty-room data)
Henson et al (2009) Neuroimage

28. 3.2 Asymmetric Integration of M/EEG+fMRI
Specifying (co)variance components (priors/regularisation):
C = Sensor/Source covariance
Q = Covariance components
λ = Hyper-parameters
1. Sensor components, (error):
# sensors
# sensors
“IID” (white noise):
Empty-room:
2. Source components, (priors/regularisation):
# sources
# sources
Multiple Sparse
Priors (MSP):
“IID” (min norm):
Friston et al (2008) Neuroimage

29. 3.2 Asymmetric Integration of M/EEG+fMRI
Specifying (co)variance components (priors/regularisation):
C = Sensor/Source covariance
Q = Covariance components
λ = Hyper-parameters
1. Sensor components, (error):
# sensors
# sensors
“IID” (white noise):
Empty-room:
2. Each suprathreshold fMRI cluster becomes a separate prior
# sources
“IID” (min norm):
fMRI Priors:
# sources
# sources
Henson et al (2010) Hum. Brain Map.

30. 3.2 Basic model for MEG or EEG (Generative Model)
Source and sensor space
Fixed
Variable
Data
Friston et al (2008) Neuroimage

31. 3.2 Asymmetric Integration of M/EEG+fMRI (Generative Model)
Source and sensor space
Fixed
Variable
Data
Henson et al (2010) Hum. Brain Map.

32. 3.2 Integration of M/EEG+fMRI (Priors)
T1-weighted MRI
{T,F,Z}-SPM
Anatomical data
Functional data …
1. Thresholding and connected component labelling
Gray matter segmentation
Cortical surface extraction …
2. Projection onto the cortical surface using the Voronoï diagram …
3D geodesic Voronoï diagram
3. Prior covariance components
Henson et al (2010) Hum. Brain Map.

33. 3.2 Integration of M/EEG+fMRI (Application)1 2
SPM{F} for faces versus scrambled faces,
15 voxels, p<.05 FWE 3 4 5
5 clusters from SPM of fMRI data from separate group of (18) subjects in MNI space
Henson et al (2010) Hum. Brain Map.

34. 3.2 Fusion of MEG+fMRI (Application)
Magnetometers (MEG) * * * *
Gradiometers (MEG)
Negative Free Energy (a.u.)
(model evidence) * * * *
Electrodes (EEG) * * *
None Global Local (Valid) Local (Invalid) Valid+Invalid
(binarised, variance priors)
Henson et al (2010) Hum. Brain Map.

35. 3.2 Fusion of MEG+fMRI (Application)
Magnetometers (MEG) * * * *
Gradiometers (MEG)
Negative Free Energy (a.u.)
(model evidence) * * * *
Electrodes (EEG) * * *
None Global Local (Valid) Local (Invalid) Valid+Invalid
(binarised, variance priors)
Henson et al (2010) Hum. Brain Map.

36. 3.2 Fusion of MEG+fMRI (Application)
Magnetometers (MEG) * * * *
Gradiometers (MEG)
Negative Free Energy (a.u.)
(model evidence) * * * *
Electrodes (EEG) * * *
None Global Local (Valid) Local (Invalid) Valid+Invalid
(binarised, variance priors)
Henson et al (2010) Hum. Brain Map.

37. 3.2 Fusion of MEG+fMRI (Application)
Magnetometers (MEG) * * * *
Gradiometers (MEG)
Negative Free Energy (a.u.)
(model evidence) * * * *
Electrodes (EEG) * * *
None Global Local (Valid) Local (Invalid) Valid+Invalid
(binarised, variance priors)
Henson et al (2010) Hum. Brain Map.

38. 3.2 Fusion of MEG+fMRI (Application)
Magnetometers (MEG) * * * *
Gradiometers (MEG)
Negative Free Energy (a.u.)
(model evidence) * * * *
Electrodes (EEG) * * *
None Global Local (Valid) Local (Invalid) Valid+Invalid
(binarised, variance priors)
Henson et al (2010) Hum. Brain Map.

39. 3.2 Fusion of MEG+fMRI (Application)
IID sources and IID noise (L2 MNM)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
None Global Local (Valid) Local (Invalid)
Henson et al (2010) Hum. Brain Map.

40. 3.2 Fusion of MEG+fMRI (Application)
IID sources and IID noise (L2 MNM)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
None Global Local (Valid) Local (Invalid)
Henson et al (2010) Hum. Brain Map.

41. 3.2 Fusion of MEG+fMRI (Application)
IID sources and IID noise (L2 MNM)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
None Global Local (Valid) Local (Invalid)
fMRI priors counteract superficial bias of L2-norm
Henson et al (2010) Hum. Brain Map.

42. 3.2 Fusion of MEG+fMRI (Application)
IID sources and IID noise (L2 MNM)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
None Global Local (Valid) Local (Invalid)
fMRI priors counteract superficial bias of L2-norm
Henson et al (2010) Hum. Brain Map.

43. 3.2 Fusion of MEG+fMRI (Application)
Right Posterior Fusiform (rPF) Right Medial Fusiform (rMF) Right Lateral Fusiform (rLF)
Differential Response
(Faces vs Scrambled) R
Left occipital pole (lOP)
Differential Response
(Faces vs Scrambled)
Gradiometers (MEG)
(5 Local Valid Priors)
Left Lateral Fusiform (lLF) L
Differential Response
(Faces vs Scrambled)
NB: Priors affect variance, not precise timecourse…
Henson et al (2010) Hum. Brain Map.
Prior 4.
Prior 5.

44. 3.2 Fusion of MEG+fMRI (Conclusions)
Adding a single, global fMRI prior increases model evidence
Adding multiple valid priors increases model evidence further
Helpful if some fMRI regions produce no MEG/EEG signal (or arise from neural activity at different times)
Adding invalid priors does not necessarily increase model evidence, particularly in conjunction with valid priors
Can counteract superficial bias of, e.g, minimum-norm
Affects variance but not not precise timecourse
Henson et al (2010) Hum. Brain Map.

45. 3.3 Fusion of fMRI and MEG/EEG?
“Neural”
Activity
Causes (hidden):
Fusion of fMRI + MEG/EEG?
Balloon
Model
Head
Model
Head
Model ?
Data:
fMRI
MEG
EEG
? (future)
Henson (2010) Biomag

46. 3.3 Fusion of fMRI and MEG/EEG?
Source and sensor space
Fixed
Variable
Data
Henson (submitted) Frontiers

47. 3.3 Fusion of fMRI and MEG/EEG?
Source and sensor space
Fixed
Variable
Data
Henson (submitted) Frontiers

48. Overall Conclusions The PEB (in SPM8) framework is advantageous
Group optimisation of MSPs can be advantageous
Full fusion of MEG and EEG is advantageous
Using fMRI as (spatial) priors on M/EEG is advantageous
Unclear that fusion of fMRI and M/EEG is advantageous

49. The End

50. 3.2. Fusion of MEG+fMRI fMRI hyperparameters Henson et al (2010)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
ln(λ)+32
Participant
fMRI hyperparameters
Local
Valid
ln(λ)+32
Participant
Local
Invalid
Henson et al (2010)
Prior 4.
Prior 5.