1. Image restoration by deconvolution
Volker Bäcker
Montpellier Rio Imaging Pierre Travo
IFR3 Giacomo Cavalli Frederic Bantignies Patrice Mollard Nicole Lautrédou-Audouy Jean-Michel Poulin

2. Overview Part 1 Part 2 introduction what is deconvolution ?
how does it work ?
when should it be used ?
Part 2
what are the parameters to know and care about for image restoration by deconvolution?

3. fluorescence microscopy
specimen has to be in focal distance
to image 3d specimen
move focal plane through specimen
creating stack of slides
fluorescence microscopy
specimen marked with dye that emists light of one wavelength while being stimulated by light of another wavelength
Microscope types
widefield whole specimen bathed in light
confocal image is constructed point by point to keep out out-of-focus light
two photon two photons needed to stimulate emission, similar effect as confocal

4. Example: 2d widefield After deconvolution (same levels)
Image from microscope
After deconvolution (same levels)
Immunostaining on whole mount drosophila Embryo
Using an antibody against a nuclear protein

5. Example 3d confocal
Image from microscope
After deconvolution

6. Example: time series 2 photon
Image from microscope
After deconvolution

7. The aquired image is not the „real“ image
Images are degraded due to the limited aperture of the objective
Deconvolution can be used to get an image nearer to the real object
by using knowledge of the imaging process and the properties of the microscope
Deconvolution can be used for all kinds of fluorescence microscope images:
2D, 3D, time series, widefield, confocal, 2 photon
Example of 2D widefield image
before and after deconvolution
example of 3D image befor and after
deconvolution
example of confocal image before and after deconvolution

8. Sources of image degradation
Noise
Blur
Can be handled by image restoration
Scatter
random distribution of light due to heterogenous refrection index within specimen
Glare
random distribution of light that occurs within the optical train of the microscope

9. Causes of image degradation Noise
Geben Sie eine Zusammenfassung der momentanen Situation
What causes the image degradation

10. Causes of image degradation Noise
Where does the noise come from ?
random fluctuations in the signal intensity
variation of the incident photon flux
interfering signals from electronic system of the captor device

11. Causes of image degradation Blur
Before restoration
After restoration

12. Causes of image degradation Blur
Where does the blur come from ?
contributions of out-of-focus light to the imaging plane
diffraction
a result of the interaction of light with matter
diffraction is the bending of light as it passes the edge of an object

13. How does deconvolution work
Image restoration
Get rid of noise
assume random noise with Poisson distribution
remove it
Get rid of blur
Compute real image from sample
by applying a model of how the microscope degraded the image
deconvolution

14. Point Spread Function Point spread function (psf)
Model of how one point is imaged by microscope
Experimental
aquired by taking an image of „point like objects“ - beads
Alternatevely, point like object
present in the acquired image
itself can be usedf.
Theoretical
computed from the microscope and captor parameters

15. Convolution (Faltung)
aquired image = real image convolved with psf
Convolution is an integral that expresses
amount of overlap of functions as g is shifted over f.
N pixel => O(N*N) operations to compute it
i(x) : aquired image
f(x) : object function
g(x) : point spread function

16. Fourier Transform (FT)
Signal can be represented as sum of sinoids
FT transforms from spacial to frequency domain

17. Fourier transform (FT)
Convolution theorem
<=>
i(x) : aquired image
f(x) : object function
g(x) : point spread function
I fourier transform of i
F fourier transform of f
G fourier transform of g *
Object function
psf
Fourier transform (FT) FT
inverse FT
FT can be computed in
O(n * log n)
Object function convolved with psf

18. Deconvolution
<=>
Deconvolution: find object function f for given image i and psf g
Unfortunatly it is not practicable to compute
G has zeros outside certain regions
Setting F zero for these would create artefacts
In practice there is noise
N/G would amplify noise
It's not possible to reconstruct the real object function

19. Deconvolution algorithms
Solution
Find an algorithm that computes a function f' so that
f' estimates f as good as possible
works in the presence of noise
Different deconvolution algorithms exist
In general best for fluorescent microscopy:
(Classical) Maximum Liklihood Estimation - MLE

20. Maximum Likelihood Estimation
Tries to optimise f' iteratively
The basic principal is (but there's more to it)
g(i|j) : psf - the fraction of light from true location j that is observed in pixel i
Fraction of light from pixel j that hit other pixels
Fraction of light from other pixels that hit pixel j
Richardson and Lucy R-L Iteration

21. fraction of light from others
0,3 1
0,1
0,2 A B C D 6 5 3 4
Denominator: get rid of foreign light that hit me 1 C C B3
Numerator realign my light to me
1 C C B3 1 2 3
psf 4
image
5 * [5*1 / (5* * *6) + 0,1*4 /(5* * *6) + 0,2*6/(5* * *6)]
5 * [5 / / /7.4]
fraction of light lost
5 * [( )/7.4]
5 * [6.6 /7.4]
5 *
New estimate
aquired image
last estimate
last estimate
fraction of light from others

22. Summary and conclusions 1
image from microscope is degraded
it contains noise and blur
blur can be described as a convolution of object function and psf
image nearer to the object function can be obtained by image restoration yielding higher resolution and better contrast
MLE is a deconvolution algorithm approriate for fluorescent microscope images
imaging process is not finished finished without deconvolution
do it whenever high quality images are needed

23. Image restoration in practice
Many deconvolution software packages are commercially available
They use various types of deconvolution algorithms
In addition to these algorithms, they might incorporate other imaging tools, such as filters of different kinds. Moreover, different types of algorithms may introduce or not some « assumptions » concerning the image sent to restoration.
In general, it is important to test the software. One basic « rule of thumb » is also that the restoration should respect the acquired image in terms of objects visible and of their relative intensity. Objects « appearing », « disappearing » or changing relative intensity with respect to neighboring structures are diagnostic of problems. These problems might be due to the setting of relevant parameters or, in the worst case, of poor quality of the software

24. Image restoration using the huygens2 software from SVI
- website of Scientific Volume Imaging (SVI)
It is the software used at the
Institute of Human Genetics

25. Relevant parameters in deconvolution Setting Microscope parameters
microscope type
widefield and multipoint confocal
work with ccd camera
single point confocal and two photon
work with photomultiplier
different point spread functions
if you don't know
Ask your imaging facility and look at the specifications of your microscope

26. Microscope parameters
Numerical aperture
measure of ability to gather light and resolve fine specimen detail at a fixed object distance
higher magnification doesn't yield higher resolution, higher NA does
Maximal value written on objective
Can't be larger than the the refractive index n of the medium

27. Sampling theorem
Imaging converts an anlog signal into a digital signal
When converting an analog signal into a digital signal the sampling theorem applies
Nyquist-Shannon sampling theorem “the sampling interval must not be greater than one-half the size of the smallest resolvable feature of the optical image”
sampling at nyquist rate means using exactly this interval
sampling interval is the pixel size in our image

28. Undersampling and oversampling
loss of information
aliasing artefacts
over sampling
higher computation times and storage requirements
longer acquisition times, photobleaching.
under sampling example. An object of a given shape (dashed line) can be interpreted as a different shape (thick line) if too few points are acquired along any of the x,y,z axes

29. Changing the Numerical Aperture (NA) for widefield / two photon
huygens2 allows under/oversampling within a range
at the borders of this range deconvolution can be done but results are not good
In this case better results when “lying” about NA
if sampling size not in range change NA
nyquist sample size

30. Microscope parameters
Excitation and emission wavelength
fluorescent dye absorbs light of one wavelength and emits light of another wavelength
filter cubes are used to ensure that only light of a wanted wavelength passes.
exitation and emission wavelengths depend on the cube used
GFP 473, 525

31. Microscope parameters
The objective magnification used
determines the pixel size in the image
ccd camera
Pixel size = ccd element size / magnification (eventually modified by other parameters)
photomultiplier
pixel size depends on resolution and magnification

32. Microscope parameters
Refractive index n of the objective medium
oil 1,51500
water 1,33810
air 1,00000
Should match the refractive index of the sample medium
Otherwise
Magnification error in axial direction
Spherical aberration (psf deteriorates with increasing depth)

33. Microscope parameters
Cmount factor
adaptor that attaches the camera to the microscope
might contain additional optic that
changes the overall magnification
and therefore the pixel size
value is 1 if no additional optic present

34. Microscope parameters
Tube factor
the tube might contain additional optics to change the tube length
this changes the overall magnification
and therefore the pixel size

35. Microscope parameters
sample medium
refractive index n
default (all media for example water) ,33810
liquid Vectashield (not polymerized) ,49000
90-10 (v:v) glycerol - PBS ph ,49000
prolong antifade ,4
limits the NA and therefore the possible resolution

36. Captor parameters size of the unitary ccd captor
image sensor of the camera
ccd – charge coupled device
diodes that convert light into electrical charge
property of the camera
Coolsnap 6450 nm
Micromax 6700 nm
For photomultiplier
the pixel size is asked
see table in help pages

37. Captor parameters Binning take nxn elements as one
more light per pixel
reduces noise
higher signal to noise ratio
lower resolution

38. Captor parameters in case of XZY in case of time series z step size
time interval

39. Captor parameters in case of confocal pinhole radius pinhole
keep out out of focus light
pinhole either fixed or adjustable
Backprojected radius in nm
Size of pinhole as it appears in the specimen plane
size should match airy disk (2d psf) size
6.66 for LSM510

40. task parameter Style of processing step process image slide by slide
converts stack into time series for processing
converts result back into stack
volume
use 3d information
step combined
do step processing
followed by volume processing with fixed parameters

41. Full restoration parameters
signal/noise ratio
the ratio of signal intensity to noise intensity
high noise case
can be measured in the image
Single photon hit intensity
find low intensity voxels from one photon hit – add values – subtract background
Max voxel value
value of brightest voxel
low noise case
single photon hits can´t be seen
rough guess is sufficient

42. Full restoration parameters
background offset
empty regions should be black
but contain some light in reality
subtract mean background to see object clearly

43. Full restoration parameters
number of iterations
too low
optimal restoration not yet achieved
too high
takes longer to compute
some signal may be removed
Usually between 30-70

44. Summary and conclusions 2
deconvolution should be used to obtain high quality images for all kind of fluorescent microscope images
parameters of the imaging system have to be entered to create a model of the image degradation

45. End of presentation volker.baecker@crbm.cnrs.fr

46. Links participants literature
Montpellier RIO Imaging
IFR3 / CCIPE
IGH
CRIC
literature
Introduction to Fluorescence Microscopy
How does a confocal microscope work?
Two-Photon Fluorescence Microscopy
Deconvolution in Optical Microscopy

47. Links
literature
Diffraction of Light
Image restoration: getting it right
Image Restoration in Fluorescence Microscopy
Image restoration in one- and two-photon microscopy
Introduction to Convolution
An Introduction to Fourier Theory
A Self Contained Introduction to Fourier Transforms
Convolution theorem

48. Links
literature
Three-Dimensional Imaging by Deconvolution Microscopy Article ID meth , available online at on IDEAL
Deconvolution of confocal images of dihydropyridine and ryanodine receptors in developing cardiomyocytes
Maximum likelihood estimation via the ECM algorithm: A general framework
The influence of the background estimation on the superresolution properties of non-linear image restoration algorithms
Numerical Aperture and Resolution
User guide for Huygens Professional and Deconvolution Recipes
Digital Image Sampling Frequency
Filter Cubes
Filters for fluorescence microscopy

49. Links
literature
Immersion Media
How Digital Cameras Work
Pixel Binning
CCD Signal-To-Noise Ratio