1. Computational Biology, Part 21 Biological Imaging I G. Steven Vanni Robert F. Murphy Copyright  1998, 2000, 2001. All rights reserved.

2. Biological imaging Significant advances in the fields of optics and electronics in the past two decades have greatly increased the utility of imaging for addressing biological questions. Significant advances in the fields of optics and electronics in the past two decades have greatly increased the utility of imaging for addressing biological questions. These advances permit These advances permit  more diverse types of information to be extracted from biological specimens  with greater accuracy  and under more demanding conditions. On the following two slides are images demonstrating the capabilities of biological imaging. On the following two slides are images demonstrating the capabilities of biological imaging.

3. Imaging by Robin DeBiaso Timelapse movie of dividing cell

4. Image acquistion and analysis can produce data to test a hypothesis This experiment supports the hypothesis that the motor protein, myosin II, (high concentration shown in red) plays a role in separating daughter cells following cell division. This experiment supports the hypothesis that the motor protein, myosin II, (high concentration shown in red) plays a role in separating daughter cells following cell division. Imaging by Robin DeBiaso

5. Biological specimens present unique challenges and advantages Challenges Challenges  Controlled environmental conditions are required to preserve processes and signals within a biological specimen.  It can be difficult to gain physical access to the desired region of a specimen. Advantages Advantages  Biological specimens present unique opportunities for the use of chemical and molecular biological probes to detect signals.

6. Controlled environmental conditions To image living specimens, stringent environmental conditions must be maintained not only to keep the specimens alive but also to allow reproducible behavior. Such conditions inlcude: To image living specimens, stringent environmental conditions must be maintained not only to keep the specimens alive but also to allow reproducible behavior. Such conditions inlcude:  temperature  partial pressure of specific atmospheric gases  bathing fluid chemistry Specimens may be chemically fixed to preserve them for long periods, but then living processes can not be observed. Specimens may be chemically fixed to preserve them for long periods, but then living processes can not be observed.

7. Physical and optical accessibility High magnification (40 to 200X) is often desirable, and this sets limits on how deeply into the specimen images may be acquired. High magnification (40 to 200X) is often desirable, and this sets limits on how deeply into the specimen images may be acquired. Typical limits range from 1 mm to 0.1 mm. Typical limits range from 1 mm to 0.1 mm. Given such limits, specimens must be prepared in ways which allow the optics of the microscope to closely approach the area of interest within the specimen. Given such limits, specimens must be prepared in ways which allow the optics of the microscope to closely approach the area of interest within the specimen. Additionally, some specimens absorb or scatter the signal being detected. Additionally, some specimens absorb or scatter the signal being detected.

8. Imaging relies on generating a detectable signal which can be used as a measure of a property of interest in the specimen. Imaging relies on generating a detectable signal which can be used as a measure of a property of interest in the specimen. This property of interest is the initial signal, but it must be transduced or changed through several forms before it becomes detectable. This property of interest is the initial signal, but it must be transduced or changed through several forms before it becomes detectable. Chemical and molecular biological probes may be targeted within a specimen

9. For example: A protein may be modified so that when it enters a cell and bumps into another protein involved in a specific activity, it fluoresces. The original activity was probably not detectable, but this newly generated fluorescence signal is detectable. For example: A protein may be modified so that when it enters a cell and bumps into another protein involved in a specific activity, it fluoresces. The original activity was probably not detectable, but this newly generated fluorescence signal is detectable. Front end of imaging system and detector Specimen Specimen may be difficult to see except where labeled by probe. Chemical and molecular biological probes may be targeted within a specimen

10. Image Formation and Acquisition Having an understanding of the specimen, the next step is the formation and acquisition of a digital image Having an understanding of the specimen, the next step is the formation and acquisition of a digital image A two dimensional image plane consists of a rectangular grid of points, or pixels A two dimensional image plane consists of a rectangular grid of points, or pixels Grid Specimen Pixel

11. A digital image plane is acquired by recording a digital value proportional to the intensity of light (or other form of energy) impinging on each pixel of a detector A digital image plane is acquired by recording a digital value proportional to the intensity of light (or other form of energy) impinging on each pixel of a detector This intensity usually corresponds to the amount of light emitted by or reflected from a corresponding point on a specimen This intensity usually corresponds to the amount of light emitted by or reflected from a corresponding point on a specimen 0002100 0001000 0378300 0688820 0288840 0048830 Projection of specimen onto dectector grid Image Specimen Image Formation and Acquisition

12. “Pixel” is used interchangeably to mean: One of multiple regions on a detector, each corresponding to the smallest area from which a signal can be distinguished One of multiple regions on a detector, each corresponding to the smallest area from which a signal can be distinguished The numerical value associated with each such region in a digital image The numerical value associated with each such region in a digital image A region on display device, such as a monitor or printer A region on display device, such as a monitor or printer 0002100 0001000 0378300 0688820 0288840 0048830 Dectector gridPixel valuesDisplay device 7-8 4-6 1-3 0 Pixel

13. Display of pixel values A pixel value is just a number in the data set representing a digital image. A pixel value is just a number in the data set representing a digital image. Pixel values may be displayed in different ways, determined by a look up table (LUT). Pixel values may be displayed in different ways, determined by a look up table (LUT). 0002100 0001000 0378300 0688820 0288840 0048830 Pixel valuesHot to cold 7-8 4-6 1-3 0 Arbitrary 7-8 4-6 1-3 0 Binary 1-8 0 LUT

14. Image Formation Biological images may be acquired via a variety of imaging modes or modalities Biological images may be acquired via a variety of imaging modes or modalities Each mode is a combination of an image formation system and a detector Each mode is a combination of an image formation system and a detector

15. Image formation system Sample Image formation system Detector

16. Optical signal transduction The image formation system further transduces the signal emanating from the specimen using optical and electronic elements. The image formation system further transduces the signal emanating from the specimen using optical and electronic elements. For example: Membranes separating cellular compartments interact with light to change the light in ways not detectable by eye. Thus, an invisible signal describing cellular organization is hidden in the light. Special optical elements and electronics transduce this signal to create intensity variation in the light which is detectable by eye. For example: Membranes separating cellular compartments interact with light to change the light in ways not detectable by eye. Thus, an invisible signal describing cellular organization is hidden in the light. Special optical elements and electronics transduce this signal to create intensity variation in the light which is detectable by eye.

17. Any image is only a partial recording It is important to consider what is or is not being recorded. It is important to consider what is or is not being recorded. Quality science relies on a careful understanding of the data from which conclusions are drawn. Quality science relies on a careful understanding of the data from which conclusions are drawn. Any image, digital or not, is an incomplete recording of a real world specimen Any image, digital or not, is an incomplete recording of a real world specimen It is incomplete because it records only one channel of the available information as determined by It is incomplete because it records only one channel of the available information as determined by  specimen preparation  selection of imaging system components  detector type

18. Any image is a partial recording Examples of different channels include: Examples of different channels include:  visible light modified by cellular morphology  visible light modified by the proximity to which a cell adheres to an underlying substratum  fluorescence emanating from an activated protein  a second wavelength of fluorescence from a different protein or from a chemical mechanism related to pH And each channel will produce measurements which more or less accurately address the questions being considered And each channel will produce measurements which more or less accurately address the questions being considered

19. Detector and image types While the examples so far have dealt with light microscope images, we will now back up for a microscope images, we will now back up for a few minutes to consider many different types few minutes to consider many different types of images before concentrating again on light of images before concentrating again on light microscopy. microscopy.

20. Detector and image types In general, images may be classified according to what is being detected: (Visible) light transmission, scattering or emission (Visible) light transmission, scattering or emission  single wavelength, 3 color, or full spectrum Electron transmission or scattering Electron transmission or scattering X-ray transmission X-ray transmission Radioactive particle emission Radioactive particle emission Magnetic field perturbation Magnetic field perturbation Physical displacement from “atomic force” Physical displacement from “atomic force”

21. Comparing types of imaging

22. Light microscopy Key concepts are filtration & detection from small specimens (1 cm to 1 um)  Optical elements filter complex light waves from specimen to generate and detect only the signal of interest, for example:  rhodamine emission at 560 nm to locate probes applied to specimento locate probes applied to specimen  phase shifted waves to locate membrane boundaries within specimento locate membrane boundaries within specimen  waves scattered from a specified angle to determine surface topologyto determine surface topology

23. Light Light is energy which travels through space Light is energy which travels through space  It is made of travelling particles or waves This is of interest in two ways: This is of interest in two ways:  (1) in transmitted light microscopy  (2) in fluorescence (emitted) light microscopy

24. Light has properties which are modified as it passes through a specimen. Light has properties which are modified as it passes through a specimen. In transmitted light microscopy, light enters the specimen, is modified by the specimen and then passes out and may be detected. In transmitted light microscopy, light enters the specimen, is modified by the specimen and then passes out and may be detected. Thus if we know how it has been modified, we can infer something about the specimen. Thus if we know how it has been modified, we can infer something about the specimen. Light

25. Properties of light Wavelength (inverse of frequency) Wavelength (inverse of frequency) Direction of travel Direction of travel Phase Phase  Constructive and destructive interference Polarization Polarization Intensity Intensity  Ultimate signal source  Selected filtration will cause intensity to vary depending on any of above properties

26. Light In both transmitted and fluorescence light microscopy, light exits different regions of the specimen. In both transmitted and fluorescence light microscopy, light exits different regions of the specimen. Because of the small wavelengths of light (< 1 um ), it is possible to resolve fine detail in a living specimen. Because of the small wavelengths of light (< 1 um ), it is possible to resolve fine detail in a living specimen.

27. Light microscopy The term “Filter” is used very generally. The term “Filter” is used very generally.  Typically “filter” means to pass only certain wavelengths  Thus allowing us to distinguish rhodamine, fluorescein and white light signals (examples)  But more general types of filtering, allow us to distinguish many changes in the properties of the light passing through a specimen

28. Light microscopy To “look at” means to detect at a level discernible from background To “look at” means to detect at a level discernible from background Three primary types of detectors Three primary types of detectors  human eye  no digital image  CCD or charge coupled device  “work horse” of modern biological imaging  acquires digital image directly  PMT or photomultiplier tube  scans to produce digital image

29. CCD cameras CCD cameras cost anywhere from $500 to $30,000 depending on their sensitivity. Courtesy of Phometrics, LTD.

30. CCD chips Penny A CCD chip is the actual detector within a CCD camera.

31. Light sources in the object Consider a fluorescent specimen made of individual molecules of fluorescent dye. Consider a fluorescent specimen made of individual molecules of fluorescent dye.  Each molecule can emit light.  Each dye molecule may be seen as a vanishingly small emitter.  Such an emitter is called a point source.  The concept of a point source is useful because a point is simple to model, and if we know how a point source is imaged, then we can easily model a complex specimen as a combination of many points and predict how it will be imaged.

32. Light sources in the object A specific example might be a microscope slide containing cells stained with fluorescent dye. A specific example might be a microscope slide containing cells stained with fluorescent dye. In an ideal image, a point source would show intensity in only one pixel In an ideal image, a point source would show intensity in only one pixel

33. Idealized Image of Point Source

34. In reality, the light from each point in the specimen is seen to spread out and affect many pixels in the image. In reality, the light from each point in the specimen is seen to spread out and affect many pixels in the image. This “spreading” is an inescapable result of the optical properties of the image formation system. This “spreading” is an inescapable result of the optical properties of the image formation system. The mathematical description of this spreading or blurring process is called a point-spread function (PSF) The mathematical description of this spreading or blurring process is called a point-spread function (PSF) Point-spread function

35. The point-spread function (PSF) is determined by the optics of the image formation system, including factors such as the refractive index, diameter and magnification of its components The point-spread function (PSF) is determined by the optics of the image formation system, including factors such as the refractive index, diameter and magnification of its components The resulting blurred region in the image can be approximated by a 2D Gaussian distribution The resulting blurred region in the image can be approximated by a 2D Gaussian distribution

36. Realistic Image of a Point Source This graph shows intensity on the z-axis for a PSF defined in the X-Y plane. This graph shows intensity on the z-axis for a PSF defined in the X-Y plane. Later we will consider a PSF defined in three dimensions. Later we will consider a PSF defined in three dimensions.

37. Light sources in the object Thus, when a 2D image is acquired, each point in the specimen will be blurred in all directions and will contribute to the recording in many pixels around that pixel to which it directly corresponds Thus, when a 2D image is acquired, each point in the specimen will be blurred in all directions and will contribute to the recording in many pixels around that pixel to which it directly corresponds

38. Introduction to 3D Microscopy The spreading of light from a point source actually occurs in three dimensions as will be shown. The spreading of light from a point source actually occurs in three dimensions as will be shown. First, however, it is necessary to understand the three dimensional (3D) nature of the object and image as acquired via 3D microscopy. First, however, it is necessary to understand the three dimensional (3D) nature of the object and image as acquired via 3D microscopy.

39. When a microscope is focused on a specimen, the detector records an image from a plane. When a microscope is focused on a specimen, the detector records an image from a plane.  This is the focal plane.  Parts of the specimen in the focal plane are in the best focus. Detector Focalplane 3D Microscopy

40. 3D data is acquired by combining data from several different focal planes into a stack of images. 3D data is acquired by combining data from several different focal planes into a stack of images. This is accomplished by changing the distance between the specimen and the microscope’s objective lens from one image acquisition to the next. This is accomplished by changing the distance between the specimen and the microscope’s objective lens from one image acquisition to the next. Objective Imagestack

41. The next slide shows a real 3D image stack. The next slide shows a real 3D image stack. The specimen is a slice from a fruit fly eye which labeled with a photoreceptor specific antibody conjugated to a fluorescent dye. The specimen is a slice from a fruit fly eye which labeled with a photoreceptor specific antibody conjugated to a fluorescent dye. The images were acquired using a conventional fluorescence microscope. The images were acquired using a conventional fluorescence microscope. The image stack is presented here as a movie with one acquired image plane per movie frame. The image stack is presented here as a movie with one acquired image plane per movie frame. Real 3D image data

42. Fruit fly photoreceptor cell axons Courtesy of Dr. John Pollock

43. Now, with a better understanding of what makes up a 3D image stack, we can better consider how light from a point source spreads out and is imaged in three dimensions. Now, with a better understanding of what makes up a 3D image stack, we can better consider how light from a point source spreads out and is imaged in three dimensions. On the following slide, intensity is shown by variation in color. On the following slide, intensity is shown by variation in color.  Warm colors indicate greater intensity.  All axes indicate real spatial dimensions as indicated. Real 3D image of a point source

44. yz xx Courtesy of Image & Graphics Inc.: http://www.imagepro.co.kr/ 3D Reconstruction of Point Spread Function (PSF) from 0.2 Micron Bead NOTE: Spreading along the Z-axis is more pronounced. Increasing intensity

45. Image Formation Image formation can be described as: Image formation can be described as:  the convolution of an array describing the original specimen or object  with a function describing the image formation system  to yield an acquired image.

46. Image Formation The mathematical view of convolution emphasizes that each point in the sample can contribute to each point in the image The mathematical view of convolution emphasizes that each point in the sample can contribute to each point in the image

47. There is a mathematical concept which works well to describe how each point in the specimen or object contributes to each point in the image. There is a mathematical concept which works well to describe how each point in the specimen or object contributes to each point in the image. This concept is called a convolution and what follows is a graphic description. This concept is called a convolution and what follows is a graphic description. z’ x’ y’ z x y A few points in the object. One example of a point in the image The concept of a convolution

48. A convolution may be written in somewhat simplified mathematical form as follows: A convolution may be written in somewhat simplified mathematical form as follows: The concept of a convolution i(x,y,z) defines the image in its 3D space according to the form of the equation above. i(x,y,z) defines the image in its 3D space according to the form of the equation above. o(x’,y’,z’) describes the specimen or object. o(x’,y’,z’) describes the specimen or object.

49. In order to permit a graphic description of convolution, the object and the image are superimposed onto the same graph. In order to permit a graphic description of convolution, the object and the image are superimposed onto the same graph. The concept of a convolution z, z’ x, x’ y, y’

50. Considering again the PSF, each object point would ideally only contribute to one image point, but actually is detected as being more spread out. Considering again the PSF, each object point would ideally only contribute to one image point, but actually is detected as being more spread out. The concept of a convolution z, z’ x, x’ y, y’ z x Point Spread Function (PSF)

51. The concept of a convolution By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. z, z’ x, x’ y, y’

52. The concept of a convolution By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. This summing of contributions is a convolution as described in the above equation. This summing of contributions is a convolution as described in the above equation. z, z’ x, x’ y, y’

53. Convolution and the PSF By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. This summing of contributions is a convolution as described in the above equation. This summing of contributions is a convolution as described in the above equation. z, z’ x, x’ y, y’ Now we consider the parameters of the PSF. Now we consider the parameters of the PSF.

54. Convolution and the PSF By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. This summing of contributions is a convolution as described in the above equation. This summing of contributions is a convolution as described in the above equation. z, z’ x, x’ y, y’ The PSF passes less contribution the further separated the points are. Thus, it is a filter. The PSF passes less contribution the further separated the points are. Thus, it is a filter.

55. Convolution and the PSF By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. By integration, a red point in the image i is defined by the sum of the contributions from all green points in the object o. This summing of contributions is a convolution as described in the above equation. This summing of contributions is a convolution as described in the above equation. z, z’ x, x’ y, y’ The PSF passes less contribution the further separated the points are. Thus, it is a filter. The PSF passes less contribution the further separated the points are. Thus, it is a filter. Ideally the contributions would fall off rapidly with increasing separation. Ideally the contributions would fall off rapidly with increasing separation.

56. Image Formats There are two general types of image formats. There are two general types of image formats. The format we have been and will continue to concentrate on is the bit map image composed of pixels filling the image space. The format we have been and will continue to concentrate on is the bit map image composed of pixels filling the image space. An alternative type of format is the vector image composed of lines or vectors which are defined only where objects exist in the image space. An alternative type of format is the vector image composed of lines or vectors which are defined only where objects exist in the image space. Bit map image Vector image

57. Image Formats An bit map image normally consists of an 8- bit or 16-bit value for each pixel. An bit map image normally consists of an 8- bit or 16-bit value for each pixel. These values are stored as computer files in various formats. These values are stored as computer files in various formats. Pixel values are normally stored linearly in a file with the values for the first row of pixels followed immediately by the values for the second row (etc.). Pixel values are normally stored linearly in a file with the values for the first row of pixels followed immediately by the values for the second row (etc.).

58. Image Formats At a minimum, an image format contains: At a minimum, an image format contains:  Image size (# of rows and columns)  Number of bits per pixel  Order in which bytes within words are stored  Number of bytes to skip at the beginning of the image (the offset)  The beginning of image files often has a text header that can be skipped if the above values are known.  This header may contain additional descriptive information about the image such as: subject of imagesubject of image name of person and/or application creating the imagename of person and/or application creating the image

59. Common Image File Formats PICT PICT  Originally for MacDraw  Used primarily by Macintosh programs  Default format for NIH Image  Readable by Simpletext, Word Reference: www.shortcourses.com/chapter07.htm

60. Common Image File Formats TIFF (Tag Image File Format) TIFF (Tag Image File Format)  Originally for scanners and frame grabbers  Used extensively on many platforms  Can be read/written by NIH Image  Supports lossless compression Reference: www.shortcourses.com/chapter07.htm

61. Common Image File Formats JPEG (“jay-peg” Joint Photographic Experts Group) JPEG (“jay-peg” Joint Photographic Experts Group)  Originally referred to a compression method but now refers to the associated file format with or without compression  Most common World Wide Web file format  Supports progressive display where an image is first displayed at low resolution and then at higher resolution.  Uses a lossy compression technique  Optimized for storing photographs and not as good for line art  Supports 24-bit color Reference: www.shortcourses.com/chapter07.htm

62. Common Image File Formats GIF (“jiff” Graphics Interchange Format) GIF (“jiff” Graphics Interchange Format)  Also widely used on the Web  Supports progressive display  Mostly used for line art as opposed to photographs  Only supports 8-bit color Reference: www.shortcourses.com/chapter07.htm

63. Image Display and Processing Next class, we will consider image display and processing. Next class, we will consider image display and processing. NIH Image is a free program used for image acquisition, display and processing. NIH Image is a free program used for image acquisition, display and processing.