1. The information contained in this document pertains to software products and services that are subject to the controls of the Export Administration Regulations (EAR). The recipient is responsible for ensuring compliance to all applicable U.S. Export Control laws and regulations. Ron Kneusel ITT Visual Information Solutions Development of Advanced Algorithms for AstroMed Applications AstroMed 2009 Sydney

2. Visual Information Solutions Outline Image enhancement via the generalized histogram Curve fitting via Particle Swarm Optimization

3. Visual Information Solutions Generalized Histogram Enhancement Motivation Locally adaptive histogram equalization often results in “unnatural” images in which contrast is over enhanced. Key Concepts Image enhancement can be based on a histogram built using non- integer values. The value added to the histogram from a particular pixel is based on the local environment of that pixel. “Image contrast enhancement based on the generalized histogram” B. Yoon, W. Song, Journal of Electronic Imaging, Vol 16(3), 2007

4. Visual Information Solutions Generalized Histogram Enhancement y = f(x) f ij (m) =  w ij (k), m = 0,1,…, Q-1 ; k = 0,1,…,m w ij (k) = h ij (k) / N ij, k = 0,1,…,Q-1 h ij (k) =   (x uv,k), k = 0,1,…,Q-1 ; u,v  A ij But, we can replace h ij (k) with g ij (k): g ij (k) =  (r uv  (x uv,k) + (1-r uv )/Q), k = 0,1,…,Q-1 Mapping function Contrast gain function Generalized histogram Standard histogram

5. Visual Information Solutions Generalized Histogram Enhancement where r uv is the fractional count for the pixel at (u,v) r uv =  (1 – e -  s uv ) s uv = | x nm – x uv | / x, x = mean intensity in 3x3 region Effectively,  affects brightness,  = 0,…,1  affects contrast,  = 0,… g ij (k) =  (r uv  (x uv,k) + (1-r uv )/Q), k = 0,1,…,Q-1

6. Visual Information Solutions Generalized Histogram Enhancement Original Global Contrast Enhanced  = 1,  = 300

7. Visual Information Solutions Generalized Histogram Enhancement The generalized histogram still considers all pixels in an image It is reasonable to apply the generalized histogram to smaller image regions as in locally adaptive histogram equalization Therefore… Create an adaptive generalized histogram equalization by applying GH to a region and updating the center pixel value as in convolution.

8. Visual Information Solutions Generalized Histogram Equalization Original Locally Adaptive (145x145 pixels)  = 1,  = 300 unsharp mask, radius 3 (both)

9. Visual Information Solutions Particle Swarm Optimization PSO is a genetic algorithm for exploring a parameter space Key concepts: The parameter space is randomly populated by a swarm of “particles” Each particle evaluates the fitness function and stores its own best position Each new step of the algorithm moves the particles according to its own best position and the best overall position of the swarm (the global best) Iteration continues until some convergence criterion has been met “Particle swarm optimization”, J. Kennedy, R. Eberhart, in Proc. of the IEEE Int. Conf. on Neural Networks, Piscataway, NJ, pp. 1942–1948, 1995.

10. Visual Information Solutions Particle Swarm Optimization

11. Visual Information Solutions Particle Swarm Optimization In order to do curve fitting with PSO, the fitness function becomes the  2 between the measured points and the fit function with the current particle parameter values. For the examples here, the fit function is a 2D Gaussian: z i = p 0 + p 1 * exp(-0.5 * (((x i – p 4 )/p 2 ) 2 + ((y i -p 5 )/p 3 ) 2 )) The goal is to fit 11x11 images of “peaks” during 3D PALM reconstruction. There are 2 to 6 million peaks to be fit. PALM = “Photo-activated Localization Microscopy” “Interferometric fluorescent super-resolution microscopy resolves 3D cellular ultrastructure”, G. Shtengel, et. al. PNAS March 3, 2009, vol 106, no 9, 3125-3130.

12. Visual Information Solutions Particle Swarm Optimization PSO is very computationally expensive On a single CPU it is about 1000x slower than gradient search (which IDL uses) So, why use it in this case? The answer lies in the fact that there are millions of images to be fit: PSO is straightforward to implement in CUDA In this case, convergence is reached in 25 to 30 iterations The CUDA architecture enables thousands of simultaneous fits Added benefit: fits always converge, no failures, no initial guesses

13. Visual Information Solutions Particle Swarm Optimization CUDA programming is in low-level C code, still difficult IDL provides the framework Visualization in 3D Easy implementation of advanced reconstruction algorithms Easy interface to CUDA CUDA provides the computational engine for time-critical tasks beyond what can be reasonably done with optimized code using multiple CPUs (IDL and C)

14. Visual Information Solutions Particle Swarm Optimization Number of images Fit time (s) IDL CUDA

15. Visual Information Solutions Particle Swarm Optimization Initialize seeds for the PRNGs (hybrid Tausworthe, CPU) On the GPU: Each image fit by single block of threads Each thread is one particle (256 particles/image) Canonical algorithm iterated set number of times (25-30) Per thread PRNG – Park & Miller MINSTD with CPU seed Parameter uncertainties estimated from the standard deviation of the last 5 global best positions Parameters, uncertainties and reduced  2 returned to IDL Block test for multi-stream PRNG demonstrates that the hybrid Tausworthe seeds work well for the MINSTD generator (Jarque-Bera 94%).

16. Visual Information Solutions Particle Swarm Optimization Number of Swarm Iterations Euclidean Distance (swarm to CURVEFIT) 30

17. Visual Information Solutions Conclusion IDL provides researchers with a powerful array processing language combined with superb 2D and 3D visualizations enabling rapid and sophisticated analysis of data CUDA adds raw “supercomputing” power to desktop IDL + CUDA enable researchers to accomplish with an inexpensive PC what in the past would have been impossible or required a large mainframe or cluster.

18. Visual Information Solutions A final example… home-grown in Australia!