Super-Resolution of Remotely-Sensed Images Using a Learning-Based Approach Isabelle Bégin and Frank P. Ferrie Abstract Super-resolution addresses the problem of estimating a high-resolution version of a low-resolution image. In this research, a learning-based approach for super-resolution is applied to remotely-sensed images. These images contain fewer sharp edges than those previously used to test the method, making the problem more difficult. The approach uses a Markov network to model the relationship between an image/scene training pair (low-resolution/high-resolution pair). Bayesian belief propagation is used to obtain the posterior probability of the scene, given a low-resolution input image. Preliminary results on Landsat-7 images show that the framework works reasonably well when given a sufficiently large training set. Training The goal of this experiment was to test whether a learning-based approach for super- resolution can be used on remotely-sensed images. These images contain fewer sharp edges than those previously used to test the method. The training set is a pair of blurred and non-blurred Landsat-7 images (Band #3). The 350x350 image was first blurred with a Gaussian filter, subsampled and interpolated back to the original resolution. Problem Is it possible to infer a high-resolution version of an image that is first given at low resolution? Stronger approaches than simple interpolation use Bayesian techniques or regularization while assuming prior probabilities or constraints. Furthermore, these techniques are independent of both the structures present in the image and the nature of the image itself. Freeman et al. propose that statistical relationships between pairs of low-resolution and high-resolution images (image/scene pairs) can be learned. Thus, given a new image to super-resolve (of the same nature as the images in the training set), it is possible to obtain a higher-resolution image based on the statistics acquired from the training pairs. The most important features learned by the method are edges. The algorithm learns what a blurry edge should look like in its corresponding high-resolution image. Thus, the training data need to contain many sharp edges. The experiments described in Freeman et al. were done mostly on such training sets. In the present research, the specific problem of super-resolution of remotely-sensed images (with few sharp edges) is addressed. Discussion Many factors will influence the quality of the results. This framework works better with training images that contain many sharp edges. This is not the case for the images used in this experiment. In the examples shown, no pre-processing steps were done on the images. Results may be improved by first enhancing the edges in the image. Because the framework is statistical, the size and completeness of the training set will have a big impact on the result. In the experiment, the size of the training set was small (350 x 350 image) and therefore might not contain the full variety of structures possibly present in a remotely-sensed image. The fitting of Gaussian mixture models is also very important. Models were obtained with the Expectation Maximization algorithm. The number of iterations, the structure of the covariance and the number of clusters can all modify the computation of the joint probabilities, which are the core of the framework. The size of the patches, the number of principal components and the number of closest patches considered will also impact the final result. Conclusions This research presented a learning-based method for super-resolution applied to remotely-sensed images. A Markov network was used to express the statistical dependencies between a low-resolution/high-resolution training pair. The maximum a posteriori estimate of the scene was obtained by Bayesian propagation. Preliminary results show that such an approach can be used on remotely-sensed images provided that the training set is sufficiently large. Future work will include pre- processing of the images for the enhancement of contained edges as well as the use of a more complete training set. Reference Freeman, W.T., Pasztor, E.C. and Carmichael, O.T., "Learning Low-Level Vision", International Journal of Computer Vision, 40(1), pp , Theory The statistical dependencies between image/scene training pairs (low-resolution and high-resolution images) are modeled in a Markov network where each node describes an image or a scene patch. The topology of the network is shown in the following diagram. Statistical dependencies are shown by lines connecting nodes. Scene patches are depicted in white and image patches in red. There are two types of connections. First, each scene patch is connected to its neighbours. Second, a patch in the scene is connected to its corresponding patch in the low-resolution image. Two nodes not directly connected are assumed to be statistically independent. The Markov assumption allows a “message-passing” rule that involves only local computations. The maximum a posteriori estimate (MAP) for node j is calculated using the following equation (Freeman et al.): where is the message in the previous iteration. In principle, this rule should only be used on chains or trees. However, the topology of the Markov network for the super- resolution problem contains loops. In this situation, it was previously demonstrated that equations (1) and (2) can still be used. It is thus assumed that the network consists of vertical or horizontal chains. Results A new image was also blurred, subsampled and interpolated to form the input image. The results of the super-resolution algorithm (Figure 3b) approximates the high- resolution image reasonably well (Figure 3c). Edges are better preserved than the interpolated version of the input image (Figure 3a). For comparison, the nearest neighbour interpolation is shown in Figure 3d, where the edges appear more choppy. However, thin structures (such as road networks) are lost in the super-resolution process. This problem might be caused by a small patch-size as well as an incomplete training set. Figure 2: Training set (left: sharp image; right: blurred image) Figure 1: Markov network for the super-resolution problem Figure 3. (a) Input image (bilinear interpolation) (b) Result of the super-resolution algorithm (c) Original full-resolution image (d) Nearest neighbour interpolation (a)(b)(c)(d) Construction of the Training Set 1.Blur and subsample a set of high-resolution images. The set should be large enough so as to include the widest range of intensity structures as possible. 2.Interpolate the images back to the original resolution. 3.Break the images and scenes into center-aligned patches. 4.Apply Principal Components Analysis on the training set. 5.Fit Gaussian mixtures to the joint probabilities P(x k, x j ) (neighbouring scene patches k and j ) and P(y k,x k ) (image/scene pair), as well as the prior probability of a scene patch P(x j ). Computation of the Conditional Probabilities 1.Break a new low-resolution (interpolated) image into patches and apply Principal Components Analysis. 2.For each patch y k, find the n closest patches in the training set. The n corresponding scenes will be used in the calculations. 3.The conditional probabilities for each candidate are computed with the following equations: where l and m run from 1 to n and are the candidates for patches x k and x j, respectively. P(y k |x k l ) is thus a n x 1 vector and P(x k l |x j m ) is a n x n matrix. The messages M j k are n x 1 vectors whose elements are set to 1 for the first iteration. 4.The MAP estimate for patch j ( x j MAP ) is obtained from equations (1) and (2). The steps involved in the construction of the training set and in the computation of the joint probabilities are the following: y2y2 y1y1 x4x4 x3x3 x1x1 x2x2 y3y3 y4y4 The advantage of this learning-based algorithm compared to other learning methods is that local support is added in the form of conditional probabilities. In other learning-based approaches, the closest patch in the training set is chosen regardless of its neighbours. This Bayesian technique also has the advantage of learning the prior probabilities instead of assuming them. (1) (2) (3) (4)