1. Generalized sampling theorem (GST) interpretation of DSR
Δ hi
Inter sample spacing needed to guarantee that band-limited image 𝑜 𝑥,𝑦 ⊛ℎ 𝑥,𝑦 is not aliased
Δ lo =𝐺 Δ hi
Inter sample spacing that we can get away with according to the GST
ℳ 1 (𝜉,𝜂)
Transfer function induced by sub-pixel shifting of the object
𝒟 𝜉,𝜂
Transfer function induced by low-resolution pixel geometry 𝒟 𝜉,𝜂 = Δ lo 2 sinc( Δ lo 𝜉)sinc( Δ lo 𝜂)
Note
The following model is incompatible with the matrix formulation of DSR that relies on a blur matrix and a down-sampling matrix.
𝑖 lo [𝑚,𝑛;1]
Band-limited
image
ℳ 1 (𝜉,𝜂)𝒟(𝜉,𝜂) ⊗
𝒢 1 (𝜉,𝜂) ⋮ ⋮ ⋮
𝑖 lo [𝑚,𝑛;𝑘]
ℋ(𝜉,𝜂)
ℳ 𝑘 (𝜉,𝜂)𝒟(𝜉,𝜂) ⊗
𝒢 𝑘 (𝜉,𝜂) ⊕
𝑜(𝑥,𝑦)
Reconstructed image
Geometric
image
OTF
𝑖 lo [𝑚,𝑛;𝐹]
ℳ 𝐹 (𝜉,𝜂)𝒟(𝜉,𝜂) ⊗
𝒢 𝐹 (𝜉,𝜂)
𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ lo ,𝑦−ℓ Δ lo
Observations
The MTF’s of the bank of filters ℳ 𝑘 (𝜉,𝜂)𝒟(𝜉,𝜂) 𝑘=1 𝐹 is identical. The only thing that changes is the PTF, on account of shifting of the object by amounts smaller than Δ lo (sub-pixel shifting)
Frequencies corresponding to nulls in 𝒟 𝜉,𝜂 cannot be recovered using any amount of sub-pixel shifting. This is because sub-pixel shifting does not affect |𝒟 𝜉,𝜂 |. Consequently frequencies that are not resolved by the filter 𝒟(𝜉,𝜂), can never be reconstructed in the super-resolved image.
This fact seems to have been conveniently ignored in DSR literature.
Proof: 𝒟 𝜉,𝜂 =0⇒sinc Δ lo 𝜉 =0 or sinc Δ lo 𝜂 =0. This occurs when either 𝜉 or 𝜂 is a harmonic of 1/Δ lo
It is obvious that more the DSR gain 𝐺, more the number of zeros due to 𝒟 𝜉,𝜂 in the interval 𝜉 , 𝜂 ≤ 1/Δ hi
Papoulis

2. ⊗ Relationship between geometric image of scene & low-resolution image
Inter sample spacing of the detector
Δ hi = Δ hi ÷𝐺
Desired inter sample spacing
ℋ 𝑜𝑝𝑡 (𝜉,𝜂)
Optical Transfer Function ( ℋ 𝑜𝑝𝑡 𝜉,𝜂 ≝ℱ ℎ 𝑜𝑝𝑡 𝑥,𝑦 )
Δ lo 2 sinc − Δ lo 𝜉 sinc − Δ lo 𝜂 = Δ lo 2 sinc Δ lo 𝜉 sinc Δ lo 𝜂
Transfer function induced by low-resolution pixel geometry
NOTE: sinc is an even function
Assumption
The detector geometry of the light sensitive element in the low-resolution imager is assumed to be square, i.e.
ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo
𝑜(𝑥,𝑦) ⊗
𝑚,𝑛∈ℤ 𝛿 𝑥−𝑚 Δ lo ,𝑦−𝑛 Δ lo
ℋ 𝑜𝑝𝑡 (𝜉,𝜂)
Δ lo 2 sinc Δ lo 𝜉 sinc Δ lo 𝜂
𝑖 lo [𝑚,𝑛]
Geometric image of scene
Aliased
image
Pixel transfer function
of low resolution detector
×OTF
𝑖 lo [𝑚,𝑛]= 𝑜 𝑥,𝑦 ⊛ ℎ 𝑜𝑝𝑡 𝑥,𝑦 ℎ 𝑑𝑒𝑡 𝑥−𝑚 Δ lo ,𝑦−𝑛 Δ lo 𝑑𝑥𝑑𝑦

3. ⊗ Equivalent model that is consistent with existing DSR models 𝑜 [𝑘,ℓ]
Δ lo
Inter sample spacing of the detector
Δ hi = Δ hi ÷𝐺
Desired inter sample spacing
ℋ(𝜉,𝜂)
Discrete Time Fourier Transform of the sampled optical PSF ℎ 𝑘,𝓁 ≝ ℎ 𝑜𝑝𝑡 (𝑘 Δ hi ,𝓁 Δ hi )
𝒟 𝜉,𝜂
Discrete Time Fourier Transform of 𝑑 𝑘,𝓁 ≝ ℎ 𝑑𝑒𝑡 𝑥,𝑦 rect 𝑥−𝑘 Δ hi Δ hi rect 𝑦−𝓁 Δ hi Δ hi 𝑑𝑥𝑑𝑦
In the absence of a focal plane mask, ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo .
𝑑 𝑘,𝓁 has exactly 𝐺 2 non-zero entries since 𝐺 is an integer. These entries are 1 in the absence of a focal plane mask, and correspond to a 𝐺×𝐺 code in the presence of a focal plane mask
𝑜(𝑥,𝑦) ⊗
𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ hi ,𝑦−ℓ Δ hi
Δ hi 2 sinc Δ hi 𝜉 sinc Δ hi 𝜂
circ 𝜆 2𝑁𝐴 𝜉 2 + 𝜂 2
𝑜 (𝑥,𝑦)
Geometric image of scene
Pixel transfer function
of high resolution detector
×Ideal low pass filter
𝑜 [𝑘,ℓ]
𝑂𝑇𝐹×Pixel transfer function
of low resolution detector
↓ Δ lo Δ hi
ℋ(𝜉,𝜂) 𝒟 𝜉,𝜂
𝑖 lo [𝑚,𝑛]
Subsampling
Aliased
image
𝑖 lo 𝑚,𝑛 = 𝑘,𝓁∈ℤ 𝑜 𝑘,𝓁 Γ 𝑑ℎ 𝑚𝐺−𝑘,𝑛𝐺−𝓁 = 𝑜 𝑘,𝓁 ⊛ Γ 𝑑ℎ 𝑘,𝓁 ↓𝐺
Observations
The above model is compatible with the matrix formulation of DSR wherein one is interested in recovering an image of the scene as sampled by a detector with smaller pixels (not just smaller inter-sample spacing)
Frequencies corresponding to nulls in either ℋ 𝜉,𝜂 or 𝒟 𝜉,𝜂 cannot be recovered. When not using focal plane masks, the detector integration pattern 𝑑[𝑘,ℓ] corresponds to a linear phase FIR filter with even or odd number of taps. The magnitude response of such a filter can be expressed as a trigonometric series.
The problem of nulls in ℋ(𝜉,𝜂) 𝒟 𝜉,𝜂 may be remedied by diversity in either the sampled PSF or the detector integration geometry (focal plane mask).

4. ⊗ ⊕ Filterbank interpretation of DSR Δ lo
Inter sample spacing of the detector
Δ hi = Δ hi ÷𝐺
Desired inter sample spacing
ℋ(𝜉,𝜂)
Discrete Time Fourier Transform of the sampled optical PSF ℎ 𝑘,𝓁 ≝ℎ(𝑘 Δ hi ,𝓁 Δ hi )
𝒟 𝜉,𝜂
Discrete Time Fourier Transform of 𝑑 𝑘,𝓁 ≝ ℎ 𝑑𝑒𝑡 𝑥,𝑦 rect 𝑥−𝑘 Δ hi Δ hi rect 𝑦−𝓁 Δ hi Δ hi 𝑑𝑥𝑑𝑦
In the absence of a focal plane mask, ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo .
𝑑 𝑘,𝓁 has exactly 𝐺 2 non-zero entries since 𝐺 is an integer. These entries are 1 in the absence of a focal plane mask, and correspond to a 𝐺×𝐺 code in the presence of a focal plane mask
Analysis filter-bank
𝑖 lo [𝑚,𝑛;1]
↓ Δ lo Δ hi
Synthesis filter-bank
↑ Δ lo Δ hi
ℋ 1 (𝜉,𝜂)
𝒢 1 (𝜉,𝜂) ⋮ ⋮ ⋮ ⋮
𝑖 lo [𝑚,𝑛;𝑘] ⊗
↓ Δ lo Δ hi
↑ Δ lo Δ hi
𝑜 (𝑥,𝑦)
ℋ 𝑘 (𝜉,𝜂)
𝒢 𝑘 (𝜉,𝜂) ⊕
Reconstructed image
Band-limited image
𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ hi ,𝑦−ℓ Δ hi
𝑖 lo [𝑚,𝑛;𝐹]
↓ Δ lo Δ hi
↑ Δ lo Δ hi
ℋ 𝐹 (𝜉,𝜂)
𝒢 𝐹 (𝜉,𝜂)
Samples from band-limited image
Observations
The above model is compatible with the matrix formulation of DSR
Introduce noise before down-sampling !!!

5. ⊗ Standard DSR workflow Δ lo Inter sample spacing of the detector
Δ hi = Δ hi ÷𝐺
Desired inter sample spacing
ℋ(𝜉,𝜂)
Discrete Time Fourier Transform of the sampled optical PSF ℎ 𝑘,𝓁 ≝ℎ(𝑘 Δ hi ,𝓁 Δ hi )
𝒟 𝜉,𝜂
Discrete Time Fourier Transform of 𝑑 𝑘,𝓁 ≝ ℎ 𝑑𝑒𝑡 𝑥,𝑦 rect 𝑥−𝑘 Δ hi Δ hi rect 𝑦−𝓁 Δ hi Δ hi 𝑑𝑥𝑑𝑦
In the absence of a focal plane mask, ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo .
𝑑 𝑘,𝓁 has exactly 𝐺 2 non-zero entries since 𝐺 is an integer. These entries are 1 in the absence of a focal plane mask, and correspond to a 𝐺×𝐺 code in the presence of a focal plane mask
Image that can be recovered by Digital Super Resolution
Pixel transfer function
of high resolution detector
×Ideal low pass filter
Analysis filter-bank
ℋ 1 (𝜉,𝜂) 𝒟 𝜉,𝜂
↓ Δ lo Δ hi
𝑖 lo [𝑚,𝑛;1] ⋮ ⋮
Δ hi 2 sinc Δ hi 𝜉 sinc Δ hi 𝜂
circ 𝜆 2𝑁𝐴 𝜉 2 + 𝜂 2
𝑜 (𝑥,𝑦) ⊗
ℋ 𝑘 (𝜉,𝜂) 𝒟 𝜉,𝜂
↓ Δ lo Δ hi
𝑖 lo [𝑚,𝑛;𝑘]
𝑜(𝑥,𝑦)
Geometric image of scene
𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ hi ,𝑦−ℓ Δ hi
ℋ 𝐹 (𝜉,𝜂) 𝒟 𝜉,𝜂
↓ Δ lo Δ hi
𝑖 lo [𝑚,𝑛;𝐹]
Observations
The above model is compatible with the matrix formulation of DSR wherein one is interested in recovering an image of the scene as sampled by a detector with smaller pixels (not just smaller inter-sample spacing)
Any frequencies lost to blurring due to detector integration i.e. 𝒟 𝜉,𝜂 =0 cannot be recovered. This occurs when either 𝜉 or 𝜂 is a harmonic of 1/Δ lo .
The optical PSF could either be diffraction limited or due to a pseudo-random phase mask as in Amit Ashok’s work
Sub-pixel shift induces changes in the phase of the transfer function ℋ(𝜉,𝜂). These transfer function are denoted as ℋ 𝑘 (𝜉,𝜂) in the above block schematic.
Introduce noise before down-sampling !!!
In this scheme, all we are doing is changing the PTF of each filter in the filter-bank
For perfect recovery, the filters ℋ 𝑖 (𝜉,𝜂) 𝒟 𝜉,𝜂 should have a broad response up-to the optical cutoff frequency.

6. ⊗ DSR workflow in Focal Plane Coding Δ lo
Inter sample spacing of the detector
Δ hi = Δ hi ÷𝐺
Desired inter sample spacing
ℋ(𝜉,𝜂)
Discrete Time Fourier Transform of the sampled optical PSF ℎ 𝑘,𝓁 ≝ℎ(𝑘 Δ hi ,𝓁 Δ hi )
𝒟 𝜉,𝜂
Discrete Time Fourier Transform of 𝑑 𝑘,𝓁 ≝ ℎ 𝑑𝑒𝑡 𝑥,𝑦 rect 𝑥−𝑘 Δ hi Δ hi rect 𝑦−𝓁 Δ hi Δ hi 𝑑𝑥𝑑𝑦
In the absence of a focal plane mask, ℎ 𝑑𝑒𝑡 𝑥,𝑦 =rect 𝑥 Δ lo rect 𝑦 Δ lo .
𝑑 𝑘,𝓁 has exactly 𝐺 2 non-zero entries since 𝐺 is an integer. These entries are 1 in the absence of a focal plane mask, and correspond to a 𝐺×𝐺 code in the presence of a focal plane mask
Image that can be recovered by Digital Super Resolution
Pixel transfer function
of high resolution detector
×Ideal low pass filter
Analysis filter-bank
ℋ 𝜉,𝜂 𝒟 1 𝜉,𝜂
↓ Δ lo Δ hi
𝑖 lo [𝑚,𝑛;1] ⋮ ⋮
Δ hi 2 sinc Δ hi 𝜉 sinc Δ hi 𝜂
circ 𝜆 2𝑁𝐴 𝜉 2 + 𝜂 2
𝑜 (𝑥,𝑦) ⊗
ℋ 𝜉,𝜂 𝒟 𝑘 𝜉,𝜂
↓ Δ lo Δ hi
𝑖 lo [𝑚,𝑛;𝑘]
𝑜(𝑥,𝑦)
Geometric image of scene
𝑘,ℓ∈ℤ 𝛿 𝑥−𝑘 Δ hi ,𝑦−ℓ Δ hi
ℋ 𝜉,𝜂 𝒟 𝐹 𝜉,𝜂
↓ Δ lo Δ hi
𝑖 lo [𝑚,𝑛;𝐹]
Observations
The above model is compatible with the matrix formulation of DSR wherein one is interested in recovering an image of the scene as sampled by a detector with smaller pixels (not just smaller inter-sample spacing)
Any frequencies lost to blurring due to detector integration i.e. 𝒟 𝜉,𝜂 =0 cannot be recovered. This occurs when either 𝜉 or 𝜂 is a harmonic of 1/Δ lo .
Each low-resolution image utilizes a different focal plane mask. The transfer function due to these masks is denoted as 𝒟 𝑘 (𝜉,𝜂) in the above block schematic.
In this scheme, we are changing the shape of the OTF of each filter in the filter-bank.

7. Papoulis Generalized sampling theorem (GST)
𝑓(𝑡)
Band-limited
signal
𝒴 𝑘 (𝜈)
𝒴 𝑀 (𝜈)
𝒴 1 (𝜈) ⋮
ℋ 𝑘 (𝜈)
ℋ 𝑀 (𝜈)
ℋ 1 (𝜈) ⊕ ⊗
𝑛∈ℤ 𝛿 𝑡−𝑛𝑀 𝑇 0
𝑓 (𝑡)
Reconstructed
𝑔 𝑘 (𝑡)
𝑔 𝑀 (𝑡)
𝑔 1 (𝑡)
Pre-filtering
Post-filtering
ℱ 𝜈 =0,
for 𝜈 ≥0.5/ 𝑇 0
Papoulis

8. Generalized sampling theorem (GST) interpretation of Digital Super Resolution
𝒪 𝑏 𝜈 =0,
for 𝜈 ≥0.5/ Δ hi
𝑜(𝑥)
Geometric
image
ℋ 𝑜𝑝𝑡 𝜈
Band-limited
𝑜 𝑏 (𝑥)
𝒴 𝑘 (𝜈)
𝒴 𝑀 (𝜈)
𝒴 1 (𝜈) ⋮
ℋ 𝑘 𝜈
ℋ 𝑀 𝜈
ℋ 1 𝜈 ⊕ ⊗
𝑚∈ℤ 𝛿 𝑥−𝑚 Δ lo
𝑜 (𝑥)
Reconstructed
𝑔 𝑘 (𝑥)
𝑔 𝑀 (𝑥)
𝑔 1 (𝑥)
Pre-filtering
Post-filtering
ℋ 𝑑𝑒𝑡 −𝜈
𝑜 𝑏𝑑 (𝑥)
Papoulis