1. Image Filtering Advanced Image Filtering Advanced Image filtering with GDI and DX HW accelerations

2. Image Filtering Advanced An introduction : domain terms ♦Bitmap (let’s assume Bitmap == GDI+ Bitmap) System memory portion containing Pixel as color’s byte ♦DX Surface System or Video memory portion representing a bitmap ♦Texture Mainly Video memory portion containing a bitmap that will be mapped onto a polygon

3. Image Filtering Advanced Enhanced Image Filter Library ♦What does a filter do? ♦What is a filter? ♦What is a filter chain? A filter is a function from image to image, A filter chain is a functional composition so, it’s a function again.

4. Image Filtering Advanced The speed part 1 : unsafe protected override void filter(Bitmap b, Bitmap[] temp) { BitmapData bmData = b.LockBits(new Rectangle(0, 0, b.Width, b.Height), ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb); int stride = bmData.Stride; System.IntPtr Scan0 = bmData.Scan0; unsafe { byte * p = (byte *)(void *)Scan0; int nOffset = stride - b.Width*3; int nWidth = b.Width * 3; for(int y=0;y<b.Height;++y) { for(int x=0; x < nWidth; ++x ) { p[0] = (byte)(255-p[0]); ++p; } p += nOffset; } b.UnlockBits(bmData); } This way is faster than accessing pixels through the GetPixel GDI function

5. Image Filtering Advanced Library structure

6. Image Filtering Advanced The key for success : the filter base class design [Serializable] public abstract class Filter { private Filter next; protected Filter(Filter f) { next = f; } protected Filter() { next = null; } protected abstract void filter(Bitmap b, Bitmap[] temp); protected abstract int NumCopies { get; } public BaseConf GetConfig(){ return new BaseConf(); } public void Apply(Bitmap b) { int n = this.NumCopies; Filter p = next; while (p != null) { n = Math.Max(n, p.NumCopies); p = p.next; } Bitmap[] tmp = new Bitmap[n]; for (int i = 0; i < n; i++) tmp[i] = new Bitmap(b.Width, b.Height); p = this; do { p.filter(b, tmp); p = p.next; } while (p != null); foreach (Bitmap bm in tmp) { bm.Dispose(); } Create all the temporary Images before your filter chain goes with its work!

7. Image Filtering Advanced The speed part 2 : ngen ♦Let’s get native! Run the ngen tool on the filter library assembly to get it compiled once for all.

8. Image Filtering Advanced ♦A little trick to achieve the real magic ♦Let’s pretend : ♦Remember you gain speed but you get also limitations! Image => plane => polygon The speed part 3 : Get HW!

9. Image Filtering Advanced DX Structure ♦We can use the HW 3D acceleration to boost 2D image filtering using the Programmable Pixel Shader

10. Image Filtering Advanced HSL with DX 9 ♦Cross Hardware language! ♦Can contains multiple techniques and passes! ♦Exposes a large number of common function as lerp and so on

11. Image Filtering Advanced HSL Stupid Example float4 Light(float3 LightDir : TEXCOORD1, uniform float4 LightColor, float2 texcrd : TEXCOORD0, uniform sampler samp) : COLOR { float3 Normal = tex2D(samp,texcrd); return dot((Normal*2 - 1), LightDir)*LightColor; }

12. Image Filtering Advanced The Idea 1.Create a DX device on a control 2.Create a 3D plane 3.Put the image you want to process on the plane as a texture 4.Use the Pixel Shader Program to make the GPU works for you 5.Use the Control Graphics class to save your processed image.

13. Image Filtering Advanced A simple but real FX example

14. Image Filtering Advanced And the cons? Here we are... ♦You can use only power of 2 sized bitmap ♦The display size is the maximum size for the output image... But you get a common PC running as it’s got a real expensive (and dedicated) DSP!

15. Image Filtering Advanced The Direct Show way ♦DShow use a graph from sources to renderes (audio and video) ♦Along the graph you can have filters ♦2 approaches –In place transformations –Not in place transformations ♦Only C++ (the unmanaged one!)

16. Image Filtering Advanced CTransformFilter::Transform ♦This filter uses the CTransformInputPin class for its input pin, and the CTransformOutputPin class for its output pin. ♦Use this base class if you want to try filter and then pass it through the graph ♦Beware of memory leak!

17. Image Filtering Advanced Example of transform HRESULT CRleFilter::Transform(IMediaSample *pSource, IMediaSample *pDest) { // Get pointers to the underlying buffers. BYTE *pBufferIn, *pBufferOut; hr = pSource->GetPointer(&pBufferIn); if (FAILED(hr)) { return hr; } hr = pDest->GetPointer(&pBufferOut); if (FAILED(hr)) { return hr; } // Process the data. DWORD cbDest = EncodeFrame(pBufferIn, pBufferOut); KASSERT((long)cbDest GetSize()); pDest->SetActualDataLength(cbDest); pDest->SetSyncPoint(TRUE); return S_OK; }

18. Image Filtering Advanced CTransInPlaceFilter::Transform ♦Transform the input sample in place ♦Use it for real real-time processing ♦What do you want more?

19. Image Filtering Advanced The Frame Dispatcer Class If you want to apply filter to a real time camera or to an avi file in C# there’s an utility from the Medialab (university of Pisa) that can help you to get your frames inside a managed application!

20. Image Filtering Advanced References ♦Enhanced Filter Library and Frame Dispatcer –http://dotnet.di.unipi.it