Download presentation
Presentation is loading. Please wait.
Published byJewel Bryan Modified over 6 years ago
1
EEC-693/793 Applied Computer Vision with Depth Cameras
Lecture 21 Wenbing Zhao
2
Outline Image processing with Emgu CV Using Emgu CV with Kinect
Shape detection Object detection Using Emgu CV with Kinect
3
Shapes Detection Contours
The Hough transform, which will enable us to detect regular shapes like lines and circles in images Calculation of bounding boxes
4
Contours Difference between contours and edges
Edges are local maxima of intensity gradients These gradient maxima are not all on the outlines of objects and they may be noisy Canny edges are a lot like contours Contours: a set of points connected to each other, most likely to be located on the outline of objects TColor value: image color
5
Finding Contours Obtain Canny edges: Image<>.Canny()
public Contour<Point> FindContours(CHAIN_APPROX_METHOD method, RETR_TYPE type, MemStorage stor); Obtain Canny edges: Image<>.Canny() Call FindContours() on the cannyEdges image Parameters: method: The type of approximation method type: The retrieval type stor: The storage used by the sequences Returns: Contour if there is any; null if no contour is found Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE, Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_LIST,
6
Hough Transform The Hough transform transforms a contour from X-Y space to a parameter space (typically the polar coordinate system) It then uses certain parametric properties of target curves (like lines or circles) to identify the points that fit the target curve in parameter space
7
Detecting Lines with Hough Transform
A point in a 2D image can be represented in Cartesian coordinate as (x,y), or in polar coordinate as (r,) Points that are collinear in the Cartesian space correspond to different sinusoids in the Hough space (i.e., polar coordinate) Points in the same line would intersect at (r,) in Hough space => How lines are detected
8
Detecting Circles with Hough Transform
Need 3 parameters: two for the center, one for the radius (a, b, r) Equation for the circle, given the center at (a,b) and a point in the circle (x,y): x = a + r cos() Y = b + r sin() When the angle sweeps through the full 360 degree range, the points (x,y) trace the perimeter of a circle Search for a, b, r Use an accumulator: 3d array
9
Hough Transform: Accumulator
Accumulator: array used for shape detection in Hough space Dimension of the array corresponds to the number of unknown parameters of a given Hough transform Transform is implemented by quantizing the Hough parameter space into finite intervals called accumulator cells One cell per each (x,y) point Accumulator cell is incremented if (x,y) lies along the curve Bins with the highest values or peaks in the accumulator array represent strong evidence that the shape is detected Taken from:
10
Hough Transform APIs public LineSegment2D[][] HoughLinesBinary( double rhoResolution, double thetaResolution, int threshold, double minLineWidth, double gapBetweenLines); Method defined on Image<> class. Must find edges first: Image<>.Canny() method Parameters: rhoResolution: Distance resolution in pixel-related units. thetaResolution: Angle resolution measured in radians threshold: A line is returned by the function if the corresponding accumulator value is greater than threshold minLineWidth: Minimum width of a line gapBetweenLines: Minimum gap between lines Returns: The line segments detected for each of the channels
11
Hough Transform APIs public CircleF[][] HoughCircles(TColor cannyThreshold, TColor accumulatorThreshold, double dp, double minDist, int minRadius, int maxRadius); Also defined on Image<> class. So do Canny edges as well as circle detection Parameters: cannyThreshold: The higher threshold of the two passed to Canny edge detector accumulatorThreshold: Accumulator threshold at the center detection stage. The smaller it is, the more false circles may be detected. dp: Resolution of the accumulator minRadius: Minimal radius of the circles to search for maxRadius: Maximal radius of the circles to search for minDist: Minimum distance between centers of the detected circles. Returns: The circle detected for each of the channels dp: Resolution of the accumulator used to detect centers of the circles. For example, if it is 1, the accumulator will have the same resolution as the input image, if it is 2 - accumulator will have twice smaller width and height
12
Bounding Box A rectangle of the minimum area enclosing a set of points
Defined in Contour class as a property public override Rectangle BoundingRectangle { get; } Requires MCvContour.rect to be pre-computed
13
Building Shape Detection App
Can be built on top of the image filtering app by adding a button for shape detection
14
Building Shape Detection App
//Convert the image to grayscale and filter out the noise Image<Gray, Byte> gray = img.Convert<Gray, Byte>().PyrDown().PyrUp(); // Detecting circles Gray cannyThreshold = new Gray(180); Gray cannyThresholdLinking = new Gray(120); Gray circleAccumulatorThreshold = new Gray(500); CircleF[] circles = gray.HoughCircles( cannyThreshold, circleAccumulatorThreshold, 4.0, //Resolution of the accumulator used to detect centers of the circles 15.0, //min distance 5, //min radius 0 //max radius )[0]; //Get the circles from the first channel
15
Building Shape Detection App
// detecting lines Image<Gray, Byte> cannyEdges = gray.Canny(cannyThreshold, cannyThresholdLinking); LineSegment2D[] lines = cannyEdges.HoughLinesBinary( 1, //Distance resolution in pixel-related units Math.PI / 45.0, //Angle resolution measured in radians. 20, //threshold 30, //min Line width 10 //gap between lines )[0]; //Get the lines from the first channel // find triangles and rectangles List<Triangle2DF> triangleList = new List<Triangle2DF>(); List<MCvBox2D> boxList = new List<MCvBox2D>(); //a box is a rotated rectangle
16
Building Shape Detection App
//allocate storage for contour approximation using (MemStorage storage = new MemStorage()) for ( Contour<Point> contours = cannyEdges.FindContours( Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE, Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_LIST, storage); contours != null; contours = contours.HNext) { Contour<Point> currentContour = contours.ApproxPoly(contours.Perimeter * 0.05, storage);
17
Building Shape Detection App
//only consider contours with area greater than 250 if (currentContour.Area > 250) { if (currentContour.Total == 3) //The contour has 3 vertices, it is a triangle { Point[] pts = currentContour.ToArray(); triangleList.Add(new Triangle2DF(pts[0], pts[1], pts[2] )); } else if (currentContour.Total == 4) //The contour has 4 vertices. // determine if all the angles in the contour are within [80, 100] degree bool isRectangle = true; LineSegment2D[] edges = PointCollection.PolyLine(pts, true); for (int i = 0; i < edges.Length; i++) { double angle = Math.Abs(edges[(i + 1) % edges.Length].GetExteriorAngleDegree(edges[i])); if (angle < 80 || angle > 100) { isRectangle = false; break; } if (isRectangle) boxList.Add(currentContour.GetMinAreaRect()); } } }
18
Building Shape Detection App
Image<Bgr, Byte> shapesImg = img.CopyBlank(); // draw triangles and rectangles foreach (Triangle2DF triangle in triangleList) shapesImg.Draw(triangle, new Bgr(Color.DarkBlue), 2); foreach (MCvBox2D box in boxList) shapesImg.Draw(box, new Bgr(Color.DarkOrange), 2); // draw circles foreach (CircleF circle in circles) shapesImg.Draw(circle, new Bgr(Color.Brown), 2); image2.Source = ToBitmapSource(shapesImg);
19
The Object Detection Problem
Object detection is typically done by comparing two sets of images One image is the model for the object you want to detect The other image contains the object you want to detect Keypoint-based object detection methods Avoid the use of the characteristics of the whole object model for comparison Find certain “important” points, i.e., keypoints, in the object model, and compare only the keypoints Using some notion of similarity and see how many keypoint descriptors match
20
Keypoints and Descriptors
Keypoint descriptors are often called features Object detection algorithm typically is scale and rotation invariant Scale invariance: actual objects can be bigger or smaller than the model image Rotation invariance: actual objects may be rotated compared with the model image Each keypoint has a scale and an orientation associated
21
SIFT Keypoint Detection
Convolves the image with Gaussians of successively increasing variance => each time, the scale is doubled and the image is down-sampled by a factor of 2 This creates a scale pyramid Successive images in this pyramid are subtracted from each other => resulting images are said to be the output of a Difference of Gaussians (DoG) operator on the original image A point is selected as a keypoint only if it is higher or lower than all its 26 neighbors In the polling model, the application opens a channel for the stream, and whenever the application needs a frame, it sends a request to get the frame
22
SIFT Keypoint Detection
23
SIFT Keypoint Detection
Keypoint locations are further filtered by checking that they have sufficient contrast and that they are not a part of an edge A square region around the keypoint corresponding to a circle with a radius of 1.5 times the scale is selected in the image The gradient orientation at every point in this region is also computed An orientation histogram is constructed. Peaks in the histogram => orientation of the keypoints If there are multiple peaks (values within 80% of the highest peak), multiple keypoints are created at the same location with multiple orientations.
24
SURF Keypoint Detection
Significantly faster than SIFT SURF uses rectangular discretized integer approximations for complicated continuous real-valued functions SURF uses maxima of the Hessian matrix determinant
25
SURF Keypoint Detection
Hessian matrix can be thought of as second-order spatial derivatives of the image after smoothing it with a Gaussian filter Second-order spatial derivatives of the image will have a peak if there are sudden intensity changes However, edges can also count as sudden intensity changes. The determinant of the matrix helps us to distinguish edges from corners The determinant of the Hessian matrix: det(H) = Lxx(x, s)*Lyy(x, s) – Lxy(x,s)2
26
SURF Keypoint Detection
det(H) = Lxx(x, s)*Lyy(x, s) – Lxy(x,s)2 Lxx and Lyy respond to vertical and horizontal edges, respectively Lxy responds most favorably to corners formed by diagonal edges The determinant therefore will have a high value when there is an intersecting pair of horizontal and vertical edges (making a corner), or when there is a corner made from diagonal edges
27
Determinant of a Matrix
The determinant is a value associated with a square matrix the absolute value of the determinant gives the scale factor by which area or volume (or a higher dimensional analogue) is multiplied under the associated linear transformation, while its sign indicates whether the transformation preserves orientation
28
SURF Keypoint Detection
Convolution with a kernel: element-wise multiplication of pixel values with the kernel elements and then summing up If kernel elements are constant, we can sum up the pixel value under the kernel and multiply the sum by the constant kernel value Box filter: kernel with constant elements in rectangular regions Integral image: For any image, its integral image at a pixel is its cumulative sum until that pixel, starting from the origin (top-left corner) Mathematically, if I is an image and H is the integral image, the pixel (x, y) of H is given by:
29
SURF Keypoint Detection
To construct the scale-space pyramid, SURF increases the size of the Gaussian filter rather than reducing the size of the image Next finds the extrema of the Hessian matrix determinant values at different scales by comparing a point with its 26 neighbors in the pyramid just like SIFT This gives the SURF keypoints with their scales Keypoint orientation is decided by selecting a circular neighborhood of radius 6 times the scale of the keypoint around a keypoint At every point in this neighborhood, responses to horizontal and vertical box filters (called Haar wavelets) are recorded Extremum: a maximum or minimum value of a function in a specified neighborhood.
30
SURF Keypoint Detection
The responses are weighted and are represented as vectors in a space with horizontal response strength along the x-axis and vertical response strength along the y-axis A sliding arc with angle of 60 degrees sweeps a rotation through this space All the responses within the window are summed to give a new vector; there are as many of these vectors as there are iterations of the sliding window The largest of these vectors lends its orientation to the keypoint Extremum: a maximum or minimum value of a function in a specified neighborhood. The responses are weighted with a Gaussian (s = 2.5 times scale),
31
Keypoint Descriptor Each SIFT keypoint has a 128-element descriptor
Each SURF keypoint has a 64-element descriptor Matching keypoint descriptors Two descriptors are considered to match if the Euclidean distance between them is low The matches are filtered using special algorithms called “nearest neighbor searches” to find out the test descriptor closest in the Euclidean sense to each train descriptor A descriptor match is eliminated if the distance from the train descriptor to its 1st nearest neighbor is greater than a threshold times the distance to the 2nd nearest neighbor This threshold is usually set to 0.8. This forces the matches to be unique and highly “discriminative” The lower the threshold, the more stringent on matching
32
EmguCV MKeyPoint Struct
public struct MKeyPoint float Angle; // orientation of the keypoint int ClassId; // object id int Octave; // octave (pyramid layer) from which the keypoint has been extracted PointF Point; // coordinates of the keypoint float Response; // the response by which the strongest keypoints have been selected: an indicator how good a point is float Size; // diameter of the useful keypoint adjacent area
33
EmguCV ImageFeature Struct
public struct ImageFeature public float[] Descriptor; public MKeyPoint KeyPoint
34
Build SURFeature App
35
Build SURFeature App Import name spaces Add member variables
using System.Diagnostics; using System.Drawing; using System.Runtime.InteropServices; using Emgu.CV; using Emgu.CV.CvEnum; using Emgu.CV.Features2D; using Emgu.CV.Structure; using Emgu.CV.UI; Image<Gray, Byte> modelImg; Image<Gray, Byte> testImg;
36
Build SURFeature App Constructor and init() public MainWindow() {
InitializeComponent(); init(); } private void init() this.modelImg = new Image<Gray, Byte>("box.png"); this.testImg = new Image<Gray, Byte>("box_in_scene.png"); image1.Source = ToBitmapSource(this.modelImg); image2.Source = ToBitmapSource(this.testImg);
37
Build SURFeature App Add button click event handler Feature detection
private void button1_Click(object sender, RoutedEventArgs e) { featureTest(); } private void featureTest() { SURFDetector surfParam = new SURFDetector(500, false); //extract features from the object image ImageFeature[] modelFeatures = surfParam.DetectFeatures(this.modelImg, null); //Create a Feature Tracker Features2DTracker tracker = new Features2DTracker(modelFeatures); // extract features from the observed image ImageFeature[] imageFeatures = surfParam.DetectFeatures(this.testImg, null);
38
Build SURFeature App Feature detection
Features2DTracker.MatchedImageFeature[] matchedFeatures = tracker.MatchFeature(imageFeatures, 2, 20); matchedFeatures = Features2DTracker.VoteForUniqueness(matchedFeatures, 0.8); matchedFeatures = Features2DTracker.VoteForSizeAndOrientation(matchedFeatures, 1.5, 20); HomographyMatrix homography = Features2DTracker.GetHomographyMatrixFromMatchedFeatures(matchedFeatures); //Merge the object image and the observed image into one image for display Image<Gray, Byte> res = this.modelImg.ConcateHorizontal(this.testImg); matchedFeatures are refined via Features2DTracker.VoteForUniqueness(), and then via Features2DTracker.VoteForSizeAndOrientation() Homography: a matrix defining an perspective transform
39
Build SURFeature App Feature detection
//draw lines between the matched features foreach (Features2DTracker.MatchedImageFeature matchedFeature in matchedFeatures) { PointF p = matchedFeature.ObservedFeature.KeyPoint.Point; // due to concatenation, need to shift horizontally by Width p.X += this.modelImg.Width; res.Draw(new LineSegment2DF(matchedFeature.SimilarFeatures[0].Feature.KeyPoint.Point, p), new Gray(0), 1); }
40
Build SURFeature App Feature detection
// draw the project region on the image if (homography != null) { Rectangle rect = this.modelImg.ROI; // region of interest PointF[] pts = new PointF[] { new PointF(rect.Left, rect.Bottom), new PointF(rect.Right, rect.Bottom), new PointF(rect.Right, rect.Top), new PointF(rect.Left, rect.Top)}; homography.ProjectPoints(pts); for (int i = 0; i < pts.Length; i++) pts[i].X += this.modelImg.Width; res.DrawPolyline(Array.ConvertAll<PointF, Point>(pts, Point.Round), true, new Gray(255.0), 5); } image3.Source = ToBitmapSource(res);
41
Build a StopSign Detection App
Need one stop sign model image and a test image
42
Build a StopSign Detection App
Import namespaces Add member variables using System.Diagnostics; using System.Drawing; using System.Text; using Emgu.CV; using Emgu.CV.Features2D; using Emgu.CV.Structure; using Emgu.Util; using System.Runtime.InteropServices; // for DllImport private Features2DTracker tracker; private SURFDetector detector; private MemStorage octagonStorage; private Contour<Point> octagon; Image<Bgr, Byte> img;
43
Build a StopSign Detection App
Constructor and initialization Button event handler public MainWindow() { InitializeComponent(); DisplayImage(); } private void DisplayImage() this.img = new Image<Bgr, Byte>("stop-sign.jpg"); image1.Source = ToBitmapSource(this.img); private void button1_Click(object sender, RoutedEventArgs e) { ProcessImage(); }
44
Build a StopSign Detection App
private void ProcessImage() { this.detector = new SURFDetector(500, false); using(Image<Bgr, Byte> imgModel = new Image<Bgr, Byte>("stop-sign-model.png")) using (Image<Gray, Byte> redMask = GetRedPixelMask(imgModel)) this.tracker = new Features2DTracker(this.detector.DetectFeatures(redMask, null)); } this.octagonStorage = new MemStorage(); this.octagon = new Contour<Point>(this.octagonStorage); this.octagon.PushMulti(new Point[] { new Point(1,0), new Point(2,0), new Point(3,1), new Point(3,2), new Point(2,3), new Point(1,3), new Point(0,2), new Point(0,1)}, Emgu.CV.CvEnum.BACK_OR_FRONT.FRONT); List<Image<Gray, Byte>> stopSignList = new List<Image<Gray, byte>>(); List<Rectangle> stopSignBoxList = new List<Rectangle>(); DetectStopSign(this.img, stopSignList, stopSignBoxList);
45
Build a StopSign Detection App
Point startPoint = new Point(10, 10); for (int i = 0; i < stopSignList.Count; i++) { Rectangle rect = stopSignBoxList[i]; this.img.Draw(rect, new Bgr(Color.Aquamarine), 2); image1.Source = ToBitmapSource(this.img); label1.Content = "Stop sign detected!"; } if (stopSignList.Count < 1) label1.Content = "No stop sign detected";
46
Build a StopSign Detection App
public void DetectStopSign(Image<Bgr, byte> img, List<Image<Gray, Byte>> stopSignList, List<Rectangle> boxList) { Image<Bgr, Byte> smoothImg = img.SmoothGaussian(5, 5, 1.5, 1.5); Image<Gray, Byte> smoothedRedMask = GetRedPixelMask(smoothImg); //Use Dilate followed by Erode to eliminate small gaps in some contour. smoothedRedMask._Dilate(1); smoothedRedMask._Erode(1); using (Image<Gray, Byte> canny = smoothedRedMask.Canny(new Gray(100), new Gray(50))) using (MemStorage stor = new MemStorage()) Contour<Point> contours = canny.FindContours( Emgu.CV.CvEnum.CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE, Emgu.CV.CvEnum.RETR_TYPE.CV_RETR_TREE, stor); FindStopSign(img, stopSignList, boxList, contours); }
47
Build a StopSign Detection App
private static Image<Gray, Byte> GetRedPixelMask(Image<Bgr, byte> image) { using (Image<Hsv, Byte> hsv = image.Convert<Hsv, Byte>()) { Image<Gray, Byte>[] channels = hsv.Split(); try { //channels[0] is the mask for hue less than 20 or larger than 160 CvInvoke.cvInRangeS(channels[0], new MCvScalar(20), new MCvScalar(160), channels[0]); channels[0]._Not(); //channels[1] is the mask for satuation of at least 10, this is mainly used to filter out white pixels channels[1]._ThresholdBinary(new Gray(10), new Gray(255.0)); CvInvoke.cvAnd(channels[0], channels[1], channels[0], IntPtr.Zero); } finally { channels[1].Dispose(); channels[2].Dispose(); } return channels[0];
48
Build a StopSign Detection App
private void FindStopSign(Image<Bgr, byte> img, List<Image<Gray, Byte>> stopSignList, List<Rectangle> boxList, Contour<Point> contours) { for (; contours != null; contours = contours.HNext) contours.ApproxPoly(contours.Perimeter * 0.02, 0, contours.Storage); if (contours.Area > 200) double ratio = CvInvoke.cvMatchShapes(this.octagon, contours, Emgu.CV.CvEnum.CONTOURS_MATCH_TYPE.CV_CONTOURS_MATCH_I3, 0); if (ratio > 0.1) //not a good match of contour shape Contour<Point> child = contours.VNext; if (child != null) FindStopSign(img, stopSignList, boxList, child); continue; }
49
Build a StopSign Detection App
Rectangle box = contours.BoundingRectangle; Image<Gray, Byte> candidate; using (Image<Bgr, Byte> tmp = img.Copy(box)) candidate = tmp.Convert<Gray, byte>(); //set the value of pixels not in the contour region to zero using (Image<Gray, Byte> mask = new Image<Gray, byte>(box.Size)) { mask.Draw(contours, new Gray(255), new Gray(255), 0, -1, new Point(-box.X, -box.Y)); double mean = CvInvoke.cvAvg(candidate, mask).v0; candidate._ThresholdBinary(new Gray(mean), new Gray(255.0)); candidate._Not(); mask._Not(); candidate.SetValue(0, mask); }
50
Build a StopSign Detection App
ImageFeature[] features = this.detector.DetectFeatures(candidate, null); Features2DTracker.MatchedImageFeature[] matchedFeatures = this.tracker.MatchFeature(features, 2, 0); int goodMatchCount = 0; foreach (Features2DTracker.MatchedImageFeature ms in matchedFeatures) if (ms.SimilarFeatures[0].Distance < 0.5) goodMatchCount++; if (goodMatchCount >= 10) { boxList.Add(box); stopSignList.Add(candidate); }
51
Analyzing Kinect Depth Images
Enable and capture Kinect depth frames Convert depth image to bitmap Use depth as color intensity Convert bitmap to Emgu CV Image Convert to Image<Gray,byte> Find contours in the converted image You can work from here on 11/7/2018 EEC492/693/793 - iPhone Application Development
52
Contour Detection in Kinect Depth Images
Start a new C# WPF project Add references Microsoft.Kinect, System.Drawing Emgu.CV, Emgu.CV.UI, Emgu.Util Copy opencv opencv_core220.dll, opencv_imgproc220.dll to your project’s bin/Debug directory Add UI, a helper class, and code Change project property to target 64bit 11/7/2018 EEC492/693/793 - iPhone Application Development
53
Contour Detection in Kinect Depth Images
11/7/2018 EEC492/693/793 - iPhone Application Development
54
EEC492/693/793 - iPhone Application Development
ImageHelper.cs Add a new class to the project called ImageHelper.cs Import namespace Change ImageHelper class type using Emgu.CV; using Microsoft.Kinect; using System.Windows; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Runtime.InteropServices; using System.IO; public static class ImageHelper { private const int MaxDepthDistance = 4000; private const int MinDepthDistance = 350; private const int MaxDepthDistanceOffset = 3650; 11/7/2018 EEC492/693/793 - iPhone Application Development
55
EEC492/693/793 - iPhone Application Development
ImageHelper.cs public static BitmapSource SliceDepthImage(this DepthImageFrame image, int min = MinDepthDistance, int max = MaxDepthDistance) { int width = image.Width; int height = image.Height; short[] rawDepthData = new short[image.PixelDataLength]; image.CopyPixelDataTo(rawDepthData); var pixels = new byte[height * width * 4]; const int BlueIndex = 0; const int GreenIndex = 1; const int RedIndex = 2; for (int depthIndex = 0, colorIndex = 0; depthIndex < rawDepthData.Length && colorIndex < pixels.Length; depthIndex++, colorIndex += 4) { // Calculate the distance represented by the two depth bytes int depth = rawDepthData[depthIndex] >> DepthImageFrame.PlayerIndexBitmaskWidth; // Map the distance to an intesity that can be represented in RGB var intensity = CalculateIntensityFromDistance(depth); if (depth > min && depth < max) {// Apply the intensity to the color channels pixels[colorIndex + BlueIndex] = intensity; //blue pixels[colorIndex + GreenIndex] = intensity; //green pixels[colorIndex + RedIndex] = intensity; //red } return BitmapSource.Create(width, height, 96, 96, PixelFormats.Bgr32, null, pixels, width * 4); 11/7/2018 EEC492/693/793 - iPhone Application Development
56
EEC492/693/793 - iPhone Application Development
ImageHelper.cs public static byte CalculateIntensityFromDistance(int distance) { // This will map a distance value to a range // for the purposes of applying the resulting value // to RGB pixels. float newMax = distance - MinDepthDistance; if (newMax > 0) return (byte)( * newMax / MaxDepthDistanceOffset); } else return (byte)255; 11/7/2018 EEC492/693/793 - iPhone Application Development
57
EEC492/693/793 - iPhone Application Development
ImageHelper.cs public static System.Drawing.Bitmap ToBitmap(this BitmapSource bitmapsource) { System.Drawing.Bitmap bitmap; using (var outStream = new MemoryStream()) // from System.Media.BitmapImage to System.Drawing.Bitmap BitmapEncoder enc = new BmpBitmapEncoder(); enc.Frames.Add(BitmapFrame.Create(bitmapsource)); enc.Save(outStream); bitmap = new System.Drawing.Bitmap(outStream); return bitmap; } [DllImport("gdi32")] private static extern int DeleteObject(IntPtr o); public static BitmapSource ToBitmapSource(IImage image) { // you have seen this } 11/7/2018 EEC492/693/793 - iPhone Application Development
58
Add Code to MainWindow.xaml.cs
Import namespace Add member variables Modify constructor using System.Drawing; using Microsoft.Kinect; using Emgu.CV; using Emgu.CV.Structure; using System.IO; KinectSensor sensor; WriteableBitmap depthBitmap; WriteableBitmap colorBitmap; DepthImagePixel[] depthPixels; byte[] colorPixels; public MainWindow() { InitializeComponent(); this.Loaded += WindowLoaded; } 11/7/2018 EEC492/693/793 - iPhone Application Development
59
Add Code to MainWindow.xaml.cs
void WindowLoaded(object sender, RoutedEventArgs e) { foreach (var potentialSensor in KinectSensor.KinectSensors) { if (potentialSensor.Status == KinectStatus.Connected) { this.sensor = potentialSensor; break; } if (null != this.sensor) { this.sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30); this.sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30); this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength]; this.depthPixels = new DepthImagePixel[this.sensor.DepthStream.FramePixelDataLength]; this.colorBitmap = new WriteableBitmap(this.sensor.ColorStream.FrameWidth, this.sensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null); this.depthBitmap = new WriteableBitmap(this.sensor.DepthStream.FrameWidth, this.sensor.DepthStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null); this.colorImg.Source = this.colorBitmap; // name of the image control this.outImg.Source = this.depthBitmap; // name of the image control this.sensor.AllFramesReady += this.allFramesReady; try { this.sensor.Start(); } catch (IOException) { this.sensor = null; } } } 11/7/2018 EEC492/693/793 - iPhone Application Development
60
Add Code to MainWindow.xaml.cs
private void allFramesReady(object sender, AllFramesReadyEventArgs e) { BitmapSource depthBmp = null; using (ColorImageFrame colorFrame = e.OpenColorImageFrame()) if (colorFrame != null) colorFrame.CopyPixelDataTo(this.colorPixels); int stride = colorFrame.Width * colorFrame.BytesPerPixel; this.colorBitmap.WritePixels( new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight), this.colorPixels, stride, 0); } 11/7/2018 EEC492/693/793 - iPhone Application Development
61
Add Code to MainWindow.xaml.cs
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame()) { if (depthFrame != null) depthBmp = depthFrame.SliceDepthImage(); if (depthBmp == null) return; Bitmap tmp = depthBmp.ToBitmap(); Image<Bgr, Byte> openCVImg = new Image<Bgr, byte>(tmp); Image<Gray, Byte> gray = openCVImg.Convert<Gray, Byte>().PyrDown().PyrUp(); Gray cannyThreshold = new Gray(180); Gray cannyThresholdLinking = new Gray(120); Image<Gray, Byte> cannyEdges = gray.Canny(cannyThreshold, cannyThresholdLinking); this.outImg.Source = ImageHelper.ToBitmapSource(cannyEdges); } } // using } // if(depthFrame!=null) } } 11/7/2018 EEC492/693/793 - iPhone Application Development
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.