A Vision-Based System that Detects the Act of Smoking a Cigarette Xiaoran Zheng, University of Nevada-Reno, Dept. of Computer Science Dr. Mubarak Shah, Dr. Niels Lobo, University of Central Florida, Computer Vision Lab Dr. George Bebis, Dr. Dwight Egbert, University of Nevada-Reno, Dept. of Computer Science Abstract: This project attempts to develop an automatic system that processes a sequence of images obtained from a video camera to detect if a person is smoking a cigarette. So far, we found methods to track mouth and arm movements through a number of image frames with the smoker’s upper body occupying 1/4 of the image size. Further work on interpreting the information extracted from tracking, and detecting smoke in an image will ultimately produce such a system. Problem Statement: Cigarette smoking poses dangers to our health, environment and safety. Therefore, when smoking is prohibited in certain areas, the enforcement of this rule is important. Using physical smoke detectors poses constraints. A smoke detector is not suitable for the level of smoke produced by a cigarette, especially in large areas. Letting a human monitor a designated area through surveillance camera is rather inefficient. If a computer is programmed to recognize this activity, an automatic system can be set up to alert humans when the computer “sees” someone smoking through a video camera. Current Approaches: There does not exist a system in which the activity of cigarette smoking can be automatically recognized by a machine. Methodology: 1. Segmentation of human skin: Skin regions are segmented out using color predicate. These regions are manually extracted in the first frame and used as a mask in the training process. Successive frames are tested against the color space generated through training. Training Image Mask Skin Regions 2. Locating the head region: Connecting components is performed to separate skin regions. The head is identified by examining the circularity of each region. Skin RegionsBoundary Points Head Region 3. Tracking of the head region during occlusion: The number of skin regions indicates whether occlusion is present. The method employed here is correlation. Since we have already found the head in the previous frame where occlusion does not occur, the search window can be established to find the head in the current frame where occlusion occurs. The method is further extended by using the boundary points of the head, which have already been defined, to correct the small variations that result from correlation. 4. Tracking of facial features: Facial features are extracted using Sobel edge detector. The size and shape of these edges are used to locate eyes. A new color space is calculated from the RGB space to identify and track lips. Edges New Color Space 5. Tracking of arms: A best-fit line that passes through the center of mass of an arm region provides orientation, as well as the end points of that region. By keeping a history of previous centroids and end points, we can track the two arms. If an arm is occluding the other arm, the program goes to the previous frame where all skin regions are separated, and takes the separated arm as a template. The template is rotated and is matched against a mask that contains all the skin regions in the current frame. If an arm is merged with the head, the program utilizes the method of rotating template for the first three frames, then uses correlation. It is also necessary to keep track of the size of each arm region, which is used to detect body rotation. Conclusion: The goal of this research project is to experiment with different ideas and methods that will enable a computer to detect if a person is smoking a cigarette. So far, we have found methods to track mouth and arms through a sequence of images with the smoker’s upper body occupying 1/4 of the image size. Future Work: The continuation of this project includes the analysis of the time a hand is positioned at the mouth, the interpretation of the gesture of an arm moving up and down, and the detection of smoke around the head region. References: Further information can be found at Results: Tracking of facial features Tracking of arms Acknowledgements: The Nation Science Foundation University of Central Florida, Computer Vision Lab University of Nevada-Reno, Computer Science Department University of Nevada-Reno, Office of Research