October 5, Memorial University of Newfoundland Faculty of Engineering & Applied Science Engineering 6806 Electrical & Computer Engineering Design Project INTRODUCTION TO MACHINE VISION II Prof. Nick Krouglicof
October 5, Memorial University of Newfoundland Presentation Outline 1.Machine vision systems for mechanical metrology A systematic approach to the calibration of machine vision systems for industrial metrology An automatic, machine vision based system for robot calibration 2.Industrial Applications of Machine Vision High speed, line scan camera-based inspection system for the food processing industry Vision based inspection of liquid crystal display (LCD) modules 3.CPLD based interfaces for sensors and actuators
October 5, Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology In the field of machine vision, camera calibration refers to the experimental determination of a set of parameters which describe the image formation process for a given analytical model of the machine vision system. Ideally, camera calibration is performed without specialized optical equipment, without modifications to the hardware, and without a priori knowledge of the vision system. Most calibration techniques are based on the observation of planar (2D) targets with a large number of control points.
October 5, Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology The machine vision parameters which must be identified include : a)The scale factor b)The frame buffer coordinates of the image center c)The effective focal length of the lens-camera assembly d)The radial lens distortion coefficient e)The pose (position and orientation) of the camera Parameters a) through d) are classified as intrinsic, e) as extrinsic.
October 5, Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology
October 5, Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology
October 5, Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology In general, most camera calibration techniques fail because: They fail to account for the mathematical dependency between the image center and the X-Y position of the camera. The determination of the intrinsic parameters assumes a priori knowledge of the extrinsic parameters (i.e. position and orientation of the camera). Therefore, the extrinsic parameters are generally determined first. In order to accurately evaluate the intrinsic parameters, the extrinsic parameters must be know to a degree of accuracy which is practically unattainable (6 significant digits).
October 5, Memorial University of Newfoundland A systematic approach to the calibration of machine vision systems for industrial metrology In order to compensate for the deficiencies of existing calibration techniques, a special fixture was designed to evaluate the critical parameters (i.e. image center and effective focal length) before the remaining parameters.
October 5, Memorial University of Newfoundland To locate the edge with sub-pixel accuracy, the gradient in the vicinity of the edge is reconstructed using a continuous Gaussian function. The coordinates of the pixel nearest to the edge are denoted by i and j. The location of the edge with sub-pixel accuracy is determined by calculating the offsets x and y which maximize the function s(x,y) below over a 5 by 5 kernel : A systematic approach to the calibration of machine vision systems for industrial metrology
October 5, Memorial University of Newfoundland An automatic, machine vision based system for robot calibration Objective: To develop a low cost, non-contact robot calibration system that can be applied on a routine basis without significantly affecting production schedules. Based on a CCD camera mounted on the robot’s wrist. The camera is used to determine the position and orientation of a fixed, passive target in six degrees-of- freedom. The system employs the Hayati and Mirmirani modeling convention for closed kinematic chains and non-linear least squares analysis to determine the real kinematic parameters specific to a particular robot.
October 5, Memorial University of Newfoundland An automatic, machine vision based system for robot calibration
October 5, Memorial University of Newfoundland An automatic, machine vision based system for robot calibration
October 5, Memorial University of Newfoundland An automatic, machine vision based system for robot calibration
October 5, Memorial University of Newfoundland Industrial Machine Vision Applications
October 5, Memorial University of Newfoundland High Speed, Line Scan Based Inspection System
October 5, Memorial University of Newfoundland Objective: To remove defects (I.e. visible, dark particle larger than 0.007”) from apple sauce System must be able to handle 12 metric tons per 8 hour shift System must remove 95% of visible defects Line Scan Based Inspection System: Specifications
October 5, Memorial University of Newfoundland 2 distinct challenges: Detection Removal High Speed, Line Scan Based Inspection System
October 5, Memorial University of Newfoundland High Speed, Line Scan Based Inspection System
October 5, Memorial University of Newfoundland High Speed, Line Scan Based Inspection System
October 5, Memorial University of Newfoundland Detection system based on a high performance line scan camera; 4096 pixels per line at 4800 lines per second. Image acquisition and processing functions implemented on a Complex Programmable Logic Device (CPLD) as opposed to a microprocessor or Digital Signal Processor (DSP). The objective is to implement image processing functions with dedicated logic gates (i.e. hardware) for real-time performance. Line Scan Based Detection System
October 5, Memorial University of Newfoundland What are CPLDs? Complex Programmable Logic Devices (CPLDs) are a class of programmable logic device that are commonly used to implement complex digital designs on a single integrated circuit. Applications of CPLDs in the field of computer engineering include the implementation of bus controllers, address decoders, priority encoder and state machines
October 5, Memorial University of Newfoundland Line Scan Camera-Based Detection System
October 5, Memorial University of Newfoundland Line Scan Camera-Based Detection System
October 5, Memorial University of Newfoundland Line Scan Camera-Based Detection System Typical Particle Typical section of apple sauce recorded with an area scan camera
October 5, Memorial University of Newfoundland Line Scan Camera-Based Detection System
October 5, Memorial University of Newfoundland Line Scan Camera-Based Detection System
October 5, Memorial University of Newfoundland Removal System
October 5, Memorial University of Newfoundland Removal System
October 5, Memorial University of Newfoundland Removal System: Flow Characterization Rheological nomenclature and associated velocity profiles for steady flow through tubes with circular cross section.
October 5, Memorial University of Newfoundland Viscosity Measurement
October 5, Memorial University of Newfoundland Viscosity Measurement Velocity profile can be characterized by power law!
October 5, Memorial University of Newfoundland Flow Profile of Apple Sauce
October 5, Memorial University of Newfoundland Particle Removal Window
October 5, Memorial University of Newfoundland System Timing Diagram
October 5, Memorial University of Newfoundland Aspiration Valve Characterization
October 5, Memorial University of Newfoundland Aspiration Valve Characterization
October 5, Memorial University of Newfoundland Particle Detection Rate Versus Flowrate
October 5, Memorial University of Newfoundland High Speed, Line Scan Based Inspection System: Current Status Industrial partner is currently developing the production version of the system. Packaging of the principle components (i.e., lenses, cameras, electronics, light sources) remains a major challenge given the environment. One possible solution is to integrate all the electronics in the camera enclosure. Partner is anxious to explore applications in the pulp and paper industry.
October 5, Memorial University of Newfoundland Vision Based Inspection of Liquid Crystal Display (LCD) modules Objective: to automate the inspection of LCD modules in order to improve quality control One step in the implementation of a Six-Sigma Program (“3.4 defects per million opportunities”) The inspection must be completed within 30 seconds for 10 predetermined LCD patterns System can “learn” new LCD modules without modifying software
October 5, Memorial University of Newfoundland Vision Based Inspection of LCD modules: System Components Pulnix camera with macro lens High frequency fluorescent light sources Coreco Bandit integrated image acquisition and VGA accelerator Software developed using with WiT graphical programming environment in combination with Microsoft VB
October 5, Memorial University of Newfoundland Vision Based Inspection of Liquid Crystal Display (LCD) modules
October 5, Memorial University of Newfoundland Original Image Showing Error in Alignment Vision Based Inspection of Liquid Crystal Display (LCD) modules
October 5, Memorial University of Newfoundland Thresholding Operation – Image Subtraction with respect to an image with no segments illuminated Vision Based Inspection of Liquid Crystal Display (LCD) modules
October 5, Memorial University of Newfoundland Blob Analysis – Reference Points are Identified Vision Based Inspection of Liquid Crystal Display (LCD) modules
October 5, Memorial University of Newfoundland Image Rotation and Translation Vision Based Inspection of Liquid Crystal Display (LCD) modules
October 5, Memorial University of Newfoundland Pixel by Pixel Image Subtraction from Reference Image – Thinning Operator Vision Based Inspection of Liquid Crystal Display (LCD) modules
October 5, Memorial University of Newfoundland Blob Analysis Vision Based Inspection of Liquid Crystal Display (LCD) modules
October 5, Memorial University of Newfoundland Blob Analysis Vision Based Inspection of Liquid Crystal Display (LCD) modules: Conclusions Inspection system was installed at BAE Systems Canada Ltd. where it was used to test between 200 and 600 LCD displays per day. Number of defective modules that passed inspection was basically reduced to zero. Occasional “false positives” proved to be technical problems with the devices that previously went unreported. Applications for this technology are numerous given the number of LCD displays produced annually.
October 5, Memorial University of Newfoundland CPLD based interfaces for smart sensors and actuators
October 5, Memorial University of Newfoundland Introduction Several novel “mechatronic” applications for CPLDs are presented. CPLD basically provides reconfigurable digital I/O that permits the implementation of interfaces for smart sensors and actuators. Applications that will be discussed include: Stepper motor controller Quadrature decoder/counter for optical encoder Pulse-Width Modulation (PWM) motor drives By incorporating a CPLD that supports In-System Programmability (ISP) the target device can be reprogrammed by the user for a variety of applications without removing it from the host system.
October 5, Memorial University of Newfoundland Mechatronic Applications of CPLDs – RSG System
October 5, Memorial University of Newfoundland Design Objectives of RSG System To develop a low cost, microcontroller-based system that allows the user to interface a variety of sensors and actuators to a laptop computer. In addition to being “user- friendly”, the system must be capable of functioning as either: 1.A portable data acquisition system, 2.An autonomous, stand-alone controller in support of mechatronic design projects, 3.A data logger that can acquire and store data at regular intervals over extended periods of time.
October 5, Memorial University of Newfoundland Design Philosophy Embedded Controller System incorporates an 80C52-BASIC Microcontroller Simplifies the management of the various onboard peripherals (Analog-to-Digital converter, LCD display, serial interface) Incorporates a powerful, imbedded BASIC interpreter Can also be programmed in assembler or C (Small Device C Compiler: SDCC) Assembled/compiled code can be “called” from Basic
October 5, Memorial University of Newfoundland Design Philosophy (continued) Programmable Logic System incorporates a user configurable Complex Programmable Logic Device (CPLD) CPLD supports in-system programmability by the user Mechatronic applications envisioned for the CPLD on the UC 2 system include quadrature decoder/counter interfaces for optical encoders, stepper motors controllers, PWM motor drives, sequential controllers, event counters/timers, Serial Peripheral Interface (SPI), etc.
October 5, Memorial University of Newfoundland Block Diagram of RSG System
October 5, Memorial University of Newfoundland Stepper Motor Controller: Synchronous Counter Design
October 5, Memorial University of Newfoundland Stepper Motor - Full-Stepping
October 5, Memorial University of Newfoundland Stepper Motor – Half Stepping
October 5, Memorial University of Newfoundland Stepper Motor Half-Step Switching Sequences STEP TRANSISTORS ONOFFONOFF 2ONOFF 3ONOFF ON 4OFF ON 5OFFONOFFON 6OFFONOFF 7 ON OFF 8 ONOFF
October 5, Memorial University of Newfoundland Truth Table for Half-Stepping Switching Sequence
October 5, Memorial University of Newfoundland Stepper Motor: Synchronous Counter Design
October 5, Memorial University of Newfoundland Schematic of stepper motor controller Stepper Motor: Synchronous Counter Design
October 5, Memorial University of Newfoundland Quadrature Decoder/Counter Interface for Optical Encoder: State Machine
October 5, Memorial University of Newfoundland Optical Encoder
October 5, Memorial University of Newfoundland Optical Encoder - Decoding
October 5, Memorial University of Newfoundland Optical Encoder – State Diagram
October 5, Memorial University of Newfoundland Optical Encoder – Edge Detection Circuitry
October 5, Memorial University of Newfoundland Summary of Quadrature Decoding Logic State Transition Encoder Signals Flipflop outputs: enc_decX.q Past State Present State Count Chan A Chan B UP1 DOWN 1101 Note: and indicate a rising and falling edge respectively.
October 5, Memorial University of Newfoundland Truth Table Implementation of Quadrature Decoder TABLE enc_dec[3..0].q=> dwn_cnt, up_cnt; B"0000"=>0,0; B"0001"=>0,1; B"0010"=>1,0; B"0011"=>0,0; B"0100"=>1,0; B"0101"=>0,0; B"0110"=>0,0; B"0111"=>0,1; B"1000"=>0,1; B"1001"=>0,0; B"1010"=>0,0; B"1011"=>1,0; B"1100"=>0,0; B"1101"=>1,0; B"1110"=>0,1; B"1111"=>0,0; END TABLE;
October 5, Memorial University of Newfoundland AHDL Implementation of Counter Module IF up_cnt THEN encoder[].d = encoder[].q + 1; ELSE IF dwn_cnt THEN encoder[].d = encoder[].q - 1; ELSE encoder[].d = encoder[].q; END IF;
October 5, Memorial University of Newfoundland Pulse-Width Modulation: VHDL/AHDL Programming
October 5, Memorial University of Newfoundland AHDL Implementation of PWM Controller totaltime[].clk = clk; totaltime[].clrn= VCC; totaltime[].prn = VCC; totaltime[].ena = io_space_2 & (add_latch[3..0].q == B"0110") & !write/; totaltime[].d= add_data[]; lowtime[].clk = clk; lowtime[].clrn= VCC; lowtime[].prn = VCC; lowtime[].ena = io_space_2 & (add_latch[3..0].q == B"0111") & !write/; lowtime[].d= add_data[];
October 5, Memorial University of Newfoundland cntr[].clk = clk; cntr[].clrn= !(cntr[].q == totaltime[].q); cntr[].prn = VCC; cntr[].ena= VCC; cntr[].d= cntr[].q + 1; switch.clk= clk; switch.clrn= VCC; switch.prn = VCC; switch.ena = VCC; pwm_out= switch.q; switch.d = (cntr[].q < lowtime[].q); AHDL Implementation of PWM Controller
October 5, Memorial University of Newfoundland Pedagogical Applications
October 5, Memorial University of Newfoundland Pedagogical Applications
October 5, Memorial University of Newfoundland Conclusions CPLDs are well suited to the implementation of interfaces for smart sensors and actuators. Incorporating a CPLD that supports In-System Programmability (ISP) into a data acquisition system provides reconfigurable digital I/O that allows the target system to be reprogrammed by the user for a variety of applications without hardware modifications. This is particularly useful for those applications that require hardware interfaces for several different sensors simultaneously.
October 5, Memorial University of Newfoundland QUESTIONS ?