1. Use 3D Convolutional Neural Network to Inspect Solder Ball Defects
Presenter：Bing-Jhang Lin 林炳彰
Phone：

2. Outline
Introduction
Methods
Experiments

3. Introduction Head-in-Pillow (HIP) is a solder ball defect. HIP pattern

4. Introduction The HIP defect can be caused by the following reasons:
Surface of solder ball oxidation
Poor wetting of the solder
Distortion of the Printed Circuit Board (PCB) by the heat of the soldering process
表面氧化
錫料濕潤不良
因加熱而變形

5. Introduction
The current diagnosis of the HIP defects usually uses 2D X- Ray inspection machine, but it is hard to find out the position of the defect.

6. Introduction
Some reliable methods are destructive tests, so they are not recommended for use.

7. Introduction
Although it is hard to find out the HIP location from the 2D X-Ray image, we can reconstruct the 3D PCB model by different angles of the 2D X-Ray images.

8. Introduction
The 3D PCB model not only provides us with more detailed information but also represents the location of HIP defects more clearly.
In recent years, Deep Learning has been widely used in many computer vision tasks, especially the Convolutional Neural Network (CNN) has an outstanding performance in object recognition.

9. Introduction
In this project, we propose a 3D CNN to inspect the HIP defects.
Our network combines the advantages of many state-of-art CNNs like: VGG, GoogleNet Inception, ResNet, and DenseNet.
We also design two types of the convolutional blocks and use the dense connectivity method to connect these blocks.

10. Introduction
Because the HIP problems do not happen often, so we use the data augmentation method to expand our 3D solder ball data.
In our experiments, our network can achieve excellent results on testing data; the size of our network is more refined; and the training and execution time is faster than other CNNs.

11. Methods 3D Solder Balls Volumetric Data
3D Convolutional Neural Network

12. 3D Solder Balls Volumetric Data
The 3D PCB volumetric data are reconstructed by different angles of 2D X-Rays images.
In order to obtain clearer 3D PCB volumetric data during the reconstruction, we control the camera to take a 2D X- Rays image every 2.81 degrees.

13. 3D Solder Balls Volumetric Data

14. 3D Solder Balls Volumetric Data
The 3D PCB volumetric data size is 1472 × 1176 × 46 pixels.
Since the volumetric pattern of solder ball is an elliptical sphere, the middle slice of the volumetric contains the biggest solder ball cross section.
Elliptical: 橢圓的
Sphere: 球

15. 3D Solder Balls Volumetric Data
The maximum bounding box size of the solder ball from the middle slice of the PCB volumetric data is 46 × 46 pixels, so the size of each 3D solder ball is 46 × 46 × 46 pixels.

16. 3D Solder Balls Volumetric Data
We totally received 277 3D solder balls.

17. 3D Solder Balls Volumetric Data
For an image object recognition task, the amount of data is not sufficient, so we also use the Data Augmentation method to obtain more data.

18. 3D Solder Balls Volumetric Data
We rotate the 3D solder ball data six times to obtain more data.

19. Methods Middle slice pattern in different plane. z y x xz Plane
yz Plane
xy Plane
Diagonal
(Right Rear to Left Front)
Diagonal
(Left Rear to Right Front)
3D Solder Ball

20. 3D Convolutional Neural Network
Our network:
Input
Convolution
3×3×3
Stride: 2, ReLU: True
Dense Connection Block
Num of Connection: 2
Convolution
1×1×1
Stride: 2, ReLU: True
Global Average Pooling
(GAP)
× 4
Softmax
Dropout

21. 3D Convolutional Neural Network
Our network contains four dense connection blocks.
The block can fully utilize the features of each layer and avoid some important features lost in deep layers.
Because the data may in different size, we add the Global Average Pooling (GAP) layer before the last layer to let our network can adapt different sizes of input.

22. 3D Convolutional Neural Network
Dense Connection Blocks:
Input
Convolution Block
(Inception)
Concatenate
ReLU
Convolution Block
(Inception)
Concatenate
ReLU

23. 3D Convolutional Neural Network
We use the dense connectivity method to connect our convolutional blocks.
The output of previous layer will be connected with the current output of the convolutional block and do the ReLU activation function.
The features of the input will be fully used in the deep network and avoid some important information loss in the deep network.

24. 3D Convolutional Neural Network
Inception Block:
Convolution
3×3×3
Stride: 1
Input Tensor
Max Pooling
3×3×3
Stride: 1
Convolution
1×1×1
Stride: 1
Concatenate
Average Pooling
3×3×3
Stride: 1
Convolution
1×1×1
Stride: 1

25. 3D Convolutional Neural Network
Inception with Residual Block:
Convolution
3×3×3
Stride: 1
Max Pooling
3×3×3
Stride: 1
Convolution
1×1×1
Stride: 1
Concatenate
Input Tensor
Avg Pooling
3×3×3
Stride: 1
Convolution
1×1×1
Stride: 1
Convolution
1×1×1
Stride: 1
ReLU
Convolution
3×3×3
Stride: 1
ReLU
Convolution
1×1×1
Stride: 1

26. 3D Convolutional Neural Network
The convolutional blocks are composed of many different convolutional paths.
The different convolutional paths allow the following layer to learn various features, so the network can choose better features to adapt the data.
The residual method is a good way to prevent gradient vanishing problem, so we can more easily increase the depth of the network.

27. Experiments
Solder Ball Data
Experiments

28. Solder Ball Data
We currently have 277 balls of 3D solder, of which 38 are labeled data, and the rest are unlabeled data.
The volumetric size of each solder ball is 46 × 46 × 46 × 1 pixels. (The last channel represents the grayscale channel.)
Half of labeled data (19) are defective solder balls, and the other are normal solder balls.

29. Solder Ball Data
We split the labeled data into two parts: training and testing.
The training part has 32 solder balls and the testing part has 6 solder balls.
We rotate to expand the labeled data 6 times, so the training part has 192 (=32 × 6) solder balls, and the testing part has 36 (=6 × 6) solder balls.

30. Solder Ball Data
Because the defective solder balls do not often appear on the PCB, so that the 239 unlabeled data are temporarily considered as normal solder balls in evaluating stage.

31. Experiments
The information of each models.

32. Experiments
The Result on testing data.

33. Experiments
The Result on unlabeled data.

34. Experiments
When the patterns of these solder balls are skewed near PCB border, we find that the models may misclassify the normal solder balls into defect.
The skew situation often happens on the PCB border, and the pattern of skew solder ball is blurry which may cause misclassification.

35. Experiments