Abstract

Surface defect identification is a crucial task in many manufacturing systems, including automotive, aircraft, steel rolling, and precast concrete. Although image-based surface defect identification methods have been proposed, these methods usually have two limitations: images may lose partial information, such as depths of surface defects, and their precision is vulnerable to many factors, such as the inspection angle, light, color, noise, etc. Given that a three-dimensional (3D) point cloud can precisely represent the multidimensional structure of surface defects, we aim to detect and classify surface defects using a 3D point cloud. This has two major challenges: (i) the defects are often sparsely distributed over the surface, which makes their features prone to be hidden by the normal surface and (ii) different permutations and transformations of 3D point cloud may represent the same surface, so the proposed model needs to be permutation and transformation invariant. In this paper, a two-step surface defect identification approach is developed to investigate the defects’ patterns in 3D point cloud data. The proposed approach consists of an unsupervised method for defect detection and a multi-view deep learning model for defect classification, which can keep track of the features from both defective and non-defective regions. We prove that the proposed approach is invariant to different permutations and transformations. Two case studies are conducted for defect identification on the surfaces of synthetic aircraft fuselage and the real precast concrete specimen, respectively. The results show that our approach receives the best defect detection and classification accuracy compared with other benchmark methods.

References

1.
Chiu
,
Y. P.
, and
Liu
,
J. Y.
,
1970
, “
An Analytical Study of the Stress Concentration Around a Furrow Shaped Surface Defect in Rolling Contact
,”
ASME J. Lubr. Technol.
,
92
(
2
), pp.
258
263
.
2.
United States
,
F. A. A.
,
1997
, “
Visual Inspection for Aircraft
,”
U.S. Dept of Transportation, Federal Aviation Administration, Advisory Circular
, pp.
43
204
.
3.
Liu
,
C.
,
Law
,
A. C. C.
,
Roberson
,
D.
, and
Kong
,
Z. J.
,
2019
, “
Image Analysis-Based Closed Loop Quality Control for Additive Manufacturing With Fused Filament Fabrication
,”
J. Manuf. Syst.
,
51
, pp.
75
86
.
4.
Tsai
,
D. M.
, and
Wu
,
S. K.
,
2000
, “
Automated Surface Inspection Using Gabor Filters
,”
Int. J. Adv. Manuf. Technol.
,
16
(
7
), pp.
474
482
.
5.
Kumar
,
A.
, and
Pang
,
G.
,
2002
, “
Defect Detection in Textured Materials Using Gabor Filters
,”
IEEE Trans. Ind. Appl.
,
38
(
2
), pp.
425
440
.
6.
Park
,
Y.
, and
Kweon
,
I. S.
,
2016
, “
Ambiguous Surface Defect Image Classification of Amoled Displays in Smartphones
,”
IEEE Trans. Ind. Inform.
,
12
(
2
), pp.
597
607
.
7.
Zhang
,
Y.
,
Lefebvre
,
D.
, and
Li
,
Q.
,
2017
, “
Automatic Detection of Defects in Tire Radiographic Images
,”
IEEE Trans. Autom. Sci. Eng.
,
14
(
3
), pp.
1378
1386
.
8.
Ngan
,
H. Y.
,
Pang
,
G. K.
,
Yung
,
S.
, and
Ng
,
M. K.
,
2005
, “
Wavelet Based Methods on Patterned Fabric Defect Detection
,”
Pattern Recognit.
,
38
(
4
), pp.
559
576
.
9.
Karimi
,
M. H.
, and
Asemani
,
D.
,
2014
, “
Surface Defect Detection in Tiling Industries Using Digital Image Processing Methods: Analysis and Evaluation
,”
ISA Trans.
,
53
(
3
), pp.
834
844
.
10.
Ng
,
M. K.
,
Ngan
,
H. Y. T.
,
Yuan
,
X.
, and
Zhang
,
W.
,
2014
, “
Patterned Fabric Inspection and Visualization by the Method of Image Decomposition
,”
IEEE Trans. Autom. Sci. Eng.
,
11
(
3
), pp.
943
947
.
11.
Yan
,
H.
,
Paynabar
,
K.
, and
Shi
,
J.
,
2017
, “
Anomaly Detection in Images With Smooth Background Via Smooth-Sparse Decomposition
,”
Technometrics
,
59
(
1
), pp.
102
114
.
12.
Yan
,
H.
,
Yeh
,
H.-M.
, and
Sergin
,
N.
,
2019
, “
Image-Based Process Monitoring Via Adversarial Autoencoder With Applications to Rolling Defect Detection
,”
2019 IEEE 15th International Conference on Automation Science and Engineering (CASE)
,
Vancouver, Canada
,
Aug. 22
, pp.
311
316
.
13.
Li
,
Y.
,
Zhao
,
W.
, and
Pan
,
J.
,
2017
, “
Deformable Patterned Fabric Defect Detection With Fisher Criterion-Based Deep Learning
,”
IEEE Trans. Autom. Sci. Eng.
,
14
(
2
), pp.
1256
1264
.
14.
Cheon
,
S.
,
Lee
,
H.
,
Kim
,
C. O.
, and
Lee
,
S. H.
,
2019
, “
Convolutional Neural Network for Wafer Surface Defect Classification and the Detection of Unknown Defect Class
,”
IEEE Trans. Semicond. Manuf.
,
32
(
2
), pp.
163
170
.
15.
Wang
,
Y.
,
Guo
,
W. G.
, and
Yue
,
X.
,
2022
, “
Tensor Decomposition to Compress Convolutional Layers in Deep Learning
,”
IISE Trans.
,
54
(
5
), pp.
481
495
.
16.
Jovancevic
,
I.
,
Pham
,
H.-H.
,
Orteu
,
J.
,
Gilblas
,
R.
,
Harvent
,
J.
,
Maurice
,
X.
, and
Brethes
,
L.
,
2017
, “
3D Point Cloud Analysis for Detection and Characterization of Defects on Airplane Exterior Surface
,”
J. Nondestruct. Eval.
,
36
(
4
), pp.
1
17
.
17.
Rao
,
P. K.
,
Kong
,
Z.
,
Duty
,
C. E.
,
Smith
,
R. J.
,
Kunc
,
V.
, and
Love
,
L. J.
,
2015
, “
Assessment of Dimensional Integrity and Spatial Defect Localization in Additive Manufacturing Using Spectral Graph Theory
,”
ASME J. Manuf. Sci. Eng.
,
138
(
5
), p.
051007
.
18.
Decker
,
N.
,
Wang
,
Y.
, and
Huang
,
Q.
,
2020
, “
Efficiently Registering Scan Point Clouds of 3D Printed Parts for Shape Accuracy Assessment and Modeling
,”
J. Manuf. Syst.
,
56
, pp.
587
597
.
19.
Xie
,
Q.
,
Lu
,
D.
,
Huang
,
A.
,
Yang
,
J.
,
Li
,
D.
,
Zhang
,
Y.
, and
Wang
,
J.
,
2021
, “
Rrcnet: Rivet Region Classification Network for Rivet Flush Measurement Based on 3-D Point Cloud
,”
IEEE Trans. Instrum. Meas.
,
70
, pp.
1
12
.
20.
Samie Tootooni
,
M.
,
Dsouza
,
A.
,
Donovan
,
R.
,
Rao
,
P. K.
,
Kong
,
Z. J.
, and
Borgesen
,
P.
,
2017
, “
Classifying the Dimensional Variation in Additive Manufactured Parts From Laser-Scanned Three-Dimensional Point Cloud Data Using Machine Learning Approaches
,”
ASME J. Manuf. Sci. Eng.
,
139
(
9
), p.
091005
.
21.
Xu
,
C.
,
Tao
,
D.
, and
Xu
,
C.
,
2013
, “
A Survey on Multi-View Learning
,” preprint arXiv:1304.5634.
22.
Sun
,
J.
,
Zhang
,
J.
,
Li
,
Q.
,
Yi
,
X.
,
Liang
,
Y.
, and
Zheng
,
Y.
,
2022
, “
Predicting Citywide Crowd Flows in Irregular Regions Using Multi-view Graph Convolutional Networks
,”
IEEE Trans. Knowl. Data Eng.
,
34
(
5
), pp.
2348
2359
.
23.
Li
,
Z.
,
Liu
,
Z.
,
Huang
,
J.
,
Tang
,
G.
,
Duan
,
Y.
,
Zhang
,
Z.
, and
Yang
,
Y.
,
2019
, “
MV-GCN: Multi-view Graph Convolutional Networks for Link Prediction
,”
IEEE Access
,
7
, pp.
176 317
176 328
.
24.
Xia
,
W.
,
Wang
,
Q.
,
Gao
,
Q.
,
Zhang
,
X.
, and
Gao
,
X.
,
2022
, “
Self-Supervised Graph Convolutional Network for Multi-view Clustering
,”
IEEE Trans. Multimedia
,
24
, pp.
3182
3192
.
25.
Makuch
,
M.
, and
Gawronek
,
P.
,
2020
, “
3D Point Cloud Analysis for Damage Detection on Hyperboloid Cooling Tower Shells
,”
Remote Sens.
,
12
(
10
), p.
1542
.
26.
Su
,
H.
,
Maji
,
S.
,
Kalogerakis
,
E.
, and
Learned-Miller
,
E.
,
2015
, “
Multi-view Convolutional Neural Networks for 3D Shape Recognition
,”
2015 IEEE International Conference on Computer Vision (ICCV)
,
Santiago, Chile
,
Dec. 7
, pp.
945
953
.
27.
Wu
,
Z.
,
Song
,
S.
,
Khosla
,
A.
,
Yu
,
F.
,
Zhang
,
L.
,
Tang
,
X.
, and
Xiao
,
J.
,
2015
, “
3D Shapenets: A Deep Representation for Volumetric Shapes
,”
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
,
Boston, MA
,
June 8
, pp.
1912
1920
.
28.
Charles
,
R. Q.
,
Su
,
H.
,
Kaichun
,
M.
, and
Guibas
,
L. J.
,
2017
, “
Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation
,”
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
,
Honolulu, HI
,
July 21
, pp.
77
85
.
29.
Wang
,
Y.
,
Sun
,
Y.
,
Liu
,
Z.
,
Sarma
,
S. E.
,
Bronstein
,
M. M.
, and
Solomon
,
J. M.
,
2019
, “
Dynamic Graph CNN for Learning on Point Clouds
,”
ACM Trans. Graph.
,
38
(
5
), pp.
1
2
.
30.
Eppstein
,
D.
,
Paterson
,
M. S.
, and
Yao
,
F. F.
,
1997
, “
On Nearest-Neighbor Graphs
,”
Discrete Comput. Geom.
,
17
(
3
), pp.
263
282
.
31.
Hauberg
,
S. R.
,
Freifeld
,
O.
, and
Black
,
M.
,
2012
, “
A Geometric Take on Metric Learning
,”
Advances in Neural Information Processing Systems (NeurIPS)
,
Lake Tahoe, NV
,
Dec. 3
, pp.
2024
2032
.
32.
Crane
,
K.
,
Weischedel
,
C.
, and
Wardetzky
,
M.
,
2017
, “
The Heat Method for Distance Computation
,”
Commun. ACM
,
60
(
11
), pp.
90
99
.
33.
Pauly
,
M.
,
Gross
,
M.
, and
Kobbelt
,
L. P.
,
2002
, “
Efficient Simplification of Point-Sampled Surfaces
,”
Proceedings of the Conference on Visualization ’02, VIS ’02
,
Boston, MA
,
Oct. 27
, pp.
163
170
.
34.
Guijt
,
C. B.
, and
Donne
,
C. D.
,
2005
, “
The Effect of Dents in Fuselage Structures on Fatigue and Static Stability
,”
Proceedings of the Symposium of the International Committee on Aeronautical Fatigue (ICAF)
,
Hamburg, Germany
,
June 6
,
Vol. 2, pp. 417–428
.
35.
Lang
,
N.
, and
Kwon
,
Y.
,
2007
, “
Investigation of the Effect of Metallic Fuselage Dents on Compressive Failure Loads
,”
J. Aircr.
,
44
(
6
), pp.
2026
2033
.
36.
Kanungo
,
T.
,
Mount
,
D.
,
Netanyahu
,
N.
,
Piatko
,
C.
,
Silverman
,
R.
, and
Wu
,
A.
,
2002
, “
An Efficient K-Means Clustering Algorithm: Analysis and Implementation
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
24
(
7
), pp.
881
892
.
37.
Zepeda-Mendoza
,
M. L.
, and
Resendis-Antonio
,
O.
,
2013
,
Hierarchical Agglomerative Clustering
,
Springer
,
New York
, pp.
886
887
.
38.
Hoppe
,
H.
,
DeRose
,
T.
,
Duchamp
,
T.
,
McDonald
,
J.
, and
Stuetzle
,
W.
,
1992
, “
Surface Reconstruction From Unorganized Points
,”
Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’92
,
Chicago, IL
,
July 1
, pp.
71
78
.
39.
Paszke
,
A.
,
Gross
,
S.
,
Massa
,
F.
,
Lerer
,
A.
,
Bradbury
,
J.
,
Chanan
,
G.
, and
Killeen
,
T.
,
2019
, “
Pytorch: An Imperative Style, High-Performance Deep Learning Library
,”
Advances in Neural Information Processing Systems (NeurIPS)
,
Vancouver, Canada
,
Dec. 8
, pp.
8026
8037
.
40.
Wang
,
R.
,
Wang
,
Y.
,
Devadiga
,
S.
,
Perkins
,
I.
,
Kong
,
Z. J.
, and
Yue
,
X.
,
2021
, “
Structured-Light Three-Dimensional Scanning for Process Monitoring and Quality Control in Precast Concrete Production
,”
PCI J.
,
66
(
6
), pp.
17
32
.
41.
Lee
,
C.
,
Wang
,
X.
,
Wu
,
J.
, and
Yue
,
X.
,
2022
, “
Failure-Averse Active Learning for Physics-Constrained Systems
,”
IEEE Trans. Autom. Sci. Eng
.
You do not currently have access to this content.