Artificial Intelligence-Based Smart Quality Inspection for Manufacturing

Article Content

Abstract

In today’s era, monitoring the health of the manufacturing environment has become essential in order to prevent unforeseen repairs, shutdowns, and to be able to detect defective products that could incur big losses. Data-driven techniques and advancements in sensor technology with Internet of the Things (IoT) have made real-time tracking of systems a reality. The health of a product can also be continuously assessed throughout the manufacturing lifecycle by using Quality Control (QC) measures. Quality inspection is one of the critical processes in which the product is evaluated and deemed acceptable or rejected. The visual inspection or final inspection process involves a human operator sensorily examining the product to ascertain its status. However, there are several factors that impact the visual inspection process resulting in an overall inspection accuracy of around 80% in the industry. With the goal of 100% inspection in advanced manufacturing systems, manual visual inspection is both time-consuming and costly. Computer Vision (CV) based algorithms have helped in automating parts of the visual inspection process, but there are still unaddressed challenges. This paper presents an Artificial Intelligence (AI) based approach to the visual inspection process by using Deep Learning (DL). The approach includes a custom Convolutional Neural Network (CNN) for inspection and a computer application that can be deployed on the shop floor to make the inspection process user-friendly. The inspection accuracy for the proposed model is 99.86% on image data of casting products.

Keywords:

artificial intelligence; deep learning; quality control; visual inspection; industry 4.0; smart manufacturing; image recognition; defect detection

PDF

1. Introduction

The manufacturing and production industry has undergone a fundamental transformation over the past century. Strategies such as Total Quality Management (TQM), Six Sigma, Lean, and Zero-Defect Manufacturing have pushed for higher yields while lowering costs. More recently, approaches such as Industry 4.0, Cyber-Physical Systems and Smart Manufacturing are allowing an interconnected shop floor environment with developments in robotics and automation. While larger enterprises have the economic prowess to implement transformative changes to the industry, Small and Medium-sized Enterprises (SMEs) are often unable to do so. When it comes to overall Quality Management (QM), SMEs face challenges due to the following reasons: there is a lack of information about the importance of QM, they face resource constraints, and there is an inadequacy in standards for SMEs [1,2]. Artificial Intelligence (AI) is tackling this with the development of novel algorithms, and with the help of low-cost sensors and computing services. The capabilities of the manufacturing paradigm are being enhanced by Machine Learning (ML), Pattern Recognition (PR), and Deep Learning (DL). Applications of AI are increasingly enabling manufacturers across the board to identify faulty components in their systems, detect defective products, and in some instances, diagnose them. On the system side, the area of Prognostics and Health Management (PHM) is transforming approaches to maintenance. On the product side, AI helps in Quality Control (QC) measures. While these systems are modernizing many of the manufacturing processes, there still seems to be a reliance on human operators for some critical decision-making steps on the shop floor. Quality inspection is one such process.

Quality inspection is a planned and organized process in which the state of the product is assessed by examination, measurement, testing, gauging, or comparison to determine if it conforms to desired specifications [3]. In most cases, quality inspection involves a human operator that inspects the product to ascertain its conformity. However, the accuracy and reliability of the inspection are often unsatisfactory. According to Harris [4], as the complexity of the product increases, the accuracy of inspection conducted by operators decreases. Similarly, in a study conducted by the Sandia National Labs [5], the accuracy of correctly rejecting precision manufactured parts by human operators was found to be 85% while the industry average was 80%. Another recent study concluded that operator errors accounted for 23% of the inaccuracies in quality control in the oil and gas industry [6]. Over the years, computer vision-based systems have been incorporated into the inspection process of various products such as disk heads, steel strips [7], syringes [8] and semiconductors [9]. Vision-based systems generally consist of an algorithm that is taught to identify discrepancies between features of the product undergoing inspection and the desired features. Although these systems help in automating the inspection process to a certain extent, there are still some challenges to their implementation on the shop floor [10]. There are also numerous works in which ML and DL-based models are applied to the quality inspection process. However, a lot of researchers focus on improving the performance of models and do not consider a holistic approach to inspection. While that is one of the main goals of the inspection process, there are several factors that affect the inspection process that go unaddressed. There is also a need for a methodology or approach that establishes how a data-driven method can be deployed on the shop floor in a user-friendly and hassle-free manner.

To that end, this paper proposes a twofold approach to the quality inspection process. First, a DL algorithm based on a custom Convolutional Neural Network (CNN) architecture is constructed for defect detection, and then a shop floor tool is designed to deploy the inspection model. The rest of the paper is structured as follows. Section 2 discusses the role of health monitoring in manufacturing from the perspective of the system and the product. Section 3 reviews the state of quality inspection and visual inspection, and outlines some of the key factors that affect the inspection process. Section 4 provides an overview of the casting operation in manufacturing and the challenges it presents in the inspection process. Section 5 discusses the role of DL in the inspection and how DL has automated some of the steps in quality inspection. Section 6 highlights the state-of-the-art in visual inspection, identifies the research gap and objectives of our work, and Section 7 describes the research process and proposes a methodology for AI-based Smart Quality Inspection. Section 8 describes the casting product data used in this research. Section 9 explains the modelling of the algorithm and the design of the shopfloor application. Section 10 presents the results of Smart Quality Inspection on casting products. Section 11 provides concluding remarks and outlines the objectives of future work.

2. Health Monitoring in Manufacturing

It would be remiss to discuss quality inspection, a crucial component of the QC process, without providing a context about the health of the entire manufacturing environment. There are two major areas of health monitoring in manufacturing. The first concerns the system’s health, ensuring that the machinery and equipment are functioning satisfactorily. The second area pertains to monitoring the product’s health throughout its lifecycle. For the system, PHM deals with component-level and system-level health monitoring. On the product side, QC techniques are relied upon to guarantee the health and quality of the product. Figure 1 portrays the different phases of PHM and QC in the context of health monitoring in manufacturing. The rest of this section presents brief overviews of these topics.

Figure 1. Health Monitoring of the manufacturing environment with PHM and QC.

2.1. Prognostics and Health Management (PHM)

Prognostics and Health Management (PHM) is a discipline that monitors the system’s health, detects failures, diagnoses failures, and predicts the Remaining Useful Life (RUL) of components [11]. Using the Internet of Things (IoT) -powered sensors and field devices, operating conditions of critical tools and components can be monitored in real-time. Nowadays, the availability of low-cost embedded devices and microcontrollers such as Raspberry Pi, STM32, Arduino, etc. enable SMEs to incorporate PHM on the shop floor. Once sufficient data has been collected, the health indexes and metrics developed can be used in models to predict failures of components and provide Remaining Useful Life (RUL) estimates. In the past, data-driven approaches to PHM have relied on ML models. However, successful implementations of ML models for prognostics or failure detection are often reliant upon expert knowledge to extract meaningful characteristics or features from the data [12]. DL techniques have the capabilities to automatically extract high-level features from inputs such as acoustic signals, vibration signals, image data, etc. Hence, there is an advantage to using DL for prognostics and diagnostics applications. A detailed methodology developed in [11] reviews the various approaches to PHM: data-driven, physics-based, and hybrid and use-case on health monitoring of a milling machine tool.

2.2. Quality Control (QC)

With the rise in modern manufacturing systems and the development of highly complex products, quality management has taken an essential role in organizational planning and strategies [13]. Quality Control (QC) is a process that involves setting quality standards, ensuring that the product meets those standards, and improving the overall product quality. Quality Inspection is a part of the QC process in which the product is inspected by operators during the various stages of manufacturing. Figure 1 depicts the QC process from the perspective of health monitoring, which involves product inspection. The overall QC process is continuously changing due to the dynamic nature of the manufacturing environment. Approaches such as Design of Experiments (DoE), Failure Mode and Effects Analysis (FMEA), Quality Function Deployment (QFD) and Acceptance Sampling prescribe their own methodologies for product inspection. While these approaches have been very successful in QC, there is an opportunity to assess 100% of the products on the modern shop floor. Continuous assessment of product quality has become a reality with developments in sensor networks and AI.

3. Overview of Quality Inspection

3.1. Quality Inspection Process

The traditional quality improvement process is cyclical—it involves generating inspection plans, implementing the plans, and checking the results [14]. Similarly, the inspection process is comprised of inspection plans that identify the different areas of manufacturing where inspection is required. It typically begins with the inspection of raw materials—also known as incoming or receiving inspection. Then there are inspections conducted periodically after various operations. The nature of these inspections is industry-specific in most cases. For instance, the inspection of structural steel products would differ greatly from the inspection of microcontrollers. At the end of the assembly line, a final inspection is conducted—where it is determined whether the product is acceptable or is to be rejected. This is analogous to outgoing inspection. In some cases, outgoing inspection refers to the inspection of the packaged product during shipping.

The inspection process is an important decision process in the manufacturing/production system [15]. According to the Signal Detection Theory (SDT), probabilistic decisions are made at every step by the decision maker (operator) to determine whether the product is to be accepted or rejected [16]. Inspection is not an independent process in the manufacturing value chain but impacts many other operations. The decision-making process for inspection involves multiple elements and should display the following characteristics as noted by [15]:

Precision: The decisions made should be well-informed, to ensure that there are no biases or errors.
Validity: Decisions made must be valid and must not differ if the product were to be available for use.
Reliability: There must be consistency in the decisions made—repeatability and reproducibility. The decision process should not require recalibration.
Robustness: The decision-making must demonstrate versatility in detecting different types of defects.
Rapidness: The process must be quick and must be able to act before any more defective products are produced.

Note that the above characteristics are desired from all inspection processes regardless of whether it is conducted by either human operators or by some form of automation.

3.2. Visual Inspection

An important type of quality inspection in manufacturing is visual inspection. Operators visually assess the state of the product at different stages and decide whether it can be moved on to the next process. Sinclair [17] suggested a four-step visual inspection operation comprising the following tasks:

Present: Present the product for inspection.
Search: Examine and analyze the product for possible flaws/defects.
Decision: Assess the flaws/defects and determine if it falls out of the desired specifications.
Action: Accept or reject the item based on the decision.

Similarly, Wang and Drury [18] characterized the visual inspection process as having a number of sub-tasks or activities: (1) orient the item, (2) search the item, (3) detect the defects/flaws, (4) recognize and classify the flaws/defects, (5) make a decision about the item, (6) dispatch the item, and (7) record any information about the item. In both approaches, the goal of the visual inspection process is to identify defects efficiently and accurately and make decisions accordingly.

3.3. Factors Affecting Visual Inspection

Any inspection process or system requires some form of human action. There cannot be a system that is entirely automated or manual [15]. Inspection involves a lot of mental effort, attention to detail, communication, and the usage of long-term and short-term memory [19]. In most cases, inspection is also required to be done quickly, i.e., defects must be identified swiftly before a decision is made. With human involvement, there arise several factors that could affect or impede the efficient implementation of visual inspection. According to research conducted by Peters et al. [20] and See et al. [21], some of the known factors that impact inspection can be categorized into task factors, environmental actors, operator or individual factors, organizational factors, and social factors.

Task factors refer to the manual and physical aspects of the inspection task. The task itself can affect the operator and influence their performance. Environmental factors can also significantly impact the outcome of visual inspection. Factors such as temperature, humidity, lighting, etc. can make the environment unsuitable which in turn influences the operator’s ability to conduct the inspection. Operator or individual factors refer to features such as an operator’s physical and mental attributes. Physical attributes could be an operator’s vision, visual acuity, gender, etc. Mental attributes could be their state of mind, aptitude, personality, biases, etc. Organizational factors concern the administration and management under which the inspection process takes place. It also includes the organizational importance given to quality inspection and visual inspection, training provided, etc. Social factors include relationships that the operator has with their peers and management, whether communication in their working environment is effective or not, and the other aspects of the social environment in which the inspection task occurs. A synopsis of all the factors that affect the visual inspection process based on [20,21] is provided in Table 1.

Table 1. Factors that affect visual inspection.

4. Casting Process

The manufacturing process of casting usually involves pouring liquefied metal into the cavity of a mold that is of the desired shape [22]. There are different types of casting processes in manufacturing. The type of process is dependent on the materials (mostly metals) used to manufacture the final product.

4.1. Types of Casting Processes

Some of the types of casting processes are listed with the types of materials that are used [23]:

(a): Sand Casting—most metal types
(b): Investment Casting—most metal types
(c): Resin Shell Molding—Primarily Iron and Copper
(d): Gravity Die Casting—Primarily Aluminum, Zinc, Magnesium, Copper, and some of their alloys
(e): Low-Pressure Die Casting—Primarily Aluminum and Magnesium
(f): High-Pressure Die Casting—Primarily Aluminum, Magnesium, and Zinc
(g): Squeeze Casting—Primarily Aluminum

4.2. Steps in the Casting Process

Most of the casting processes listed in Section 4.1 generally follow similar steps to go from the raw material to the finished product. A list of these steps is as follows [22]:

Patternmaking—Designing and preparing a pattern
Preparing the mold that is approximately the same shape/size as the desired pattern
Identifying the material to be used in casting (usually metals or allows)
Liquefying the material in a furnace
Pouring the liquefied metal into the cavity of the mold
Opening the mold to access the casting
Fettling—removing excess material, surface cleaning, and finishing
Heat treatment based on requirements
Final inspection

4.3. Inspection in the Casting Process

The quality inspection of casting products is the most critical step in determining whether the product is acceptable for use or must be rejected and scrapped/reworked. There are a few types of inspections for casting products: visual inspection, dimensional inspection, metallurgical inspection, chemical and physical inspection, and other methods involving Non-Destructive Testing (NDT) [24]. While multiple inspection methods are used concurrently during different stages of the casting process, we shall limit the discussion to a brief on visual inspection methods.

This visual inspection process for casting generally involves the examination of the product by an operator or a group of operators. Operators look for surface defects, cracks, tears, molding flaws, scabs, blowholes, runouts, adhesions, and various other types of defects [25]. Many of the defects can be attributed to flaws in mold design, the incorrect composition of materials used in mold construction, the equipment used in pouring liquefied metals into the molds, etc. Some visual inspection processes for casting products can be automated. Vision-based inspection systems rely on software for color matching and in some instances contour matching and dimension checking [26]. In recent years, ML and DL techniques have been used to perform visual inspections of casting products.

5. Deep Learning for Quality Inspection

Owing to the numerous factors that can affect an operator during the visual inspection process, data-driven approaches are being used increasingly to detect defective products. While traditional ML methods often require domain knowledge in the feature generation or feature engineering process, DL methods can automatically select and learn abstract features [27]. DL methods based on CNN, Autoencoders, and Recurrent Neural Networks (RNN) provide excellent results on a variety of inspection applications [28].

Chang et al. [29] apply a deep ensemble learning model to inspect defects on car body surfaces. Their method outperforms human inspectors in performing the same task. Researchers in [30] use a CNN to identify defects in textured surfaces. Results show high accuracy of defect detection on a multi-class dataset. For defect identification in semiconductor manufacturing, Imoto et al. [31] use a transfer learning approach based on CNN while Lee et al. [32] propose a CNN model that is receptive to time series data. In the inspection of sewer systems, Kumar et al. [33] propose deep CNNs. They use image data with high variation and claim that the CNN-based methods outperform other methods requiring manual feature extraction. For the inspection of laser welding, Yang et al. [34] use an optimized VGG model. A transfer learning approach is used where the VGG model is pre-trained on a large variety of images. Ullah et al. [35] propose an approach that uses a pre-trained AlexNet for feature extraction combined with Random Forest (RF) and Support Vector Machines (SVM) for defect detection. Their proposed method outperforms LeNet and VGG algorithms in an experiment conducted on hig- voltage electrical equipment. To inspect rivet joints in aircraft products, Amosov et al. [36] apply YOLOv5 and MobileNetv3 to images. In binary classification and in the multi-class scenario, they achieve very high accuracy of defect detection. To inspect the gas lighter manufacturing process, researchers in [37] develop a DL model based on YOLOv4. Results show good performance in detecting defects with changing illuminance and distance.

6. State-of-the-Art in Visual Inspection and Research Gap

There are several works that have implemented image-based quality inspection methods for defect detection in manufacturing products. He et al. [38] use CNN for defect detection on product surfaces at the pixel level. A convolutional variational autoencoder is proposed by Yun et al. [39] to study a multi-class surface defect identification problem on metals. In the case of welding products, Sassi et al. [40] apply a transfer learning approach and achieve a good performance on a small dataset. For inspection of casting products, Oborski et al. [41] use a CNN model in a holonic shopfloor based setting. In detecting defects of welded nuts, Lee et al. [42] use a model based on VGG-16. They conducted experiments with CNN models before achieving the desired performance with the VGG-16 model. Some more state-of-the-art defect detection methods in manufacturing are summarized in Table 2.

Table 2. Defect detection methods in manufacturing.

These research works achieve good performance on real-world data but do not provide a holistic approach to deploying DL-based models to the shop floor. While most of the works that apply AI to visual inspection focus on improving the model performance, very few of them take into consideration the various factors involved in the visual inspection process. An even fewer attempt to minimize these factors in their proposed methods. Factors outlined in Section 3.3 are crucial considerations while designing a visual inspection system. An inspection system should attempt to minimize the maximum number of the factors out of the task, environmental, operator, organizational, and social factors. Additionally, we have learned that even with automation, the inspection process will involve some form of human participation. Based on these arguments, there is a need for a visual inspection system that

performs well in detecting defects, i.e., shows high accuracy, precision, recall, etc.,
minimizes the factors affecting the visual inspection process, and
allows documentation of decisions made.

Considering the above requirements, we propose Smart Quality Inspection—an AI-based approach to the visual inspection process and demonstrate a use case on a benchmark image dataset from a casting process.

7. Smart Quality Inspection

The Smart Quality Inspection (SQI) approach aims to improve model performance and address several factors that affect the visual inspection process. By automating the inspection process to an extent, the effects of many of the task factors, environmental factors, and individual factors can be controlled. Figure 2 displays a flowchart that depicts the process used to develop SQI. The research gaps identified via the literature review process tie in directly with the development of the inspection algorithm. The approach outlined to develop SQI is informative in proposing the methodology to implement AI-based visual inspection on the shopfloor. Figure 3 shows the different stages involved in implementing SQI in the manufacturing/production area. There are a total of six stages—from receiving the product at the inspection area to inspecting it using AI and documenting the results. The processes and steps involved in each of the stages are described below.

Figure 2. Flowchart depicting the process used to develop Smart Quality Inspection—algorithm construction and identification of research gap via literature review.

Figure 3. Artificial Intelligence based Smart Quality Inspection Methodology.

Stage 1: Manufacturing product arrives at the inspection area:

In the first stage, the product from the assembly line is brought to the inspection area. The item is placed in a designated location to allow the inspection process to begin.
Stage 2: Product image is captured

In this stage, a high-quality camera is used to capture images of the product undergoing inspection. The lighting conditions and distance from the product are measured based on the product size and camera equipment in use.
Stage 3: Image preprocessing

It is identified if grayscale or color images would be appropriate based on the availability of computational resources and desired precision and accuracy of predictions. Any augmentation or transformation is done at this stage—flips, shears, rotation, shifts, whitening, contrast adjustment, etc.
Stage 4: CNN-based defect detection

A custom CNN architecture is used to detect defects in images. The architecture has the versatility to handle different types of images with just a small number of changes. The model is trained on images of defective products and non-defective products to learn the necessary feature representations. The defect detection model is built into an application that can be used on the shop floor to make the inspection process trouble-free.
Stage 5: Decision stage—accept/reject the product

The operator inspects the product using the defect detection algorithm and instantaneously receives the inspection results from the computer application. Based on the results, a decision is made whether to accept or reject the product.
Stage 6: Document results in the inspection log

The results of the inspection process are input into the SQI shop floor application and are automatically stored in a spreadsheet.

Related Articles

Contact us