Benchmarking Nine SMOTE-Balanced Classifiers Including Artificial Neural Network for CNC Predictive Maintenance

Didiek Trisatya; Priyo Haryoko

doi:10.22441/ijimeam.v8i1.38645

Authors

Didiek Trisatya Universitas Pancasakti Tegal, Indonesia https://orcid.org/0000-0002-8092-6983
Priyo Haryoko Universitas Pancasakti Tegal, Indonesia

DOI:

https://doi.org/10.22441/ijimeam.v8i1.38645

Keywords:

predictive maintenance, CNC manufacturing, SMOTE oversampling, machine learning, class imbalance, gradient boosting

Abstract

Unplanned equipment failure in CNC manufacturing causes significant economic losses, driving demand for effective predictive maintenance (PdM). A critical research gap persists: existing studies on the AI4I 2020 Predictive Maintenance Dataset apply isolated classifiers under inconsistent preprocessing pipelines, preventing fair algorithmic comparison. No prior study has benchmarked nine diverse classifier families under a unified pipeline integrating SMOTE oversampling with domain-driven feature engineering. This study addresses that gap by systematically evaluating nine ML classifiers—Logistic Regression, K-Nearest Neighbors, Decision Tree, Random Forest, Gradient Boosting, AdaBoost, SVM (RBF kernel), Naive Bayes, and MLP Neural Network—on the AI4I 2020 dataset (10,000 records; 3.4% failure rate; 1:28 class imbalance). Two domain-engineered features were constructed: mechanical power (P = n × T × (π/30)) and thermal gradient (ΔT = T_process - T_air). Features were normalized; SMOTE was applied to training folds only; and 10-fold stratified cross-validation assessed six performance metrics. Three novel contributions are presented: (1) the first nine-classifier benchmark on AI4I 2020 under a unified SMOTE-and-feature-engineering pipeline enabling fair model comparison; (2) empirical demonstration that Average Precision is a more discriminating evaluation metric than AUC-ROC under severe 1:28 class imbalance; and (3) physical interpretation of feature importance linking dominant predictors to CNC failure mechanisms. Gradient Boosting achieved the best-balanced performance (F1-score: 0.6782, Accuracy: 97.20%, AUC-ROC: 0.9723); Random Forest attained the highest AUC-ROC (0.9772). Mechanical power (25.51%) and tool wear (23.91%) were dominant predictors, corresponding to tribological, fatigue loading, and thermal failure mechanisms. These findings support cost-effective condition-based maintenance strategies in industrial CNC environments.

Downloads

Download data is not yet available.

References

[1] I. Hector and R. Panjanathan, "Predictive maintenance in Industry 4.0: A survey of planning models and machine learning tech-niques," PeerJ Comput. Sci., Art. no. 10:e2016, 2024, doi: 10.7717/peerj-cs.2016.

[2] N. Grabill, S. Wang, H. A. Olayinka, T. P. De Alwis, Y. F. Khalil, and J. Zou, "AI-augmented failure modes, effects, and criticality analy-sis (AI-FMECA) for industrial applications," Reliab. Eng. Syst. Saf., vol. 250, Art. no. 110308, 2024, doi: 10.1016/j.ress.2024.110308.

[3] M. Soori, B. Arezoo, and R. Dastres, "Internet of Things for smart factories in Industry 4.0: A review," Internet Things Cyber-Phys. Syst., vol. 3, pp. 192-204, 2023, doi: 10.1016/j.iotcps.2023.04.006.

[4] O. Fink, Q. Wang, M. Svensen, P. Dersin, W.-J. Lee, and M. Ducoffe, "Potential, challenges and future directions for deep learning in prognostics and health management applications," Eng. Appl. Artif. Intell., vol. 92, Art. no. 103678, 2021, doi: 10.1016/j.engappai.2020.103678.

[5] L. Polverino, R. Abbate, P. Manco, D. Perfetto, F. Caputo, R. Macchiaroli, and M. Caterino, "Machine learning for prognostics and health management of industrial mechanical systems and equipment: A systematic literature review," Int. J. Eng. Business Manag., vol. 15, 2023, doi: 10.1177/18479790231186848.

[6] M. Soori, B. Arezoo, and R. Dastres, "Machine learning and artificial intelligence in CNC machine tools: A review," Sustain. Manuf. Serv. Econ., vol. 2, Art. no. 100009, 2023, doi: 10.1016/j.smse.2023.100009.

[7] H. A. B. Alfarizi, J. Vatn, and S. Yin, "An extreme gradient boosting aided fault diagnosis approach: A case study of fuse test bench," IEEE Trans. Artif. Intell., vol. 3, no. 6, pp. 974-983, Dec. 2022, doi: 10.1109/TAI.2022.3165599.

[8] M. Zare, P. Kebria, A. Khosravi, and S. Nahavandi, "An investigation of SMOTE based methods for imbalanced datasets with data complexity analysis," IEEE Trans. Knowl. Data Eng., vol. 35, no. 6, pp. 6106-6120, Jun. 2023, doi: 10.1109/TKDE.2022.3179381.

[9] S. N. Tang, J. T. Ma, Z. Q. Yan, Y. Zhu, and B. C. Khoo, "Deep transfer learning strategy in intelligent fault diagnosis of rotating ma-chinery," Eng. Appl. Artif. Intell., vol. 134, Art. no. 108678, 2024, doi: 10.1016/j.engappai.2024.108678.

[10] A. Soualhi, K. Medjaher, and N. Zerhouni, "Bearing health monitoring based on Hilbert–Huang transform, support vector machine, and regression," IEEE Trans. Instrum. Meas., vol. 64, no. 1, pp. 52–62, 2015, doi: 10.1109/TIM.2014.2330494.

[11] T. von Hahn, and C. K. Mechefske, "Machine learning in CNC machining: Best practices," Machines, vol. 10, no. 12, Art. no. 1233, 2022, doi: 10.3390/machines10121233.

[12] Y. Han, Z. Wei, and G. Huang, "An imbalance data quality monitoring based on SMOTE-XGBOOST supported by edge computing," Sci. Rep., vol. 14, Art. no. 10151, May 2024, doi: 10.1038/s41598-024-60607-w.

[13] E. F. Swana, W. Doorsamy, and P. Bokoro, "Tomek link and SMOTE approaches for machine fault classification with an imbalanced dataset," Sensors, vol. 22, no. 9, Art. no. 3246, 2022, doi: 10.3390/s22093246.

[14] Y. Hou, J. Ma, J. Wang, C. Liu, D. Li, and H. Li, "Enhanced generative adversarial networks for bearing imbalanced fault diagnosis of rotating machinery," Appl. Intell., vol. 53, pp. 25201-25215, 2023, doi: 10.1007/s10489-023-04785-8.

[15] A. Windmann, P. Wittenberg, M. Schieseck, and O. Niggemann, "Artificial intelligence in Industry 4.0: A review of integration chal-lenges for industrial systems," in Proc. IEEE Int. Conf. Ind. Inform. (INDIN), Beijing, China, 2024, pp. 1-8, doi: 10.1109/INDIN58382.2024.10774488.

[16] E. Jovic, D. Primorac, M. Cupic, and A. Jovic, "Publicly available datasets for predictive maintenance in the energy sector: A review," IEEE Access, vol. 11, pp. 73505-73520, 2023, doi: 10.1109/ACCESS.2023.3294998.

[17] J. Zheng, C. Liu, and L. Zhang, "A comprehensive review of machine learning techniques for condition-based maintenance," Int. J. Progn. Health Manag., vol. 15, no. 2, 2024, doi: 10.36001/ijphm.2024.v15i2.3850.

[18] J. Li, "Area under the ROC curve has the most consistent evaluation for binary classification," PLOS ONE, vol. 19, no. 12, Art. no. e0316019, 2024, doi: 10.1371/journal.pone.0316019.

[19] K. Riehl, M. Neunteufel, and M. Hemberg, "Hierarchical confusion matrix for classification performance evaluation," J. Roy. Stat. Soc. C, vol. 72, no. 5, pp. 1394-1412, Nov. 2023, doi: 10.1093/jrsssc/qlad057.

[20] J. Zhu, B. Chen, and W. Qin, "Feature engineering-based machine learning approaches for predictive maintenance in smart manu-facturing," IEEE Trans. Autom. Sci. Eng., vol. 19, no. 4, pp. 2801-2815, Oct. 2022, doi: 10.1109/TASE.2021.3106671.

[21] G. Niu, E. Liu, X. Wang, P. Ziehl, and B. Zhang, "Enhanced discriminate feature learning deep residual CNN for multi-task bearing fault diagnosis with information fusion," IEEE Trans. Ind. Inform., vol. 19, no. 1, pp. 762-770, Jan. 2023, doi: 10.1109/TII.2022.3179011.

[22] C. Lou, M. A. Atoui, and X. Li, "Recent deep learning models for diagnosis and health monitoring: A review of research works and future challenges," Trans. Inst. Meas. Control, vol. 46, no. 5, pp. 879-901, Mar. 2024, doi: 10.1177/01423312231197052.

[23] D. Dablain, B. Krawczyk, and N. V. Chawla, "DeepSMOTE: Fusing deep learning and SMOTE for imbalanced data," IEEE Trans. Neu-ral Netw. Learn. Syst., vol. 34, no. 9, pp. 6390-6404, 2023, doi: 10.1109/TNNLS.2021.3136503.

[24] O. Matania, L. Bechhoefer, and J. Bortman, "Signal processing for the condition-based maintenance of rotating machines via vibra-tion analysis: A tutorial," Sensors, vol. 24, no. 2, p. 454, Jan. 2024, doi: 10.3390/s24020454.

[25] M. Tiboni, C. Remino, R. Bussola, and C. Amici, "A review on vibration-based condition monitoring of rotating machinery," Appl. Sci., vol. 12, no. 3, Art. no. 972, Jan. 2022, doi: 10.3390/app12030972.

[26] L. Wang, M. Han, X. Li, N. Zhang, and H. Cheng, "Review of classification methods on unbalanced data sets," IEEE Access, vol. 9, pp. 64606–64628, 2021, doi: 10.1109/ACCESS.2021.3074243.

[27] X. Thelen, X. Zhang, O. Fink, Y. Lu, S. Ghosh, and B. D. Youn, "A comprehensive review of digital twin—Part 1: Modeling and twin-ning enabling technologies," Struct. Multidiscip. Optim., vol. 65, p. 354, Dec. 2022, doi: 10.1007/s00158-022-03425-4.

[28] Q. Qian, J. Zhou, and Y. Qin, "Relationship transfer domain generalization network for rotating machinery fault diagnosis under different working conditions," IEEE Trans. Ind. Inform., vol. 19, no. 9, pp. 9898-9908, Sep. 2023, doi: 10.1109/TII.2023.3234970.

[29] B. A. Tama, M. Vania, S. Lee, and S. Lim, "Recent advances in the application of deep learning for fault diagnosis of rotating ma-chinery using vibration signals," Artif. Intell. Rev., vol. 56, no. 5, pp. 4667-4709, May 2023, doi: 10.1007/s10462-022-10293-3.

[30] E. Zio, "Prognostics and health management (PHM): Where are we and where do we (need to) go in theory and practice," Reliab. Eng. Syst. Saf., vol. 218, Art. no. 108119, Feb. 2022, doi: 10.1016/j.ress.2021.108119.

[31] M. Owusu-Adjei, J. B. Hayfron-Acquah, T. Frimpong, and G. Abdul-Salaam, "Imbalanced class distribution and performance evalua-tion metrics: A systematic review of prediction accuracy for determining model performance in healthcare systems," PLOS Digit. Health, vol. 2, no. 11, Art. no. e0000290, Nov. 2023, doi: 10.1371/journal.pdig.0000290.

[32] M. S. Khaled, R. Hassan, K. Choi, and M. A. Samad, "Accuracy, precision, recall, F1-score, or MCC? Empirical evidence from advanced statistics, ML, and XAI for evaluating business predictive models," J. Big Data, vol. 12, no. 1, Art. no. 268, 2025, doi: 10.1186/s40537-025-01313-4.