Evaluasi Keandalan Peta Saliensi Grad-CAM dan Perilaku Galat Lintas Stratum Luminansi pada Klasifikasi Lubang Jalan

Agung Ramadhanu; Maheza  Novrayuda

doi:10.35134/komtekinfo.v11i3.721

Authors

Agung Ramadhanu Universitas Putra Indonesia YPTK Padang
Maheza Novrayuda Universitas Putra Indonesia YPTK Padang

DOI:

https://doi.org/10.35134/komtekinfo.v11i3.721

Keywords:

Explainable AI, Grad-CAM, sanity check, Spearman rank correlation, pothole classification

Abstract

Grad-CAM heatmaps are often shown as evidence that a road-damage classifier looks at the damage itself, yet such maps are rarely tested for dependence on the learned parameters. This study examines Grad-CAM for EfficientNet-B0 and MobileNetV3-Large, each trained with three augmentation strategies on 676 unique pothole and normal-road images (five-fold cross-validation, three seeds). Using the model parameter randomization test of Adebayo et al., maps from trained models were compared with maps after randomizing the classification head or all parameters, on 12 test images from fold 0 (seed-42 models). Maps of clean and shadowed images (severity 3) were also compared. Agreement was measured with Spearman rank correlation. For MobileNetV3-Large, full randomization reduced the mean correlation to 0.016–0.045 and head randomization to 0.21–0.36, whereas clean-versus-shadow maps stayed at 0.959–0.968. For EfficientNet-B0, shadow stability was 0.953–0.975 and head randomization gave 0.14–0.21, but full randomization produced all-zero maps, so the correlation was undefined in all 36 comparisons per strategy. Of 239 misclassifications on clean images (18 model configurations), the medium-luminance stratum had the highest error rate for the illum strategy (3.11–3.41%, mostly false positives), against 1.03–1.64% in other strata, and three normal images were misclassified by every configuration. The results indicate that MobileNetV3-Large maps depend on learned weights, but the small sample (12 images, one fold, one seed) and the absence of causal tests limit the conclusions

References

Y. Safyari, M. Mahdianpari, and H. Shiri, "A review of vision-based pothole detection methods using computer vision and machine learning," Sensors, vol. 24, no. 17, art. 5652, 2024, doi: 10.3390/s24175652.

R. Kothai, N. Prabakaran, Y. V. Srinivasa Murthy, L. R. Cenkeramaddi, and V. Kakani, "Pavement distress detection, classification, and analysis using machine learning algorithms: A survey," IEEE Access, vol. 12, pp. 126943–126960, 2024, doi: 10.1109/ACCESS.2024.3455093.

V. Plevris and G. Papazafeiropoulos, "AI in structural health monitoring for infrastructure maintenance and safety," Infrastructures, vol. 9, no. 12, art. 225, 2024, doi: 10.3390/infrastructures9120225.

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-CAM: Visual explanations from deep networks via gradient-based localization," Int. J. Comput. Vis., vol. 128, no. 2, pp. 336–359, 2020, doi: 10.1007/s11263-019-01228-7.

Y. Zanevych, V. Yovbak, O. Basystiuk, N. Shakhovska, S. Fedushko, and S. Argyroudis, "Evaluation of pothole detection performance using deep learning models under low-light conditions," Sustainability, vol. 16, no. 24, art. 10964, 2024, doi: 10.3390/su162410964.

J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim, "Sanity checks for saliency maps," in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 31, 2018, pp. 9505–9515.

S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, "Unmasking Clever Hans predictors and assessing what machines really learn," Nat. Commun., vol. 10, art. 1096, 2019, doi: 10.1038/s41467-019-08987-4.

R. Geirhos et al., "Shortcut learning in deep neural networks," Nat. Mach. Intell., vol. 2, no. 11, pp. 665–673, 2020, doi: 10.1038/s42256-020-00257-z.

C. Rudin, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead," Nat. Mach. Intell., vol. 1, no. 5, pp. 206–215, 2019, doi: 10.1038/s42256-019-0048-x.

A. Hedström et al., "Quantus: An explainable AI toolkit for responsible evaluation of neural network explanations and beyond," J. Mach. Learn. Res., vol. 24, no. 34, pp. 1–11, 2023. [Online]. Available: https://jmlr.org/papers/v24/22-0142.html

A. Hedström, L. Weber, S. Lapuschkin, and M. M.-C. Höhne, "Sanity checks revisited: An exploration to repair the model parameter randomisation test," in NeurIPS 2023 Workshop XAI in Action: Past, Present, and Future Applications, 2023. [Online]. Available: https://openreview.net/forum?id=vVpefYmnsG

B. Bilodeau, N. Jaques, P. W. Koh, and B. Kim, "Impossibility theorems for feature attribution," Proc. Natl. Acad. Sci. U.S.A., vol. 121, no. 2, art. e2304406120, 2024, doi: 10.1073/pnas.2304406120.

L. Longo et al., "Explainable artificial intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions," Inf. Fusion, vol. 106, art. 102301, 2024, doi: 10.1016/j.inffus.2024.102301.

A. Saporta et al., "Benchmarking saliency methods for chest X-ray interpretation," Nat. Mach. Intell., vol. 4, no. 10, pp. 867–878, 2022, doi: 10.1038/s42256-022-00536-x.

L. Fan, S. Li, Y. Li, B. Li, D. Cao, and F.-Y. Wang, "Pavement cracks coupled with shadows: A new shadow-crack dataset and a shadow-removal-oriented crack detection approach," IEEE/CAA J. Autom. Sin., vol. 10, no. 7, pp. 1593–1607, 2023, doi: 10.1109/JAS.2023.123447.

X. Xu, T. Wu, W. Ge, L. Dai, J. Gong, and D. Chen, "Enhanced pothole detection in low-light conditions using DCE-Pothole and YOLOv10 fusion," Results Eng., vol. 28, art. 107322, 2025, doi: 10.1016/j.rineng.2025.107322.

J. Frnda, S. Bandyopadhyay, M. Pavlicko, M. Durica, M. Savrasovs, and S. Banerjee, "Analysis of pothole detection accuracy of selected object detection models under adverse conditions," Transp. Telecommun., vol. 25, no. 2, pp. 209–217, 2024, doi: 10.2478/ttj-2024-0016.

A. Kumar, "Pothole Detection Dataset," Kaggle. [Online]. Available: https://www.kaggle.com/datasets/atulyakumar98/pothole-detection-dataset (accessed Oct. 13, 2026).

S. Kapoor and A. Narayanan, "Leakage and the reproducibility crisis in machine-learning-based science," Patterns, vol. 4, no. 9, art. 100804, 2023, doi: 10.1016/j.patter.2023.100804.

M. Tan and Q. V. Le, "EfficientNet: Rethinking model scaling for convolutional neural networks," in Proc. 36th Int. Conf. Mach. Learn. (ICML), PMLR 97, 2019, pp. 6105–6114.

A. Howard et al., "Searching for MobileNetV3," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2019, pp. 1314–1324, doi: 10.1109/ICCV.2019.00140.

I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," in Proc. Int. Conf. Learn. Represent. (ICLR), 2019.

P. Virtanen et al., "SciPy 1.0: Fundamental algorithms for scientific computing in Python," Nat. Methods, vol. 17, no. 3, pp. 261–272, 2020, doi: 10.1038/s41592-019-0686-2.

O. Rainio, J. Teuho, and R. Klén, "Evaluation metrics and statistical tests for machine learning," Sci. Rep., vol. 14, art. 6086, 2024, doi: 10.1038/s41598-024-56706-x.

M. Novrayuda, "Evaluasi ketahanan fotometrik model jaringan konvolusi ringan melalui augmentasi iluminasi sintetis pada klasifikasi lubang jalan," naskah belum dipublikasikan, 2026.