banner
Home > Open Journal Systems

IoT-enabled image captioning with deep learning for healthcare domain

P. Steffy Sherly, P. Velvizhy

Abstract


The ever-increasing volume of medical images greatly strains clinicians who are in the process of reviewing it and writing reports. It would be more efficient and cost-effective if an image captioning model could automatically create report drafts from matching photos, thereby relieving physicians from this tedious work. The Internet of things (IoT) has switched its emphasis from its initial binary concept to that of the Internet of multimedia things (IoMT) because of the explosive rise of multilingual-on-demand data in various sound, footage, picture forms. This work proposed a deep learning-based image caption network (DL-ICN) for healthcare domain. The work originality is shown using DL to identify various class labels of the patient X-ray and ECG images. With the help of bilateral encoder representations from transformers (BERT) method for captioning pictures, a detailed written summary of a person’s medical picture may be generated automatically. Results of simulations showed that the proposed model achieved good compression performance, good quality reconstruction and good classification results for image captioning.


Keywords


Bidirectional Encoder Representations from Transformers (BERT); Deep Learning (DL); Image Caption Network (ICN); Internet of things (IoT); Self-Control Differential Evolution (SCDE)

Full Text:

PDF

References


1. Singh S, Rathore S, Alfarraj O, et al. A framework for privacy-preservation of IoT healthcare data using Federated Learning and blockchain technology. Future Generation Computer Systems 2022; 129: 380–388. doi: 10.1016/j.future.2021.11.028

2. Mehbodniya A, Webber JL, Neware R, et al. Modified Lamport Merkle Digital Signature blockchain framework for authentication of internet of things healthcare data. Expert Systems 2022; 39(10): e12978. doi: 10.1111/exsy.12978

3. Abdullah S, Arshad J, Khan MM, Alazab M, Salah Ket al. PRISED tangle: A privacy-aware framework for smart healthcare data sharing using IOTA tangle. Complex & Intelligent Systems 2023; 9(3): 3023–3041. doi: 10.1007/s40747-021-00610-8

4. Shahid J, Ahmad R, Kiani AK, et al. Data protection and privacy of the internet of healthcare things (IoHTs). Applied Sciences 2022; 12(4): 1–22. doi: 10.3390/app12041927

5. Venkatesh S, Narasimhan K, Adalarasu K. An Overview of Interpretability Techniques for Explainable Artificial Intelligence (XAI) In Deep Learning-Based Medical Image Analysis, 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS); 17–18 March 2023; Coimbatore, India, 175–182. IEEE; 2023.

6. Borys K, Schmitt YA, Nauta M, et al. Explainable AI in medical imaging: An overview for clinical practitioners–Beyond saliency-based XAI approaches. European journal of radiology 2023; 1–11. doi: 10.1016/j.ejrad.2023.110786

7. Chai Y, Liu H, Xu J, et al. A multi-label classification with an adversarial-based denoising autoencoder for medical image annotation. ACM Transactions on Management Information Systems 2023; 14(2): 1–21. doi: 10.1145/3561653

8. Yashaswini S, Jayanthi MG, Subhash J. Captioning and Classification of Brain Tumor from MRI Images using Deep Learning Methods. Journal of Coastal Life Medicine 2023; 11: 302–307.

9. Jain K, Gandhi S, Singhal S, Rajput S. Semantic Image Captioning using Cosine Similarity Ranking with Semantic Search, Proceedings of the 2023 Fifteenth International Conference on Contemporary Computing 3–5 August 2023; Noida, India, 220–223. Association for Computing Machinery, New York, United States; 2023.

10. Gao D, Kong M, Zhao Y, et al. Simulating doctors’ thinking logic for chest X-ray report generation via Transformer-based Semantic Query learning. Medical Image Analysis 2023; 102982. doi: 10.1016/j.media.2023.102982

11. Wong KK, Ayoub M, Cao Z, et al. The Synergy of Cybernetical Intelligence with Medical Image Analysis for Deep Medicine: A Methodological Perspective. Computer Methods and Programs in Biomedicine 2023; 107677. doi: 10.1016/j.cmpb.2023.107677

12. Yin Y, Han Z, Jian M, et al. AMSUnet: A neural network using atrous multi-scale convolution for medical image segmentation. Computers in Biology and Medicine 2023; 107120. doi: 10.1016/j.compbiomed.2023.107120

13. Dewi C, Chen RC, Yu H, Jiang X. XAI for Image Captioning using SHAP. Journal of Information Science & Engineering 2023; 39(4): 711–724. doi: 10.6688/JISE.202307_39(4).0001

14. Beddiar R, Oussalah M. Explainability in medical image captioning, Explainable Deep Learning AI, Academic Press; 2023. pp. 239–261.

15. Elbedwehy S, Medhat T, Hamza T, Alrahmawy MF. Enhanced descriptive captioning model for histopathological patches. Multimedia Tools and Applications 2023; 1–20. doi: 10.1007/s11042-023-15884-y

16. Lin Y, Lai K, Chang W. Skin Medical Image Captioning Using Multi-Label Classification and Siamese Network. IEEE Access 2023; 11: 23447–23454. doi: 10.1109/ACCESS.2023.3249462

17. Pang T, Li P, Zhao L. A survey on automatic generation of medical imaging reports based on deep learning. BioMedical Engineering OnLine 2023; 22(1): 1–16. doi: 10.1186/s12938-023-01113-y




DOI: https://doi.org/10.32629/jai.v7i5.1542

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 P. Steffy Sherly, P. Velvizhy

License URL: https://creativecommons.org/licenses/by-nc/4.0/