## Introduction Mortality of ICU patients is one of the biggest challenges for clinicians and scientists. The objective of this study was to develop a multimodal deep learning model that can predict mortality of ICU patients using structured and unstructured clinical data. ## Materials and methods We used data from MIMIC-III, MIMIC-IV, eICU, and HiRID to develop the model. The model was evaluated using metrics such as AUROC (Area Under the Receiver Operating Characteristic curve), AUPRC (Area Under the Precision-Recall Curve) and Brier score. ## Results The results show that the model achieved an AUROC of 92%, AUPRC of 53% and Brier score of 19%. External validation was performed on temporally separated MIMIC, HiRID and eICU datasets. The results showed that the model achieved an AUROC ranging from 0.84 to 0.92. ## Discussion The results of this study highlight the importance of integrating different types of data to improve performance of machine learning models. Structured and unstructured data can provide fundamental information that can help predict mortality of ICU patients with greater precision. ## Conclusion In summary, this study has shown that integration of structured and unstructured data can significantly improve performance of machine learning models for predicting mortality of ICU patients.