ABSTRACT
Over the coming decades, artificial intelligence (AI) technologies have a lot of potential to improve healthcare. In particular, AI systems that make use of many data sources and input modalities are soon to be a practical way to provide results that are more accurate and to create deployable pipelines that can be used for a variety of applications. In this work, we develop and assess a unified framework called Holistic AI in Medicine (HAIM) to make the development and testing of AI systems that use multimodal inputs easier. Our method makes use of machine learning modeling steps and generalizable data pre-processing, which are easily adaptable for usage in healthcare research and deployment. This demonstrates that this framework can reliably and consistently generate models that perform better than comparable single-source methods in a variety of healthcare demonstrations, such as different diagnoses for chest pathology and forecasts for length of stay and mortality. Using Shapley values, this also quantifies the contribution of each modality and data source, highlighting the variability in data modality relevance and the requirement for multimodal inputs across various healthcare-related tasks. Our Holistic AI in Medicine (HAIM) framework's adaptable features and generalizability may present a viable path for multimodal predictive systems in clinical and operational healthcare settings in the future.
