A deep learning model combining circulating tumor cells and radiological features in the multi-classification of mediastinal lesions in comparison with thoracic surgeons: a large-scale retrospective study

Abstract

Background

CT images and circulating tumor cells (CTCs) are indispensable for diagnosing the mediastinal lesions by providing radiological and intra-tumoral information. This study aimed to develop and validate a deep multimodal fusion network (DMFN) combining CTCs and CT images for the multi-classification of mediastinal lesions.

Methods

In this retrospective diagnostic study, we enrolled 1074 patients with 1500 enhanced CT images and 1074 CTCs results between Jan 1, 2020, and Dec 31, 2023. Patients were divided into the training cohort (n = 434), validation cohort (n = 288), and test cohort (n = 352). The DMFN and monomodal convolutional neural network (CNN) models were developed and validated using the CT images and CTCs results. The diagnostic performances of DMFN and monomodal CNN models were based on the Paraffin-embedded pathologies from surgical tissues. The predictive abilities were compared with thoracic resident physicians, attending physicians, and chief physicians by the area under the receiver operating characteristic (ROC) curve, and diagnostic results were visualized in the heatmap.

Results

For binary classification, the predictive performances of DMFN (AUC = 0.941, 95% CI 0.901–0.982) were better than the monomodal CNN model (AUC = 0.710, 95% CI 0.664–0.756). In addition, the DMFN model achieved better predictive performances than the thoracic chief physicians, attending physicians, and resident physicians (P = 0.054, 0.020, 0.016) respectively. For the multiclassification, the DMFN achieved encouraging predictive abilities (AUC = 0.884, 95%CI 0.837–0.931), significantly outperforming the monomodal CNN (AUC = 0.722, 95%CI 0.705–0.739), also better than the chief physicians (AUC = 0.787, 95%CI 0.714–0.862), attending physicians (AUC = 0.632, 95%CI 0.612–0.654), and resident physicians (AUC = 0.541, 95%CI 0.508–0.574).

Conclusions

This study showed the feasibility and effectiveness of CNN model combing CT images and CTCs levels in predicting the diagnosis of mediastinal lesions. It could serve as a useful method to assist thoracic surgeons in improving diagnostic accuracy and has the potential to make management decisions.

Background

Mediastinal lesions are a broad-spectrum disease consisting of either benign or malignant tumors [1]. The diagnosis of mediastinal lesions should be established based on the pathology [2]. However, due to the relatively challenging nature of the region, the diagnosis and treatment have traditionally been mainly based on empirical experiences and clinical auxiliary examinations according to the National Comprehensive Cancer Network (NCCN) guidelines [3]. In addition, the therapy and prognosis vary among distinct kinds of mediastinal lesions. For example, complete thymectomy is the standard treatment for thymoma at any stage, except for stage I thymoma. For patients with stage I thymoma, thymomectomy might be the more reasonable surgical option [4, 5]. Therefore, a simple, accurate, and feasible method for differentiating mediastinal lesions is imperative, especially for the treatment strategy and clinical management.

While the histopathological examination is the gold diagnosis standard, the process is invasive, risky, and expensive. To avoid unnecessary operations, preoperative enhanced CT provides adequate quantitative information on the lesions, such as size, density, vessels, and adjacent structures. Owing to its better visibility, versatility, and timeliness, enhanced CT has been increasingly considered as an ideal diagnostic imaging method for mediastinal lesions, although it has been proven that CT combined with other methods could improve the accuracy of chest diseases at or above the level of thoracic surgeons [6, 7]. However, the accuracy of CT alone remains limited, particularly in the early detection of small or atypical lesions. In addition, the complexity of enhanced CT for primary surgeons to monitor and accurately identify different mediastinal lesion types could not be ignored.

Recently, liquid biopsy has drastically revolutionized the field of precision oncology, offering personalized molecular genomic data. Its non-invasive nature also paved a new era in thymoma [8, 9]. The combination of CT imaging and CTCs in a multimodal approach has shown promise in enhancing diagnostic accuracy [10, 11]. Studies have indicated that integrating radiological features with molecular biomarkers can improve cancer detection and prognostication. For example, radiomics, which extracts quantitative features from medical images, has been combined with CTC analysis in several studies to enhance the performance of cancer detection systems [12]. Despite these advances, most methods rely on either CT imaging or CTCs for diagnostic purposes. Additionally, CT imaging often requires advanced post-processing or fusion techniques to achieve high diagnostic accuracy, and CTC detection methods can be expensive and technically challenging [12, 13].

Recently, a growing number of studies have applied deep learning to classify thymoma and achieved excellent diagnostic accuracy [14, 15]. However, the integration of deep learning models with molecular data such as CTCs is still in its early stages. Additionally, these studies mainly focused on the specific disease (i.e., thymoma) and they were mostly used for the binary classification of mediastinal lesions (i.e., benign vs malignant). On the other hand, their diagnostic performance in examining the broad spectrum of mediastinal lesions was insufficient.

This study introduces a novel deep multimodal fusion network (DMFN) that integrates CTCs and CT imaging for the multi-classification of mediastinal tumors, providing a more comprehensive approach to diagnosis. Our model differs significantly from previous studies by combining radiological and molecular data, addressing the limitations of current diagnostic methods. Unlike traditional monomodal convolutional neural network (CNN) models, which rely on either CT images or CTCs alone, our approach combines both datasets to improve diagnostic accuracy. We demonstrate that the DMFN model significantly outperforms both monomodal CNN models and clinical experts, as evidenced by superior AUC scores and diagnostic performance. This work represents a significant step forward in the integration of multimodal data for cancer diagnosis and has the potential to transform the clinical management of mediastinal tumors.

Methods

Study cohort

This is a retrospective and single-center study. The inclusion criteria were as follows: (1) complete clinical basic information, chest-enhanced CT images, and circulating tumor cells within 2 weeks after admission. (2) Initial treatments including surgery, and neoadjuvant chemoradiotherapy were not received. (3) No distant metastasis or multiple primary cancers. (4) Patients aged 18 years or older. (5) Follow-up: Patients with sufficient follow-up data or surgical intervention records. Exclusion criteria: (1) Patients under 18 years old. (2) Inadequate imaging: patients with poor-quality CT images (e.g., significant motion artifacts, incomplete scans) were excluded. (3) Incomplete data: Patients lacking CTCs data or follow-up records or surgical records. (4) Non-mediastinal lesions: Patients with lesions located outside the mediastinal region were excluded. (5) Coexisting medical conditions: Patients with other major coexisting conditions (e.g., metastases to the mediastinum) that could complicate the diagnosis were excluded. This study was compliant with the Standards for Reporting of Diagnostic Accuracy (STARD) reporting guidelines.

We initially identified 2000 patients from the doctor workstation system who had CT scans taken between January 1, 2020, and December 31, 2023. After the inclusion and exclusion criteria, a total of 1074 patients whose data met all requirements. The study flow chart is provided in Fig. 1.

Fig. 1

The study flow chart of the study

Our study was approved by the ethics committee of Shanghai Pulmonary Hospital, School of Medicine, Tongji University. Informed consent was waived because of the retrospective nature.

CT examination and image preprocessing

All patients in the cohort underwent chest-enhanced CT using Siemens Somatom Definition AS scanners (Siemens Medical Systems, Erlangen, Germany) or Philips Brilliance 40 scanners (Philips Healthcare, Cleveland, USA) within 2 weeks after admission. After intravenous injection of iodine contrast agent with a 60–65-s delay, CT images were standardized with a window width of 350 and a window level of 50. Mathematical algorithms and detector advancements have helped to reduce noises and reconstruct CT images at 1-mm slice intervals. Two thoracic surgeons draw the regions of interest (ROI) in the reconstructed images by manually segmenting them. Through objective and quantitative image analysis and clinical experience, thoracic surgeons make differential diagnoses of the delineated lesions.

Plasma CTCs testing

Venous blood samples were collected from patients on the day of admission, and the blood samples were stored in the CellSave tubes. Then, the samples were centrifugated at 800 g for 10 min. After centrifugation, the residual supernatant was removed. CTCs were detected through the Immunomagnetic bead sorting combined with the immunofluorescence staining method, and the data were obtained using the CellSearch Circulating Tumor Cell Kit (GENO, 20,163,400,061, China).

Pathological ground truth

The diagnosis of mediastinal nodules was confirmed based on histological pathology results [16]. Benign diseases included cysts, lipoma, benign thymoma, mature teratoma, cystic lymphangioma, hemangioma, fibroma, and neuroma. Malignant diseases included malignant thymoma, sarcoma, neuroblastoma, immature teratoma, lymphoma, and metastatic tumor. The frequencies of mediastinal nodules diagnosed in the training, validation, and test cohort were summarized in Table 1.

Table 1 Summary of diagnosis in the training, validation, and test cohorts n = 1074

Data preprocessing

In the chest-enhanced CT image processing stage, we first used an easy fourth-order partial differential equation diffusion model in the noise reduction process to solve the problem of blurred edges and texture details. And then we extract the region of interest separately from the CT fault images. Due to the fact that the spacing between tomographic scans is usually larger than the pixel size of two-dimensional images, cubic voxel data with consistent directions are generated in three-dimensional space, which can outline the three-dimensional shape of the measured tissues. Based on the reconstructed 3D image, we use various geometric transformation operations to achieve measurement of different projection imaging methods and indicators such as volume, area, and thickness. We complete the reconstruction of different sections, display views in different directions, determine the nature of lesions, and proposed management plans.

Deep multimodal fusion network model development

To better help the machine classifier distinguish the characteristics of mediastinal lesions, we utilized the 3D-Slicer software to segment the mediastinal region. Next, we cropped and extracted the lesions from the CT images. Subsequently, these extracted images were prepared for the classification tasks based on the prior knowledge and the segmentation mask.

To integrate the internal and external information, we developed a deep multimodal fusion network (DMFN) model that combined the mediastinal lesions’ appearance, CTC data, and CT images for deep feature fusion. The DMFN input contains various data from three modal, including clinical close-up manifestation, CT images, and 3D-Slicer images. The CT images use a 2D convolutional neural network (CNN) to process CT scan images. And this architecture consists of several convolutional layers followed by max-pooling layers to down-sample the image and retain important features. The CTC data were processed by the fully connected neural network, which consists of dense layers that learn the relationships between different levels of CTC-related features. DMFN will extract multi-layer data from the above aspects, and each layer of metrics will be extracted, fused, and preserved. The monomodal CNN typically processes a single modality of data, such as CT images, using convolutional layers for feature extraction, followed by pooling and fully connected layers for classification. We used the monomodal CNN model to generate a similar training result for comparison to DMFN [17, 18].

In this study, we used the Regulated Networks (RegNet) to predict the pathology of the mediastinal lesions, which was incorporated as a backbone architecture within our DMFN model. It consists of convolutional blocks, each containing a set of convolutional filters followed by batch normalization and activation functions. Specifically, we utilized RegNet for feature extraction from CT images. Meanwhile, we used the neutral cross-entropy loss to mitigate the over-sharpness of entropy minimization. The architecture of the DMFN and RegNet models are summarized in Fig. 2. In the statistical process, the output outcomes were the top 3 prediction results. We used the top 1 as the optimal model and the performance was repeated 3 times. The statistical methods and algorithms explanations are cited from Prof. Xu J study [19].

Fig. 2

The structures of DMFN and RegNet models. A DMFN model: Combines CT image features and CTC data through convolutional and fully connected layers, fusing them for multi-class classification via a softmax layer. B RegNet model: Extracts features from CT images using sequential convolutional layers with dimensionality reduction to enhance generalization and improve feature extraction efficiency

Performance evaluation of CNN model and thoracic surgeons

Experienced thoracic surgeons were divided into three groups, including resident physicians (n = 2), attending physicians (n = 2), and chief physicians (n = 2). The resident physicians have 1–3 years of clinical experience. They mainly focused on the patients’ manifestations and CT images, initial patient assessments, and routine monitoring, under the supervision of more experienced physicians and they are mainly responsible for the binary classification of mediastinal lesions, for example, benign vs malignant. The attending physicians usually have 5 or more years of clinical experience. They were responsible for image interpretation, detailed diagnostic assessments, and contributing to treatment planning, and were asked to verify the accuracy of classification and proposed therapy strategies, for example, surgery or not, follow-ups. The chief physicians are experts in the field with 10 or more years of experience. They took responsibility for final clinical decisions, particularly in complex or ambiguous cases. They provided guidance and supervision for the entire medical team and ensured that the final treatment plans aligned with national clinical guidelines and best practices. Management decisions (e.g., surgery, follow up or conservative treatment) are established according to the clinical guidelines (NCCN and ESMO guidelines), multidisciplinary team, and individualized treatment. All physicians were asked to make management decisions and performed the work in a blinded manner to patients’ information. Disputes and inconsistencies are resolved through discussion. At last, the results of diagnostic performance were compared between thoracic physicians and CNN.

Statistical analysis

Chi-square and Mann–Whitney U were used to compare categorical variables and continuous variables, respectively. Receiver operating characteristic (ROC) curves and the area under the ROC curves (AUC) were calculated to assess the diagnostic performances of CNN and thoracic physicians. Subsequently, the sensitivity, specificity, positive, predictive value (PPV), negative predictive value (NPV), and accuracy were also calculated to evaluate the diagnostic performances. The DeLong test and McNemar’s test were used to compare the AUCs and sensitivity and specificity. Differences were considered statistically significant at 2-sided P < 0.05. All the data were calculated using the R software (version 4.3.3).

Results

Patients’ information and characteristics of mediastinal lesions

A total of 1074 patients were included in the study. The average age of the population was 53.18 ± 13.29, and 501 (46.65%) were male. Among the patients, 434 patients were in the training cohort, and 288 patients were in the validation cohort. In addition, 352 patients were in the test cohort. This study produces 1500 enhanced CT slices, with 596 from the training cohort, 419 from the validation cohort, and 485 from the test cohort. The most frequent mediastinal lesions were thymoma (34.73%, 373), followed by cyst (28.49%, 306), Schwannoma (7.73%, 83), and thymic squamous cell carcinoma (5.49%, 59). Detailed information is shown in Table 2.

Table 2 Basic information of different mediastinal lesions and CTCs levels in the whole cohort

Diagnostic performance of CNN models

In the validation cohort, a total of 288 patients were collected for analysis. For the differential diagnosis of benign and malignant mediastinal lesions, the performance of DMFN combined with CT images and CTCs was superior to the monomodal CNN model both in AUC, sensitivity, and accuracy. The AUC of DMFN combined with CT images and clinical close-up manifestations and monomodal CNN were 0.941 (95% confidence interval [CI] 0.901–0.982) and 0.710 (95% CI 0.664–0.756) respectively. The sensitivities were 0.809 (0.785–0.833) and 0.702 (0.676–0.728), and the accuracies were 0.927 (0.910–0.944) and 0.779 (0.743–0.815). In addition, the positive predictive value of correctly diagnosed benign mediastinal lesions is 0.885 (0.837–0.932) and 0.692 (0.671–0.713), suggesting the diagnostic accuracy of DMFN combined with CT images and clinical close-up manifestations was better than monomodal CNN (Table 3).

Table 3 Diagnosis performance of CNN models and different thoracic surgeons in the validation cohort

After conducting the benign lesions, we also compared the correct diagnostic performance of the malignant mediastinal lesions. A total of 190 malignant lesions were identified by the DMFN combined with CT images and CTCs and monomodal CNN in the test cohort. The accuracy of them were 0.905 (95%CI 0.861–0.949) and 0.793 (95%CI 0.744–0.842) respectively. Collectively, these data strongly indicated that the DMFN model has higher accurate diagnostic performances of malignant mediastinal lesions (Additional file 1: Table 1). Similar results could also be found in the test cohort (Fig. 3).

Fig. 3

Typical diagnostic performance matrices of binary classification in the test cohort. CNN: convolutional neural network; DMFN: deep multimodal fusion network

Multiclassification performance of CNN

The diagnostic parameters of multiclassification diagnosis in the training cohort were described in Fig. 4. The DMFN achieved the highest AUC (0.884, 95%CI 0.836–0.932) among all predictive methods, significantly outperforming the monomodal CNN (AUC = 0.722, 95%CI 0.681–0.763, P = 0.030) (Additional file 2: Fig. S1). For example, 20% of the benign thymoma were diagnosed as malignant mediastinal lesions by monomodal CNN mistakenly, but were correctly diagnosed as benign mediastinal lesions by DMFN subsequently. Furthermore, the DMFN demonstrated an excellent AUC of 0.935 in the validation cohort about the specific mediastinal lesions, obviously surpassing that of the monomodal CNN with the AUC = 0.782. Similarly, the DMFN significantly reduced the possibility of monomodal CNN mistakenly diagnosing certain malignant diseases as benign diseases in both the training cohort and validation cohort.

Fig. 4

ROC curves of CNN models and different thoracic surgeons in the binary and multiclass classifications. A ROC curves of CNN models and different thoracic surgeons in the binary classifications. B ROC curves of CNN models and different thoracic surgeons in the multiclass classifications. ROC: receiver operating characteristic; AUC: area under the ROC curve; CNN: convolutional neural network; DMFN: deep multimodal fusion network

Diagnostic performance of thoracic surgeons

For the binary classifications, the diagnostic performances of chief physicians were significantly better than resident physicians and attending physicians in the test cohort, with the AUC of 0.802 (95%CI 0.746–0.858), 0.688 (95%CI 0.649–0.714), and 0.52 (95%CI 0.475–0.566). When combined with the CTCs, the diagnostic performances of resident physicians significantly improved compared to not combining, with an AUC of 0.711 (95%CI 0.683–0.747) and 0.636 (95%CI 0.568–0.711). Additionally, the attending physicians significantly outperformed the diagnostic performance with an addition of CTCs, achieving an AUC of 0.805 (95%CI 0.773–0.840).

For the benign and malignant lesions, the diagnostic accuracy of resident physicians was lowest compared to resident physicians and attending physicians (Table 3). However, resident physicians showed a superior accuracy with the addition of CTCs, with the accuracy of 0.719 (95%CI 0.624–0.814) and 0.753 (95%CI 0.728–0.778).

Management decisions

For planning the optimal treatment for patients, all the thoracic surgeons were required to propose management strategies. As expected, the chief physicians made the correct management decisions with the highest accuracy of 0.907(95%CI 0.883–0.935), followed by attending physicians 0.839 (95%CI 0.801–0.870) and resident physicians 0.711 (95%CI 0.685–0.734). The differences were statistically significant (P = 0.0425, 0.0203) (Additional file 3: Table 2).

Diagnostic performance of CNN models vs. thoracic surgeons

For the classification of benign and malignant lesions in the test cohort, the DMFN model achieved a higher AUC than resident physicians, with an AUC of 0.932 (95%CI 0.871–0.993), higher than chief physicians (AUC = 0.767, 95%CI 0.695–0.839). The DMFN achieved better predictive performances than the chief physicians, attending physicians and resident physicians (P = 0.054, 0.020, 0.016) (Additional file 4: Fig. S2).

When predicting the malignant mediastinal lesions, the monomodal CNN model and attending physicians performed similar predictive abilities (AUC = 0.665, 95%CI 0.619–0.712 vs AUC = 0.723, 95%CI 0.649–0.797), both lower than the DMFN model (AUC = 0.843, 95%CI 0.779–0.844). The predictive abilities were significantly different among the DMFN model and monomodal CNN model and physicians (Fig. 5).

Fig. 5

Typical diagnostic performance matrices of multiclass classification in the test cohort. MaT: mature teratoma; IT: immature teratoma; TSCC: Thymic squamous cell carcinoma; MT: metastatic tumor; MALT: mucosa-associated lymphoid tissue

Overall, for the classifications of the 14 types of mediastinal lesions, the DMFN model with an addition of CTCs data showed better performances than the chief physicians in the test cohort, and the difference reached statistical significance (P = 0.040). The representative data are shown in Additional file 3: Table 2.

Discussion

In this study, we developed a DMFN to predict a broad spectrum of mediastinal lesions based on the CT images, clinical information, and CTCs levels, and we also compared its diagnostic performances with thoracic surgeons, especially for the overall classifications. The results demonstrated that the DMFN model significantly outperformed monomodal CNNs in multi-classification tasks, particularly for lesions with overlapping radiological appearances, such as benign and malignant thymomas. While CT is the preferred examination method, the CTCs could reveal the morphological or genetic information of primary tumor cells, supplementing key characteristics. Therefore, for the classification of mediastinal lesions, we suggest that DMFN combined with CTCs for optimal solutions.

Several studies have explored stratification and classification methods for mediastinal lesions using deep learning models, integrating clinical, radiomic, and deep features, and have demonstrated substantial potential for mediastinal tumor classification [20,21,22,23,24]. For example, Lin et al. developed a deep learning model combined with images to differentiate malignant and benign mediastinal lesions, achieving a diagnostic accuracy of 82% and an area under the curve of 0.8812 [23]. Similarly, Liu et al. demonstrated the utility of a 3D DenseNet model for detecting myasthenia gravis in thymoma patients using CT images, with an accuracy of 0.790 [24]. However, using a limited set of clinical factors may result in overestimating or underestimating model diagnostic performances and lead to a lack of generalizations to other diseases. Meanwhile, though effective, their model is slightly inferior to our model, indicating the added benefit of incorporating CTCs data alongside radiological features. These studies collectively highlight the superior diagnostic performance achievable with multimodal fusion models, where combining imaging and molecular data not only enhances accuracy but also outperforms single-modality models.

The CNN model requires a large number of participants to effectively reflect the true underlying features and enhance the overall generalizability of the model [25]. Existing research has mainly been limited to insufficient data, leading suboptimal performance or overfitting [26, 27]. By utilizing the backbone pretrained in the large data training cohort, we achieved improvements in both diagnostic abilities and accuracy compared to physicians in the real world. This is the first model to integrate CTCs with a large number of patients, and it also has the potential to generalize CNN to other types of thoracic diseases, assuming lesions share similar CTCs levels.

For binary classification tasks, the DMFN has superior performances than attending surgeons, but lower than the chief surgeons. This is maybe binary classifications were more common in clinical routine, and the CT images could provide more intuitive judgments. For the multi-classifications, the DMFN had advantages in the diagnosis of mediastinal lesions, and demonstrated comparable or even superior performances compared to the chief surgeons. One reason is that some uncommon mediastinal lesions are resected with a rare incidence rate. Another reason may be that the DMFN with CTCs could easily identify mediastinal lesions through changes in CTC levels.

Since CTCs are gradually becoming popular in patients with mediastinal lesions [28, 29], our results suggested that DMFN with CTCs could be a suitable tool to improve diagnostic accuracy. According to the Ottaviano M’s study, CTCs level is tightly associated with high mediastinal lesions disease burden and advanced stage [28]. In line with them, our study also supported that CTCs are higher in patients with malignant lesions than those with benign lesions. Given its low sensitivity, any steps to improve diagnosis should involve combining clinical data, such as CT images. While the DMFN demonstrates superior performance in complex multi-classification tasks, it may not offer significant advantages in simpler cases, such as mediastinal cysts, where physicians relying on CT images alone can achieve comparable results.

For clinical applicability, our DMFN model could be integrated into clinical practice as a secondary diagnostic tool to assist thoracic surgeons in pre-operative assessments. It can help classify mediastinal tumors based on CT images and CTC data, offering a non-invasive and automated diagnostic approach. However, the adoption of this model in clinical settings will require overcoming challenges such as ensuring data availability (e.g., high-quality CT scans and CTCs data), addressing the initial costs associated with model integration, and obtaining regulatory approvals from agencies like the FDA. Despite these challenges, the long-term benefits, including improved diagnostic accuracy, faster patient triage, and reduced need for invasive procedures, make this model a promising candidate for real-world clinical use. Once implemented, this system could significantly reduce diagnostic costs and improve patient outcomes, especially for those undergoing complex cancer treatments.

Some limitations should be considered. First, there are some inherent flaws in retrospective research. For example, we cannot control the external factors that affect CTCs. Second, the ability of DMFN model to generalize across large, diverse datasets further highlights its robustness, particularly when applied to cases with varying patient demographics and disease progression. Third, despite passing the validation of the test dataset, single-center research may still have selection bias. Multi-center studies are necessary to confirm our predictive models.

Conclusions

In this study, we proposed the DMFN model combined with CTCs and CT images, as a useful method to predict the classification of mediastinal lesions. And this model exhibited satisfactory diagnostic performance compared to thoracic surgeons. Therefore, this novel model may facilitate the diagnosis and help surgeons determine surgery strategy.