Multi-View Information Fusion in Medical Image Analysis: Beyond Feature Concatenation

The combination of multi-view information is crucial in both CEM (contrast-enhanced magnetic resonance imaging) and mammographic image analysis. Currently, there is no comprehensive and systematic method for combining multi-view information in CEM imaging. In mammographic imaging, there are several methods for combining multi-view information, but most of them use a simple strategy called feature concatenation. For example, Sun et al. used two independent CNN networks to extract features from CC (cranio-caudal) and MLO (mediolateral oblique) views, and then concatenated these features and inputted them into a subsequent network. Nasir et al. used four independent CNN networks to extract features from four views and concatenated these features after global pooling layer. Then, they inputted the fused features into a fully connected (fc) layer to obtain the final result. Wu et al. also concatenated features after global pooling, but their work differs from Nasir et al. in two aspects: (1) they used two consecutive fc layers after feature concatenation to better learn the relationship between different view features; (2) all CC views shared one feature extraction network, and all MLO views shared another feature extraction network. Additionally, they compared the performance of different fusion strategies. Li et al. compared the performance of extracting features from different views using independent model parameters and shared model parameters, and concluded that using shared model parameters to extract features from different views achieved better results. Other networks also used feature concatenation strategy at different positions in the model to fuse multi-view information (e.g., Liu et al., Yan et al., Yang et al.). However, since different views may not be aligned, most concatenation-based methods fuse multi-view information after the global pooling layer, leading to the loss of a large amount of local information and the underutilization of multi-view information.