Recent years have witnessed great advances in applying deep learning to improve fluorescence microscopy imaging. However, enhancing the fidelity of image restoration networks and improving their robustness under fluorescence noise remain significant challenges. Professor Xi Peng's team from the College of Future Technology at Peking University has developed LargePNet, a novel general-purpose fluorescence image restoration network. By exploiting large-view structural correlations in biological fluorescence images, LargePNet aggregates large-view statistical information through a dedicated network architecture, overcoming the loss of global contextual information caused by conventional patch-based training. The method significantly improves restoration accuracy and large-size image inference efficiency. The work, entitled 'Pushing the limits of fluorescence imaging with a restoration neural network aggregating large-view statistics,' was recently published in Nature Communications, providing powerful imaging support for long-term live-cell fluorescence imaging and multicolor super-resolution microscopy.
Deep neural networks such as UNet, RCAN, and SwinIR have achieved remarkable success in image restoration and enhancement and have been widely adopted in fluorescence microscopy. By learning mappings from low-quality images to high-quality images, restoration networks reduce the photon dose required for microscopy imaging, thereby improving imaging speed and duration. However, most existing studies follow practices from natural-image processing by randomly cropping large images into small patches, typically 128×128 pixels, for data augmentation and training. This strategy ignores long-range biological structural correlations present in fluorescence images and causes the loss of critical global contextual information. Furthermore, differences in structural statistics between small patches and full-field images introduce additional errors when patch-trained networks are applied to large-image restoration tasks.
To address these limitations, the researchers systematically investigated how to efficiently utilize large-view information for fluorescence image restoration. Because spatial self-attention is computationally expensive for ultra-large fields of view, the team adopted re-parameterized large-kernel convolutions (RepLKConv) for long-range modeling. To compensate for the limited nonlinear representation capability of large-kernel convolutions, they designed a pyramid architecture incorporating a low-frequency branch from conventional deep networks. Instance normalization was further introduced to improve training stability on large images. Ablation studies demonstrated the complementary roles of the two branches. Through training experiments using different image sizes and effective receptive field analyses, the authors showed that LargePNet effectively captures large-view information, with restoration performance improving as training image size increases. The team also developed several extensions, including LargeP-GAN for generative restoration, LargeP-TISR for video super-resolution, 3D-LargePNet for volumetric restoration, and LargeP-SN2N for self-supervised denoising.
The researchers evaluated LargePNet on eight representative fluorescence imaging tasks, including denoising, deblurring, single-image and video super-resolution, sampling recovery, and background removal across multiple microscopy modalities. LargePNet was trained directly on images larger than 512×512 pixels without random cropping, preserving large-view structural information during learning. Compared with state-of-the-art CNN methods such as DFCAN, Transformer-based methods such as SwinIR, and foundation-model fine-tuning approaches such as UniFMIR, LargePNet achieved PSNR improvements of 0.5–2 dB over the best existing patch-based networks. For large-image inference, its computational efficiency was approximately four times higher than advanced CNNs and twenty times higher than Transformer-based models.
Leveraging the superior performance of LargePNet, the team achieved important advances in live-cell imaging. First, they demonstrated continuous live-cell organelle imaging for up to 30 hours at 200 nm resolution, enabling stable monitoring of cytoskeletal dynamics. Second, they performed hour-long three-color STED super-resolution imaging, clearly resolving interactions among the endoplasmic reticulum, mitochondria, and microtubules. These achievements provide a highly precise and stable imaging platform for studying cellular biological mechanisms.
By efficiently extracting large-view structural information through a carefully designed architecture, LargePNet significantly improves fluorescence image restoration accuracy and overall imaging performance, representing an important advance for computational live-cell imaging. The team further analyzed LargePNet using gray-level co-occurrence matrix (GLCM) statistics and found that the greater the discrepancy between patch-level and full-image GLCM statistics, the larger the performance advantage of LargePNet over conventional patch-trained networks. This observation is consistent with most experimental results reported in the study and provides guidance for the practical deployment of LargePNet.
To facilitate adoption of LargePNet, the research team has publicly released the complete Python source code, training datasets, and pretrained models: https://github.com/YiweiHou/LargePNet-for-fluorescence-image-restoration
