Overcoming the tradeoff between resolution and depth-of-field
Optical microscopy has witnessed notable advancements in recent decades, but this progress has often come at the cost of increased complexity and expense. Conventional wide-field microscopy (WFM) provides a simple and cost-effective solution, but it suffers from a shallow depth-of-field (DOF) and limited resolution, which restricts its practical applications in biological research.
More advanced techniques like confocal and light sheet microscopy have become the new workhorses for high-resolution 3D imaging. However, these approaches require complex scanning systems to acquire multiple axial planes, which increases the overall cost and time required for imaging. Moreover, the extended dwell time of the excitation light can induce stronger phototoxicity in biological samples.
To address these limitations, researchers have explored light field microscopy (LFM) as a method for capturing signals from a large DOF in a single snapshot. LFM utilizes a microlens array (MLA) to separate the spatial and angular information of the light, allowing for efficient capture of volumetric data. However, traditional LFM often compromises spatial resolution to retrieve more 3D pixels, and the computational costs for 3D reconstruction can be prohibitively high, making it impractical for many biological applications.
Introducing deep focus microscopy
To overcome the tradeoff between resolution and DOF, we have developed an efficient imaging framework called deep focus microscopy. This approach combines the LFM setup with a lightweight deep learning network, termed the deep focus network (DFnet), to extract high-resolution information from the large axial coverage captured by the LFM system.
The deep focus microscopy pipeline works as follows:
-
The LFM setup, with an MLA inserted at the image plane, captures macro-pixel images that contain spatial and angular information from a wide DOF.
-
The DFnet, designed with a symmetrical encoder-decoder architecture and attention mechanisms, processes the LFM macro-pixel images to generate high-resolution 2D images that maintain the extended DOF.
-
By leveraging the spatial-angular correlations in the LFM data and employing efficient neural network architectures, deep focus microscopy achieves significantly enhanced resolution, extended DOF, and superior generalization across diverse sample structures, compared to previous LFM reconstruction methods.
The key advantages of deep focus microscopy include:
- Resolution Enhancement: Deep focus microscopy achieves a lateral resolution of ~260 nm, which is more than 10 times higher than conventional WFM.
- Depth-of-Field Extension: The system demonstrates a DOF of over 30 μm, a substantial improvement over the shallow DOF of WFM.
- Computational Efficiency: The DFnet-based reconstruction is approximately 300 times faster than conventional LFM processing, reducing the computational burden and enabling practical application in biological research.
- Versatility: Deep focus microscopy can effectively image a wide range of biological samples, including fluorescently labeled cells, tissues, and even live organisms, with high fidelity and contrast.
Principle of deep focus microscopy
The depth-of-field (DOF) of an optical microscope is inversely proportional to the numerical aperture (NA) of the objective lens. By separating the light rays based on their angles at the pupil plane, LFM can achieve an extended DOF compared to WFM, which collects all the light rays without discrimination.
In deep focus microscopy, we adopt a simple yet cost-effective LFM setup by inserting an MLA at the image plane of a commercial microscope. This configuration allows us to capture macro-pixel images that contain spatial and angular information over a wide DOF (Figure 1a).
However, traditional LFM reconstruction methods, which aim to retrieve 3D volumes from the limited measured pixels, often struggle with resolution and computational efficiency. To address this, deep focus microscopy directly reconstructs a high-resolution 2D image that preserves the extended DOF information, without the need for computationally expensive 3D volume reconstruction (Figure 1b).
Furthermore, deep focus microscopy demonstrates performance comparable to confocal microscopy in terms of resolution and contrast, while requiring significantly fewer axial planes (up to 100-fold reduction) to capture the volumetric information (Figure 1c).
Figure 1. Principle of deep focus microscopy. (a) Schematics of WFM and deep focus microscopy. WFM collects the intensity of all photons, so it suffers from a shallow DOF and limited optical sectioning, while deep focus microscopy, which relies on a microlens array (MLA) in hardware and a neural network in algorithm, obtains an extended DOF within a single 2D image. (b) The comparison of a fixed L929 cell in the membrane channel, obtained by WFM, light field deconvolution (LFD), and deep focus microscopy. The corresponding Fourier components are shown on the right with estimated resolution by Fourier ring correlation (FRC). The processing time of the latter two methods is also marked, measured on data with an output size of 1989 × 1989 pixels. (c) The comparison of imaging mode between confocal microscopy and deep focus microscopy. In confocal microscopy, different axial planes are acquired individually and then projected into a plane for visualization, whereas deep focus microscopy allows for capturing volumetric information within a single image with comparable resolution. The right shows the reduction in the number of axial layers using deep focus microscopy.
Optimizing the deep focus network (DFnet)
To fully exploit the physical priors offered by the spatial-angular correlations in the LFM data, we have designed the DFnet with a symmetrical encoder-decoder architecture and an attention mechanism within the skip connections. This approach allows the network to effectively extract and enhance fine-grained spatial details and informative contextual features from the input macro-pixel images (Figure 2a).
The raw LFM measurements, with a size of 1989 × 1989 pixels, are first rearranged into a 3D image stack of 153 × 153 × 169 pixels (height × width × angle). This input is then fed into the DFnet, which outputs a high-resolution 2D image of 1989 × 1989 pixels.
To construct the training dataset, we also utilized a scanning LFM (sLFM) system to acquire high-resolution reference images. By realigning the angular components from different microlenses, sLFM can provide 2D images with a resolution significantly higher than the original LFM measurements. These high-resolution sLFM images are used as the ground truth targets for training the DFnet.
The network is trained using a mean squared error (MSE) loss function, and the training process typically converges within 10,000 iterations (Figure 2b). The final DFnet model is able to generate high-quality, high-resolution images that maintain the extended DOF captured by the LFM setup.
Figure 2. Overview of DFnet architecture. (a) The structure of DFnet, which employs a multi-scale symmetrical encoder-decoder architecture with an attention mechanism followed by an up-sampling block. The pixel size of each layer is indicated. The detailed architectures of the channel and spatial attention (CASA) module and cascaded up-sampling layers are shown below with symbol explanations. (b) Example loss-versus-iteration curve trained on 100-nm-diameter fluorescent beads. Mean squared error (MSE) is used as the loss function for network training. The network usually converges at 10 k iterations. The outputs of DFnet at different training phases are illustrated on the right.
Comparative evaluation of deep focus microscopy
To demonstrate the capabilities of deep focus microscopy, we compared its performance against other methods, including light field deconvolution (LFD) and a deep learning-based approach called VCD-Net.
For the LFD method, we downloaded the source code and implemented it with our data. The LFM images, with a size of 153 × 153 × 169 pixels, were used as input, and the output reconstructed volumes had a size of 1989 × 1989 × 101 pixels. Finally, we projected the maximum intensity of the reconstructed volumes along the axial dimension to obtain the MIPs for comparison.
For the VCD-Net method, we directly ran the original training code with our dataset, treating the LFM images as input and the high-resolution volumes reconstructed by iterative tomography as targets.
The hardware system for deep focus microscopy was built upon a commercial inverted microscope equipped with a 63×/1.4 NA oil-immersion objective. A customized MLA with a pitch size of 84.5 μm and a focal length of 1800 μm was inserted at the image plane before the camera.
We also tested the robustness of deep focus microscopy to different MLA settings, using four configurations with varying pitch sizes and focal lengths. The results demonstrated that the performance was stable across the different MLA parameters, indicating the flexibility of our approach (Figure 3).
Figure 3. Robustness test of deep focus microscopy to different MLAs. (a) Simulation of 1-μm-diameter tubulins under different MLA settings imaged by our deep focus microscopy. The network was pretrained and validated on simulated dataset under the same setting. The zoom-in patches are shown in the second row. (b)-(c) Boxplots of PSNR and SSIM by deep focus microscopy with different MLAs (n = 10 samples per MLA setting).
Resolution and contrast enhancement
To assess the resolution enhancement of deep focus microscopy, we imaged fluorescent beads with sub-diffraction-limit diameters (100 nm) randomly distributed in a 3D space. The results demonstrate that deep focus microscopy can clearly resolve two closely positioned beads, while WFM, LFD, and VCD-Net struggled to distinguish them due to the limited resolution (Figure 4a-c).
Quantitative analysis of the full-width at half-maximum (FWHM) of the bead intensity profiles showed that deep focus microscopy achieved a more than 10-fold improvement in resolution compared to WFM, a 5-fold improvement over LFD, and a 2-fold improvement over VCD-Net (Figure 4e).
Figure 4. Resolution characterization of deep focus microscopy. (a) Images of 100-nm fluorescence beads obtained by WFM and deep focus microscopy with a 63×/1.4 NA oil-immersion objective. (b) The four zoom-in regions applied for different methods corresponding to subregions marked in (a). (c) Normalized intensity profiles along the white dashed lines marked in (b). (d) The four zoom-in regions corresponding to subregions marked in (a) obtained by deep focus microscopy, which is trained with the dataset with multiple sample types. (e) Bar chart of lateral resolution of four different methods (n = 10 beads per method), which was calculated by measuring FWHMs after Gaussian fit.
We further validated the extended DOF achieved by deep focus microscopy by imaging a USAF-1951 resolution chart at different axial positions. While WFM and digital refocusing methods suffered from severe blurring and contrast loss as the object moved away from the focal plane, LFD and VCD-Net maintained focus over a larger DOF but exhibited detail loss and artifacts.
In contrast, deep focus microscopy was able to simultaneously obtain high resolution and large DOF, clearly distinguishing the line pairs (up to group 1, line 6) with high contrast throughout the axial range (Figure 5a-c). Quantitative analysis of the modulation transfer function (MTF) curves confirmed that deep focus microscopy achieved an extended DOF of over 30 μm, a significant improvement compared to WFM (Figure 5c-d).
Figure 5. Extended DOF of deep focus microscopy. (a) Simulation of a USAF-1951 resolution chart at different axial positions, obtained by WFM, digital refocusing, LFD, VCD-Net, deep focus microscopy, and confocal microscopy (as the ground truth). The intensity profile of the yellow line is attached beside each sub-image. (b) The modulation transfer function (MTF) of the line pairs (group 1 line 6, 1.1 cycles/μm) at different axial positions. (c) The DOF comparisons between five methods. (d) Simulation of a USAF-1951 resolution chart at the axial position of z = 0 μm, obtained by deep focus microscopy trained with the dataset with multiple sample types.
Generalization and efficiency
Biological samples exhibit diverse structures and functions, which can pose challenges for deep learning methods in terms of their generalization ability. Retraining the network on newly captured data can be time-consuming, negating the advantage of fast processing provided by deep learning.
To address this, we constructed a large dataset consisting of 1,400 pairs of light field raw macro-pixel images and their corresponding high-resolution 2D counterparts, spanning both experimental and synthetic data (Figure 6e). This dataset allowed the DFnet to learn the universal physical mapping from the input to the desired output, rather than overfitting to specific sample textures.
Comparative evaluation on L929 cells labeled with membrane and mitochondria channels showed that the deep focus microscopy approach, trained on the constructed dataset, achieved higher image correlation and a wider frequency range compared to VCD-Net (Figure 6a-d).
Furthermore, the DFnet in deep focus microscopy exhibited significantly higher efficiency, with 345 times fewer floating-point operations, 2-fold fewer network parameters, and a 100-fold faster inference speed than VCD-Net (Figure 6f-h).
Figure 6. Evaluation of generalization ability and efficiency. (a)-(b) Cross-sample test for VCD-Net (a) and DFnet (b) on fixed L929 cells labeled with membrane and mitochondria. (c) The ground-truth data were captured by sLFM after reconstruction. (d) Boxplots of Pearson correlations between VCD-Net and DFnet trained with different datasets. (e) Illustration of our constructed dataset, which includes both experimental and synthetic data. (f)-(h) Comparisons between VCD-Net and DFnet on network parameters, floating-point operations per second (FLOPs), and processing time.
Applications in biological imaging
The high-resolution and extended DOF capabilities of deep focus microscopy make it a powerful tool for observing intricate subcellular dynamics and complex tissue structures within living organisms.
We demonstrated the performance of deep focus microscopy in imaging zebrafish embryos and mouse livers, capturing processes such as cell division and migrasome formation with high fidelity (Figure 7). Compared to WFM and VCD-Net, deep focus microscopy was able to clearly visualize these fine details without background contamination or artifacts.
Figure 7. In-vivo high-resolution recordings of zebrafish embryo development. (a)-(b) Images of a fixed zebrafish embryo by WFM (a) and deep focus microscopy (b) with a 63×/1.4 NA oil-immersion objective. (c)-(e) Time stamps showing the cell division process clearly visualized by deep focus microscopy within the enlarged regions.
We also applied deep focus microscopy to image the mouse liver vasculature in an ischemic stroke model, leveraging the extended DOF to capture the full 3D vascular network without the need for serial sectioning and reconstruction (Figure 8). This capability allowed us to effectively visualize the disease-related changes in the vascular structure, demonstrating the versatility of deep focus microscopy in neuroscience research.
Figure 8. Intravital high-contrast imaging of neutrophils in living mouse livers. (a) Whole slide image of the mouse kidney tissue section captured with the E2E-BPF microscope. (b), (d) Magnified images from the standard microscope of the regions marked with yellow dashed lines in (a). (c), (e) Magnified images from the E2E-BPF microscope of the same regions.
Conclusion
In this work, we have developed deep focus microscopy, an efficient framework that combines light field microscopy with a deep learning-based approach to address the tradeoff between resolution and depth-of-field. By integrating the LFM setup with the DFnet, our method achieves a significant enhancement in spatial resolution, extended depth-of-field, and superior generalization across diverse biological samples, while reducing the computational burden by several orders of magnitude compared to traditional LFM reconstruction techniques.
The key advantages of deep focus microscopy include: