Photogrammetry: Creating 3D Models from Photos with AI

Introduction

Photogrammetry is a 3D scanning process that creates digital 3D models from 2D photographs. It works by analyzing overlapping photographs of an object or scene taken from multiple angles. Sophisticated photogrammetry software, like RealityCapture or Metashape, can process the photographs and generate incredibly detailed 3D models. The capabilities of photogrammetry have grown enormously thanks to advancements in computer vision and AI. Photogrammetry provides an accessible and low-cost method for creating high-quality 3D models, with applications in industries like engineering, architecture, media, and more.

In this article, I will provide an in-depth guide on photogrammetry and how AI empowers the process. I will cover topics like:

• The history and evolution of photogrammetry
• How photogrammetry works to construct 3D models
• Photogrammetry software and usage
• The role of AI in modern photogrammetry
• Applications and use cases for photogrammetry
• Tips for capturing optimal photographs
• The future of AI-powered photogrammetry

By the end of this article, you will have a comprehensive understanding of this transformative technology.

A Brief History of Photogrammetry

The principles behind photogrammetry date back to the 19th century, when early pioneers like Laussedat experimented with measuring photographs. But photogrammetry really took off with aerial surveillance for military mapping in the early 20th century. Stereophotogrammetry with overlapping aerial photos enabled the creation of topographic maps and 3D models of terrain.

In the 1930s and 40s, analog machines like the Stereoplanigraph revolutionized photogrammetry by optically processing stereo imagery. In the 1960s, the DSR-1 was the first digital photogrammetric workstation. In the 1990s, digital cameras, powerful computers and advanced software brought major advances in automation and computational photogrammetry.

Modern photogrammetry has now reached a state of high precision and automation thanks to 3D computer vision and AI. Emerging techniques like neural networks and deep learning have overcome many limitations of traditional photogrammetry.

How Photogrammetry Works

The photogrammetry pipeline consists of several key steps:

1. Image Capture
2. Feature Matching
3. Triangulation / Point Cloud Generation
4. Mesh Generation
5. Texture Mapping

Image Capture

The first step is capturing a set of overlapping photographs surrounding an object or area of interest. Ideal photos will have 60% side overlap and 80% forward overlap at varying angles all around the subject. More detail requires more, denser image capture. Enough overlap is crucial for the software to match common points across the photos.

Ideal photo overlap for photogrammetry

Feature Matching

Photogrammetry software detects and matches distinguishing features across the multiple photos. It identifies feature points like corners, edges, and textures that appear in multiple images from different angles. Sophisticated feature detection and matching algorithms are a key area where AI advances photogrammetry.

Triangulation / Point Cloud Generation

By determining the camera positions and angles from the matched features, photogrammetry triangulates approximate real-world coordinates for the matched points. This generates a sparse point cloud showing matched feature points in 3D space.

Mesh Generation

From the point cloud, a polygonal 3D mesh is constructed by connecting nearby points to approximate surfaces and shapes of the subject. Advanced AI techniques can produce impressively smooth and precise mesh reconstructions.

Texture Mapping

Finally, the original photographic textures are mapped onto the 3D geometry. This adds surface colors, tones, and details to create an ultra-realistic 3D model.

The Role of AI in Modern Photogrammetry

AI has brought major leaps forward in automating and perfecting photogrammetry reconstruction. Here are some of the key areas where AI contributes:

• Feature detection – Machine learning trains algorithms to identify robust features for matching, eliminating the need for special targets.

• Feature matching – AI matching can handle challenging scenarios like low texture, repetitive patterns, complex geometry etc.

• Point cloud generation – Neural networks densely reconstruct 3D points with fewer input photos.

• Mesh generation – Deep learning produces incredible mesh detail and smoothness from minimal point clouds.

• De-noising – AI filters out reconstruction noise, outliers, artifacts etc. for clean models.

• Camera pose optimization – Algorithms precisely determine camera orientations for accurate geometry.

• Texturing – Smart texture application and blending create hyper-realistic 3D models.

In summary, AI handles the most complex aspects of photogrammetry reconstruction in an automated way. This reduces the need for manual effort while improving speed, precision, detail, and robustness.

Photogrammetry Software

There are a variety of specialized photogrammetry applications available to generate 3D models. The leading options include:

• RealityCapture – A top professional choice, using AI for unparalleled automation and precision.

• Metashape – A powerful photogrammetry suite with advanced capabilities.

• 3DF Zephyr – A robust commercial package with excellent results.

• Meshroom – A popular free and open source photogrammetry tool.

• AliceVision – Academic open source project with state-of-the-art algorithms.

• Regard3D – Free software using neural networks for impressive quality.

Key factors in choosing software are reconstruction quality, automation, advanced capabilities, and suitability for your use case. Processing times can range from minutes to hours depending on variables like image resolution, number of images, and hardware.

Photogrammetry Use Cases

Photogrammetry enables digital capture of physical objects and environments. Some of the top use cases and applications include:

Cultural Heritage – Scanning artifacts, monuments, and historic sites for preservation and virtual experiences.

Engineering & Manufacturing – Capturing commercial products, prototypes, or machinery for design and quality control.

Architecture & Construction – Modeling buildings, infrastructure, construction sites, etc. for planning or documentation.

Film & VFX – Creating digital doubles of props, actors, textures, and environments for visual effects.

AR/VR – Scanning real-world objects and places to create 3D assets for augmented and virtual reality.

Gaming – Generating realistic 3D models for game environments and assets.

Science & Medicine – Documenting artifacts, fossils, biological subjects, medical injuries/conditions etc.

Surveying & Mapping – Modeling outdoor sites and terrain for surveying, agriculture, mining, geology, forensics etc.

3D Printing – Scanning objects to 3D print accurate replicas or miniatures.

Photogrammetry offers high fidelity capture for an immense array of applications.

Photogrammetry Photography Tips

To produce optimal 3D reconstructions, follow these photography best practices:

• Use a high resolution camera with low distortion lens. DSLR or mirrorless cameras are ideal.

• Capture a large volume of photos – the more the better! Shoot around every surface from all angles.

• Take photos in a grid pattern with plenty of side and forward overlap.

• Keep a consistent distance to the subject and use focal length consistency.

• Avoid overexposure, blur, glare, shadows, cluttered backgrounds, etc. Capture controlled, clear photos.

• Use diffuse, indirect lighting. No harsh shadows or highlights.

• Enable RAW image capture for maximum processing quality.

• Use small apertures for wide depth of field.

• Shoot handheld for flexibility or use a tripod for stability.

• For moving subjects, use high shutter speed to freeze motion.

With practice and the right techniques, you can master taking photogrammetry pictures suitable for reconstructing highly detailed 3D models.

The Future of AI in Photogrammetry

AI-powered photogrammetry is an extremely promising field with much innovation ahead. Here are some emerging areas of AI research:

• End-to-end deep learning systems that automate the full pipeline.

• Generative networks capable of filling reconstruction gaps.

• Multi-view neural networks for robust feature correspondence.

• Adversarial training for photorealistic texturing and completion.

• Increased use of depth sensors and photometric stereo techniques.

• 3D deep learning on voxelized shape representations.

• Reconstruction from small numbers of images or even single images.

As algorithms grow more advanced, photogrammetry will become faster, easier, more automated, and produce more detailed accurate models from less input data.

Conclusion

Photogrammetry has rapidly evolved into a highly powerful 3D scanning methodology thanks to computer vision and AI techniques. It provides an accessible and low-cost approach to digitizing real-world objects extremely accurately. With the right tools and techniques, anyone can harness photogrammetry to create professional quality 3D models with applications across many industries. The capabilities of AI-powered photogrammetry will only continue to improve in the future, opening new possibilities for digitizing and interacting with the world around us in 3D.