Lidar Scanning for Filmed Entertainment or How I Learned to Stop Worrying & Love the Point Cloud
From ADG Art Direction Wiki
by Alan Lasky, Digital Effects Artist
Contents |
The Rise Of 3D Capture
The current trend of films, commercials and television programs featuring extensive visual effects has given rise to a new craft in motion picture production. Although this craft is divided into many separate disciplines (match moving, 3D integration, on-set data collection, etc), the domain can generally be defined as 3D Capture. The goal of 3D Capture is to construct an accurate three-dimensional representation of a set, location, object, or camera move in order to facilitate the creation of computer generated visual effects.
The last several years have witnessed an explosion of visual effects tools specifically designed for the generation of 3D models from reality. Techniques such as photogrammetry, image based modeling, land survey, and a host of others are used regularly in the film industry to derive three-dimensional data from real world objects. Many of these tools have been around for decades but are only now being utilized for filmed entertainment. Recently many of us have seen crew members from the visual effects department measuring, photographing, scanning and marking sets and locations for post production processing. All of this work falls under the banner of 3D Capture.
Enter Lidar
Into this situation comes one of the most powerful, yet often most misunderstood tools for 3D Capture: Lidar. Lidar, often erroneously referred to as Laser Radar, actually is an acronym for LIght Detection And Ranging. So, what exactly is Lidar? The term covers a broad range of technologies used in a number of different fields such as atmospheric research, terrain mapping, and astronomy. However, the Lidar domain of interest to film professionals is a set of technologies used mainly in the construction and survey industry known collectively as 3D Laser Scanning.
3D Laser Scanning with off the shelf Lidar equipment has replaced many standard survey techniques and the use of Lidar scanners has become standard in the building and construction industries for the creation of as built surveys. Lidar scanners from companies such as Leica Geosystems, Trimble, Optech and Riegl are very portable, fixed-head pulsed lasers that provide high-speed scanning capability to accurately portray existing or as-built environments. In operation, the pulsed lasers emit a series of light pulses at a rate of up to 1800 points per second to capture the surface of any object in three-dimensional space.
Lidar scanners work on a relatively simple principle. Because the speed of light is known, the scanner can measure the time of flight of a laser pulse from an emitter back to a receiver and record an x, y, and z coordinate for each reflected point. Robust and descriptive 3-D point clouds are created, providing coordinate information accurate to ¼” for millions of points. These point clouds are stitched together and intelligently reduced to produce deliverables in a number of different formats. This process makes up the range of techniques known as Lidar Scanning.
In practice, Lidar scanning is not much different than conventional surveying. The scanner is connected to a host computer (usually a field laptop) where the collected laser range measurements are stored in a file. This collection of measured coordinates is called a point cloud and it forms the basis for all Lidar 3D operations. Often multiple scans are necessary to fill in the 3D topology of an object from all angles and cover any occlusion that may occur from fixed point scans.
Some Lidar scanners cover 360 degrees horizontally and, depending upon the power of the laser, their range can be as high as 1000 meters. Quite large areas can be scanned, and by stitching together multiple point clouds there is no theoretical limit (exclusive of computer storage and processing) to the size of the area that can be captured. A registered point cloud of a Venice Beach basketball court was scanned for the video game Street Hoops (2002) and the area captured went well beyond the court itself. The basketball court was scanned with a relatively low-power Lidar unit; other higher powered scanners could capture data well outside this range. The trade-off in Lidar scanning is always range versus eye safety. As the range increases, so must the power of the laser. Obviously the greater the laser power, the greater the threat to the human eye, so most Lidar scanners (apart from those used in military applications) are limited to eye-safe power levels.
Lidar And The Film Industry: Early Projects
Around 1996 Lidar began to be used sporadically in the film industry. One of the first applications was in the area of live action/CGI integration, especially in cases where 3D character animation needed to interact with live-action environments. Combining live action photography and digital characters is a very tricky process, and capture of detailed spatial data about a set or location is critical to creating a seamless blend between photographed elements and CGI.
Starship Troopers (1997, Allan Cameron, Production Designer) was one of the first films to employ 3D scanning for this purpose. The 3D computer generated characters in the film, in this case the giant bugs that menaced the future soldiers, needed to be accurately locked to the topology of the sets and locations photographed during the live action shoot. Many of the sets and locations on Starship Troopers consisted of organic shapes such as rocks, hills and caves where standard measurement and modeling techniques proved inadequate. The decision was made by the visual effects crew to capture some of the sets and locations using Lidar scanning technology. Lidar allowed for detailed 3D models of the sets and locations to be created relatively quickly, thereby assisting the animators in combining the 3D CGI warrior bugs with the live action plates. It was at this time that Lidar scanners gained a reputation as 3D plate cameras among visual effects professionals and more productions began to consider Lidar scanning as a necessary addition to the VFX camera pipeline.
Lidar’s use for 3D plate photography was taken a step further on Dinosaur (2000, Walter Martishius, Production Designer), Disney’s bold experiment in the fusion of live action and 3D CGI characters. In fact, Dinosaur was unique in that the production consisted exclusively of 3D computer generated characters composited with Vista-Vision liveaction photography. The entire film was a large scale plate shoot with crews filming visual effects elements in Florida, Hawaii, Venezuela and other locations without principal actors. Dinosaur was one of the first films to field a dedicated 3D capture crew consisting of surveyors, field match-movers, and photogrammetry specialists as part of the visual effects camera department. Late in production, the Dinosaur visual effects crew decided to use Lidar to capture some of the more topologically complex locations appearing in the film.
The Lidar data was again used to register 3D character animation with live action photography, but its use was expanded for 3D shadow geometry and particle system collision.
Once the fundamentals of using Lidar for visual effects had been resolved, more films began to exploit the benefits of 3D scanning technology. Projects such as End Of Days (1999, Richard Holland), Scooby Doo (2002, Bill Boes), The Day After Tomorrow (2004, Barry Chusid), Enemy At The Gates (2001, Wolf Kroeger), Batman Begins (2005, Nathan Crowley), World Trade Center (2006, Jan Roelfs) and many others employed 3D scanning to solve complex visual effects problems. Due to the high cost of the gear ($100K plus) very few visual effects facilities chose to purchase in-house Lidar equipment. Scanning was traditionally out-sourced to service bureaus and survey companies, often with little or no experience in film production or visual effects.
This situation created growing pains as the two very different cultures of the film industry and the construction survey industry had to find common ground in order to solve very difficult technical challenges. Adding fuel to the fire was the fact that Lidar scanning was frequently added to a project very late in the production cycle without adequate consideration given to budget or scheduling. In its infancy Lidar scanning struggled with a reputation for being expensive, overly complex, and time consuming mainly due to poor planning and oversight. Thankfully, many of those problems are behind us and Lidar has grown into a robust tool for 3D capture in film production.
Lidar In Production
Lidar scanning has many applications in modern film production, from pre-visualization to visual effects. Starting in preproduction Lidar can be used for pre-visualization of complex scenes by sending a Lidar crew out on a 3D location survey. Much as a location scout takes photographs, the Lidar crew scans the location and delivers an accurate 3D model to production. This model can be used to plan the logistics of a complex location shoot. Although it may seem extravagant to use Lidar for this purpose, significant benefits can be gained from this process. Detailed 3D geometry representing a location can be used to plot camera angles with photographic accuracy. Camera moves can be designed with exact measurements of dolly track and crane clearance. Geographically referenced and aligned models can be used to track sun positions throughout the day.
Of course a precise 3D model of a set or location is also beneficial to the Art Department. Lidar scan data can be fed into CAD software to enhance the set design process. A 3D scan gives precise data to construction crews in order to facilitate rapid fabrication of set pieces and props. We scanned a building in Prague that was used for the greens department to determine how many prop vines would be needed to cover the exterior for A Sound Of Thunder (2005, Richard Holland). This model was also used to calculate how much material would be required to cover the Czech writing on the outside of the building. Certainly Lidar data can be used anywhere accurate CAD drawings are needed in pre-production.
Visual Effects is the department where Lidar has the most applications. The integration of CGI and live action photography is always a difficult problem, and Lidar scanning provides a significant advantage to the visual effects pipeline. Lidar can be used in a number of ways to assist the blending of computer generated elements with live-action cinematography. In fact, use of Lidar data in preproduction can substantially affect the quality and efficiency of visual effects material. Lidar data can act as a bridge uniting the Art Department and visual effects crew very early in a production through the use of shared data.
One of the more interesting uses for Lidar scan data is for what is known as collision geometry for particle systems. A particle system is a computer graphics technique used to simulate natural phenomena such as fire, smoke, or rushing water. Particle systems are usually implemented in 3D space and can be programmed to appear subject to external physical forces such as gravity, wind, friction and collision with other objects. Lidar data can be used as collision geometry to constrain particle system simulations to the topology of realworld sets and locations. A prime example of this is the flood waters on the New York streets in The Day After Tomorrow. Accurate Lidar scans of the buildings and streets were used to guide the particle system flood waters along geometry that exactly matched the photographed environment.
Of particular interest to Art Department professionals is the application of Lidar data to 3D set extensions and 3D matte paintings. One of the more common applications of CGI in current filmed entertainment is the use of computer generated elements to extend sets beyond the scope of what is physically built. These set extensions serve the same purpose as traditional matte paintings, and indeed they have been called 3D matte paintings in certain circumstances. Of course the nature of 3D computer generated imagery allows for much more freedom of camera movement on these set extension elements than was possible with traditional matte paintings.
Like all computer generated elements, it is imperative that set extensions be locked to the live action in order to be convincing. To successfully blend a computer generated set extension with a live action set, some form of 3D model representing the physical set or location is necessary. Limited measurements can be taken on set, or blueprints from the Art Department can be acquired, but these are often imperfect solutions. What is needed is an as-built survey of the set or location to facilitate the creation of elements that lock perfectly into their real world counterparts. Lidar provides an effective method of gathering on set data for the creation of 3D set extensions. A point cloud version of the set provides all the measurement data essential for an accurate lock between the real world and the virtual world. A polygonal or surfaced version can be used along with 3D camera projection techniques to further enhance the realism of the shot.
Another important and time consuming task for today’s visual effects artists is the art of match moving: literally matching the movement of the 3D virtual camera to that of its live action counterpart. Although software exists to perform camera tracking, it is by no means an automated task. A great deal of manual input is required in order to successfully track a shot. Current match-moving software packages have the capability to incorporate measurement data into their mathematical camera solvers, and in fact most recommend the use of constrained points to help the solver more accurately derive a result.
Lidar scans are by definition measurement data. This measurement data can easily be incorporated into the match-moving/camera-tracking pipeline. Sets and locations often contain many elements useful for feature tracking. Window corners, edges, architectural elements, and other features can be used as track points to resolve 3D camera motion. Lidar can significantly enhance the use of these features by providing extremely accurate distance constraints between these tracked points. This measurement data is valuable when tracking complex camera moves. Any point in the Lidar scan can provide accurate measurement reference; therefore any tracked feature in the live action photography can be measured in the scan data. Several weeks and thousands of dollars can be saved by scanning sets and locations requiring match moving.
Room For Improvement
Lidar scanning is a powerful tool for film production but like most high tech tools there are some potential pitfalls inherent in the process. Currently the single largest problem is in the area of point cloud to surface conversion. Lidar data typically has been converted into a CAD environment where clean polygonal or NURBS surfaces are not a priority. Most of the software used to process Lidar data is geared toward CAD -based deliverables, not animation software. Processing the enormous point clouds generated by Lidar scanners requires powerful computation environments, plenty of storage and algorithms that can effectively convert these massive point clouds into usable surfaces. A great deal of work has been done by companies like Geomagic, Inus Technolgy, and Innovmetric to create practical software tools for point cloud to surface processing, but much work remains before these tools can provide one-click Lidar modeling.
Lidar provides a unique solution for 3D capture in filmed entertainment. New uses for scan data are being developed every day and Lidar technology continues to improve. Anyone considering Lidar scanning for a project can contact me or any Lidar professional to assess your needs. No matter what, please do so as early in the production cycle as possible!
