Compositing is the combining of visual elements from separate sources into single images, often to create the illusion that all those elements are parts of the same scene. Live-action shooting for compositing is variously called "chroma key", "blue screen", "green screen" and other names. Today, most, though not all, compositing is achieved through digital image manipulation. Pre-digital compositing techniques, however, go back as far as the trick films of Georges Méliès in the late 19th century; and some are still in use.

Basic procedure[link]

All compositing involves the replacement of selected parts of an image with other material, usually, but not always, from another image. In the digital method of compositing, software commands designate a narrowly defined color as the part of an image to be replaced. Then every pixel within the designated color range is replaced by the software with a pixel from another image, aligned to appear as part of the original. For example, a TV weather person is recorded in front of a plain blue or green screen, while compositing software replaces only the designated blue or green color with weather maps.

Typical applications[link]

In television studios, blue or green screens may back news-readers to allow the compositing of stories behind them, before being switched to full-screen display. In other cases, presenters may be completely within compositing backgrounds that are replaced with entire “virtual sets” executed in computer graphics programs. In sophisticated installations, subjects, cameras, or both can move about freely while the computer-generated imagery (CGI) environment changes in real time to maintain correct relationships between the camera angles, subjects, and virtual “backgrounds.”

Virtual sets are also used in motion pictures filmmaking, some of which are photographed entirely in blue or green screen environments; as for example in Sky Captain and the World of Tomorrow. More commonly, composited backgrounds are combined with sets – both full-size and models – and vehicles, furniture, and other physical objects that enhance the “reality” of the composited visuals. “Sets” of almost unlimited size can be created digitally because compositing software can take the blue or green color at the edges of a backing screen and extend it to fill the rest of the frame outside it. That way, subjects recorded in modest areas can be placed in large virtual vistas. Most common of all, perhaps, are set extensions: digital additions to actual performing environments. In the film, Gladiator, for example, the arena and first tier seats of the Roman Colosseum were actually built, while the upper galleries (complete with moving spectators) were computer graphics, composited onto the image above the physical set. For motion pictures originally recorded on film, high-quality video conversions called “digital intermediates” are created to enable compositing and the other operations of computerized post production. Digital compositing is a form of matting, one of four basic compositing methods. The others are physical compositing, multiple exposure, and background projection.

Physical compositing[link]

In physical compositing the separate parts of the image are placed together in the photographic frame and recorded in a single exposure. The components are aligned so that they give the appearance of a single image. The most common physical compositing elements are partial models and glass paintings.

Partial models are typically used as set extensions such as ceilings or the upper stories of buildings. The model, built to match the actual set but on a much smaller scale, is hung in front of the camera, aligned so that it appears to be part of the set. Models are often quite large because they must be placed far enough from the camera so that both they and the set far beyond them are in sharp focus.^[1]

Glass shots are made by positioning a large pane of glass so that it fills the camera frame, and is far enough away to be held in focus along with the background visible through it. The entire scene is painted on the glass, except for the area revealing the background where action is to take place. This area is left clear. Photographed through the glass, the live action is composited with the painted area. A classic example of a glass shot is the approach to Ashley Wilkes’ plantation in Gone with the Wind. The plantation and fields are all painted, while the road and the moving figures on it are photographed through the glass area left clear.

A variant uses the opposite technique: most of the area is clear, except for individual elements (photo cutouts or paintings) affixed to the glass. For example, a ranch house could be added to an empty valley by placing an appropriately scaled and positioned picture of it between the valley and the camera.

Multiple exposure[link]

An in-camera multiple exposure is made by recording on only one part of each film frame, rewinding the film to exactly the same start point, exposing a second part, and repeating the process as needed. The resulting negative is a composite of all the individual exposures. (By contrast, a “double exposure” records multiple images on the entire frame area, so that all are partially visible through one another.) Exposing one section at a time is made possible by enclosing the camera lens (or the whole camera) in a light-tight box fitted with maskable openings, each one corresponding to one of the action areas. Only one opening is revealed per exposure, to record just the action positioned in front of it.

Multiple exposure is difficult because the action in each recording must match that of the others; so multiple exposure composites typically contain only two or three elements. However, as early as 1900 Georges Méliès used seven-fold exposure in L'homme-orchestre/The One-man Band; and in the 1921 film The Playhouse, Buster Keaton used multiple exposures to appear simultaneously as nine different actors on a stage, perfectly synchronizing all nine performances.

Background projection[link]

Background projection throws the background image on a screen behind the subjects in the foreground while the camera makes a composite by photographing both at once. The foreground elements conceal the parts of the background image behind them. Sometimes, the background is projected from the front, reflecting off the screen but not the foreground subjects because the screen is made of highly directional, exceptionally reflective material. (The prehistoric opening of 2001: A Space Odyssey uses front projection.) However, rear projection has been a far more common technique.

In rear projection, background images (called “plates”, whether they are still pictures or moving) are photographed first. For example, a camera car may drive along streets or roads while photographing the changing scene behind it. In the studio, the resulting “background plate” is loaded into a projector with the film "flipped" (reversed), because it will be projected onto (and through) the back of a translucent screen. A car containing the performers is aligned in front of the screen so that the scenery appears through its rear and/or side windows. A camera in front of the car records both the foreground action and the projected scenery, as the performers pretend to drive.

Like multiple exposure, rear projection is technically difficult. The projector and camera motors must be synchronized to avoid flicker and perfectly aligned behind and before the screen. The foreground must be lit to prevent light spill onto the screen behind it. (For night driving scenes, the foreground lights are usually varied as the car “moves” along.) The projector must use a very strong light source so that the projected background is as bright as the foreground. Color filming presents additional difficulties, but can be quite convincing, as in the famous crop duster sequence in Alfred Hitchcock’s North by Northwest. Because of its complexity, rear projection has been largely replaced by digital compositing with, for example, the car positioned in front of a blue or green screen.

Matting[link]

Traditional matting is the process of compositing two different film elements by printing them, one at a time, onto a duplicate strip of film. After one component is printed on the duplicate, the film is re-wound and the other component is added. Since the film cannot be exposed twice without creating a double exposure, the blank second area must be masked while the first is printed; then the freshly exposed first area must be masked while the second area is printed. Each masking is performed by a “traveling matte:” a specially altered duplicate shot which lies on top of the copy film stock.

Like its digital successor, traditional matte photography uses a uniformly colored backing – usually, but not always a special blue or green (fig. 1). Because a matching filter on the camera lens screens out only the backing color, the background area records as black, which, on the camera’s negative film, will develop clear (fig. 2).

First, a print from the original negative is made on high-contrast film, which records the backing as opaque and the foreground subject as clear (fig. 3). A second high-contrast copy is then made from the first, rendering the backing clear and the foreground opaque (fig. 4).

Next, a three-layer sandwich of film is run through an optical printer. On the bottom is the unexposed copy film. Above it is the first matte, whose opaque backing color masks the background. On top is the negative of the foreground action. On this pass, the foreground is copied while the background is shielded from exposure by the matte (fig. 5).

Then the process is repeated; but this time, the copy film is masked by the reverse matte, which excludes light from the foreground area already exposed (fig. 6). The top layer contains the background scene (fig. 7), which is now exposed only in the areas protected during the previous pass. The result is a positive print of the combined background and foreground (fig. 8). A copy of this composite print yields a “dupe negative” (fig. 9) that will replace the original foreground shot in the film’s edited negatives.

Advantages of digital mattes[link]

Digital matting has replaced the traditional approach for two reasons. In the old system, the five separate strips of film (foreground and background originals, positive and negative mattes, and copy stock) could drift slightly out of registration, resulting in halos and other edge artifacts in the result. Done correctly, digital matting is perfect, down to the single-pixel level. Also, the final dupe negative was a “third generation” copy, and film loses quality each time it is copied. Digital images can be copied without quality loss.

This means that multi-layer digital composites can easily be made. For example, models of a space station, a space ship, and a second space ship could be shot separately against blue screen, each "moving" differently. (In such shots, it is the camera that moves, not the model). The individual shots could then be composited with one another, and finally with a star background. With pre-digital matting, the several extra passes through the optical printer would degrade the film quality and increase the probability of edge artifacts. Elements crossing behind or before one another would pose additional problems.

See also[link]

• Alpha compositing
• Broadcast designer
• Character generator
• Clean feed (TV)
• Compositing window manager
• Digital asset
• Digital compositing
• Digital on-screen graphic (Bug or DOG)
• Graphics coordinator
• Motion graphic design
• Primatte chromakey technology

References[link]

Further reading[link]

• T. Porter and T. Duff, "Compositing Digital Images", Proceedings of SIGGRAPH '84, 18 (1984).
• Ron Brinkmann, The Art and Science of Digital Compositing (ISBN 0-12-133960-2)
• Steve Wright, Digital Compositing for Film and Video, Second Edition (ISBN 0-240-80760-X)
• American Cinematographer Manual, 2nd ed., Mascelli, Joseph V., A.S.C. and Miller, Arthur, A.S.C, eds. Los Angeles, 1966, p. 500 ff.

External links[link]

Look up compositing in Wiktionary, the free dictionary.

• Compositing tutorial on FXhome.com

This article may require copy editing for grammar and tense. You can assist by editing it. (August 2011)

Today's practicality of green screen compositing is demonstrated by Iman Crosson in a self-produced YouTube video.
Top panel: A frame of Crosson in a full motion video that was shot in his own living room.
Bottom panel: The corresponding frame in the final version, in which Crosson, impersonating Barack Obama, "appears" in the White House's East Room.^[1]

Chroma key compositing, or chroma keying, is a special effects / post-production technique for compositing (layering) two images or video streams together, used heavily in many fields to remove a background from the subject of a photo or video - particularly the newscasting, motion picture and videogame industries. A color range in the top layer is made transparent, revealing another image behind. The chroma keying technique is commonly used in video production and post-production. This technique is also referred to as color keying, colour-separation overlay (CSO; primarily by the BBC^[2]), or by various terms for specific color-related variants such as green screen, and blue screen - chroma keying can be done with backgrounds of any color that are uniform and distinct, but green and blue backgrounds are more commonly used because they differ most distinctly in hue from most human skin colors and no part of the subject being filmed or photographed may duplicate a color used in the background.^[3] It is commonly used for weather forecast broadcasts, wherein the news presenter appears to be standing in front of a large map during live television newscasts, though in actuality it is a large blue or green background. When using a blue screen, different weather maps are added on the parts of the image where the color is blue. If the news presenter wears blue clothes, his clothes will also be replaced with the background video. A complementary system is used for green screens. Chroma keying is also used in the entertainment industry for special effects in movies and videogames. The advanced state of the technology and much commercially available computer software, such as Pinnacle Studio, Adobe Premiere, and dozens of other computer programs, makes it possible and relatively easy for the average home computer user to create videos using the "chromakey" function and easily affordable greenscreen or bluescreen kits.

History[link]

In filmmaking, a complex and time-consuming process known as "travelling matte" was used prior to the introduction of digital compositing. The blue screen and traveling matte method were developed in the 1930s at RKO Radio Pictures and other studios, and were used to create special effects for The Thief of Bagdad (1940). At RKO, Linwood Dunn used a travelling matte to create "wipes" – where there were transitions like a windshield wiper in films such as Flying Down to Rio (1933).

The development of the bluescreen is credited to Larry Butler, who won the Academy Award for special effects for The Thief of Bagdad.

In 1950, Warner Brothers employee and ex-Kodak researcher Arthur Widmer began working on an ultra violet travelling matte process. He also began developing bluescreen techniques: one of the first films to use them was the 1958 adaptation of the Ernest Hemingway novella, The Old Man and the Sea, starring Spencer Tracy.^[4]

The background footage is shot first and the actor or model is filmed carrying out their actions against a bluescreen. To simply place the foreground shot over the background shot would create a ghostly image over a blue-tinged background. The actor or model must be separated from the background and placed into a specially-made "hole" in the background footage. The bluescreen shot was first rephotographed through a blue filter so that only the background is exposed. A special film is used that creates a black and white negative image — a black background with a subject-shaped hole in the middle. This is called a 'female matte'. The bluescreen shot was then rephotographed again, this time through a red and green filter so that only the foreground image was cast on film, creating a black silhouette on an unexposed (clear) background. This is called a 'male matte'.

The background image is then rephotographed through the male matte, and the shot rephotographed through the female matte. An optical printer with two projectors, a film camera and a 'beam splitter' combines the images together one frame at a time. This part of the process must be very carefully controlled to ensure the absence of 'black lines'. During the 1980s, minicomputers were used to control the optical printer. For The Empire Strikes Back, Richard Edlund created a 'quad optical printer' that accelerated the process considerably and saved money. He received a special Academy Award for his innovation.

One drawback to the traditional traveling matte is that the cameras shooting the images to be composited can't be easily synchronized. For decades, such matte shots had to be done "locked-down" so that neither the matted subject nor the background could shift their camera perspective at all. Later, computer-timed motion control cameras alleviated this problem, as both the foreground and background could be filmed with the same camera moves.

Petro Vlahos was awarded an Academy Award for his development of these techniques. His technique exploits the fact that most objects in real-world scenes have a color whose blue color component is similar in intensity to their green color component. Zbigniew Rybczyński also contributed to bluescreen technology.

For Star Trek: The Next Generation, an ultraviolet light matting process was proposed by Don Lee of CIS and developed by Gary Hutzel and the staff of Image G. This involved a fluorescent orange backdrop which made it easier to generate a holdout matte, thus allowing the effects team to produce effects in a quarter of the time needed for other methods.^[5]

Some films make heavy use of chroma key to add backgrounds that are constructed entirely using computer-generated imagery (CGI). Performances from different takes can even be composited together, which allows actors to be filmed separately and then placed together in the same scene. Chroma key allows performers to appear to be in any location without even leaving the studio.

Computer development also made it easier to incorporate motion into composited shots, even when using handheld cameras. Reference-points can now be placed onto the colored background (usually as a painted grid, X's marked with tape, or equally spaced tennis balls attached to the wall). In post-production, a computer can use the references to adjust the position of the background, making it match the movement of the foreground perfectly. Modern advances in software and computational power have even eliminated the need to use grids or tracking marks – the software analyzes the relative motion of colored pixels against other colored pixels and solves the 'motion' to create a camera motion algorithm which can be used in compositing software to match the motion of composited elements to a moving background plate.

Weathermen often use a field monitor to the side of the screen to see where they are putting their hands. A newer technique is to project a faint image onto the screen.

Process[link]

Film set for The Spiderwick Chronicles, where a special effects scene using bluescreen chroma key is in preparation.

The principal subject is filmed or photographed against a background consisting of a single color or a relatively narrow range of colors, usually blue or green because these colors are considered to be the furthest away from skin tone.^[3] The portions of the video which match the preselected color are replaced by the alternate background video. This process is commonly known as "keying", "keying out" or simply a "key".

Green is currently used as a backdrop more than any other color because image sensors in digital video cameras are most sensitive to green, due to the bayer pattern allocating more pixels to the green channel, mimicking the human eye's increased sensitivity to green light.^[6] Therefore, the green camera channel contains the least "noise" and can produce the cleanest key/matte/mask. Additionally, less light is needed to illuminate green, again because of the higher sensitivity to green in image sensors.^[7] Bright green has also become favored as a blue background may match a subject's eye color or common items of clothing, such as jeans, or a dark-navy suit.

Blue was used before digital keying became commonplace because it was necessary for the optical process, but it needed more illumination than green. However, it is also further in the visual spectrum from red, the predominant color in human skin.

The most important factor for a key is the color separation of the foreground (the subject) and background (the screen) – a bluescreen will be used if the subject is predominately green (for example plants), despite the camera being more sensitive to green light.

In analog color TV, color is represented by the phase of the chroma subcarrier relative to a reference oscillator. Chroma key is achieved by comparing the phase of the video to the phase corresponding to the preselected color. In-phase portions of the video are replaced by the alternate background video.

In digital color TV, color is represented by three numbers (red, green, blue). Chroma key is achieved by a simple numerical comparison between the video and the preselected color. If the color at a particular point on the screen matches (either exactly, or in a range), then the video at that point is replaced by the alternate background video.

Clothing[link]

A chroma key subject must not wear clothing similar in color to the chroma key color(s) (unless intentional), because the clothing may be replaced with the background video. An example of intentional use of this is when an actor wears a blue covering over a part of his body to make it invisible in the final shot. This technique can be used to achieve an effect similar to that used in the Harry Potter films to create the effect of an invisibility cloak. The actor can also be filmed against a chroma key background and inserted into the background shot with a distortion effect in order to create a cloak that is marginally detectable.^[8]

Difficulties emerge with bluescreen when a costume in an effects shot must be blue, such as Superman's traditional blue outfit. In the 2002 film Spider-Man, in scenes where both Spider-Man and the Green Goblin are in the air, Spider-Man had to be shot in front of the greenscreen and the Green Goblin had to be shot in front of a bluescreen, because Spider-Man wears a costume which is red and blue in color and the goblin wears a costume which is entirely green in color. If both were shot in front of the same screen, one character would have been partially erased from the shot.

Background[link]

Demonstration of the creation of special effects techniques utilizing chroma key.

Blue is generally used for both weather maps and special effects because it is complementary to human skin tone. The use of blue is also tied to the fact that the blue emulsion layer of film has the finest crystals and thus good detail and minimal grain (in comparison to the red and green layers of the emulsion.) In the digital world, however, green has become the favored color because digital cameras retain more detail in the green channel and it requires less light than blue. Green not only has a higher luminance value than blue but also in early digital formats the green channel was sampled twice as often as the blue, making it easier to work with. The choice of color is up to the effects artists and the needs of the specific shot. In the past decade, the use of green has become dominant in film special effects. Also, the green background is favored over blue for outdoors filming where the blue sky might appear in the frame and could accidentally be replaced in the process. Although green and blue are the most common, any color can be used. Red is usually avoided due to its prevalence in normal human skin pigments, but can be often used for objects and scenes which do not involve people.

Occasionally, a magenta background is used, as in some software applications where the magenta or fuchsia key value #FF00FF is sometimes referred to as "magic pink".^[9]

With better imaging and hardware, many companies are avoiding the confusion often experienced by weather presenters, who must otherwise watch themselves on a monitor to see the image shown behind them, by lightly projecting a copy of the background image onto the blue/green screen. This allows the presenter to accurately point and look at the map without referring to monitors.

A newer technique is to use a retroreflective curtain in the background, along with a ring of bright LEDs around the camera lens. This requires no light to shine on the background other than the LEDs, which use an extremely small amount of power and space unlike big stage lights, and require no rigging. This advance was made possible by the invention of practical blue LEDs in the 1990s, which also allow for emerald green LEDs.

There is also a form of color keying that uses light spectrum invisible to human eye. Called Thermo-Key, it uses infrared as the key color, which would not be replaced by background image during postprocessing.^[10][11]

Even lighting[link]

The biggest challenge when setting up a bluescreen or greenscreen is even lighting and the avoidance of shadow, because it is best to have as narrow a color range as possible being replaced. A shadow would present itself as a darker color to the camera and might not register for replacement. This can sometimes be seen in low-budget or live broadcasts where the errors cannot be manually repaired. The material being used affects the quality and ease of having it evenly lit. Materials which are shiny will be far less successful than those that are not. A shiny surface will have areas that reflect the lights making them appear pale, while other areas may be darkened. A matte surface will diffuse the reflected light and have a more even color range. In order to get the cleanest key from shooting greenscreen it is necessary to create a value difference between the subject and the greenscreen. In order to differentiate the subject and screen a two-stop difference can be used, either by making the greenscreen two stops higher than the subject or vice versa.

Sometimes a shadow can be used to create a special effect. Areas of the bluescreen or greenscreen with a shadow on them can be replaced with a darker version of the desired background video image, making it look like the person casting the shadow is actually casting a shadow on the background image instead.

Another common, related, problem is lighting the subjects in a manner that is complementary to the scene being keyed in behind. Attention has to be paid to matching the direction of lighting and the color balance in the background. Any spill of the chroma key color will make the result look unnatural. Even a difference in the focal length of the lenses used can affect the success of chroma key.

Exposure[link]

Another challenge for bluescreen or greenscreen is proper camera exposure. Underexposing or overexposing a colored backdrop can lead to poor saturation levels. In the case of video and digital cinema cameras, underexposed images can contain high amounts of noise, as well. The background must be bright enough to allow the camera to create a bright and saturated image.

Programming[link]

There are several different quality- and speed-optimized techniques for implementing color keying in software.^[12]

In most versions, a function f(r, g, b) → α is applied to every pixel in the image. α (alpha) has a meaning similar to that in alpha compositing techniques. α ≤ 0 means the pixel is the green screen, α ≥ 1 means the pixel is in the foreground object. Values between 0 and 1 indicate a pixel that is partially covered by the foreground object. A usable green screen example, which matches how chroma key was done on an optical printer, is f(r, g, b) = K₀ * b − K₁ * g + K₂ (K_0..2 are user-adjustable constants, 1 is a good initial guess for all of them).

Often the software does screen spill removal from the colors as well as figure out the alpha. This may be a separate function g(r, g, b) → (r, g, b), a very simple green screen example is g(r, g, b) → (r, min(g, b), b). Or f is changed to return (r, g, b, a) all at once, this is useful if part of the calculation is shared.

Most keyers use far more complicated functions. A popular approach is to describe a closed 3D surface in RGB space and determine the signed distance the point (r, g, b) is from this surface, or to find the distance the point (r, g, b) is between two closed nested surfaces. It is also very common for f() to depend on more than just the current pixel's color, it may also use the (x, y) position, the values of nearby pixels, the value from reference images, and values from user-drawn masks.

A different class of algorithm tries to figure out a 2D path that separates the foreground from the background. This path can be the output, or the image can be drawn by filling the path with α = 1 as a final step. An example of such an algorithm is the use of active contour. Most research in recent years has been into these algorithms.

See also[link]

• Compositing
• Federal Standard 1037C
• Filmmaking
• Drew Carey's Green Screen Show
• Film production
• Front projection effect
• Live action
• Matte (filmmaking)
• Muslin
• Optical printer
• Primatte chromakey technology
• Rear projection effect
• Reverse bluescreen
• Schüfftan process
• Signal processing
• Sodium vapor process
• Special effects
• Video switcher
• Virtual set

References[link]

External links[link]

• Building a Chroma Key on a low budget (Google Knol)
• How Blue Screens Work
• Stargate Studios Virtual Backlot Reel 2012 -- An awesome demonstration of green screen work!!!