Cut detection

Cut detection is a field of research of computer science. Its subject is the automated detection of "cuts" in digital video.

Use

Cut detection is used to split up a film into basic scenes. Therefore, it is of great use in software for post-production of videos. It is also a fundamental part of automated indexing of huge video archives, where from each scene one representative picture is chosen to create a visual overview of the whole film. By processing such indexes a search engine can process search items like "show me all films where there's a scene with a lion in it." Generally speaking, cut detection can do nothing that a human editor couldn't do manually, but it saves a lot of time.

Basic technical terms

In simple terms cut detection is about finding the positions in a video in that one scene is replaced by another one with different visual content. Technically speaking the following terms are used:

A digital video consists of frames that are presented to the viewer's eye in rapid succession to create the impression of movement. "Digital" in this context means both that a single frame consists of pixels and the data is present as binary data, such that it can be processed with a computer. Each frame within a digital video can be uniquely identified by its frame index, a serial number.

A shot is a sequence of frames shot uninterruptedly by one camera. A cut is the blending, in some way, of one shot into another one. Cut detection distinguishes hard cuts from soft cuts. While a hard cut is a sudden transition from one shot to another, i. e. one frame belongs to the first shot, the next frame belongs to the second shot, a soft cut is a gradual transition between two shots, i. e. a sequence of frames that belongs to both, the first "and" the second shot. "Detecting a cut" means that the position of a cut is gained; more precisely a hard cut is gained as "hard cut between frame i and frame i+1", a soft cut as "soft cut from frame i to frame j".

A cut that is detected correctly is called a hit, a cut that is there but was not detected is called a missed hit and a position in that the software assumes a cut, but where actually no cut is present, is called a false hit.

"An introduction to film editing and an exhaustive list of shot transition techniques can be found at film editing."

Vastness of the problem

Although cut detection appears to be a simple task for a human being, it is a non-trivial task for computers. Cut detection would be a trivial problem if each frame of a video was enriched with additional information about "when" and "by which camera" it was taken. Possibly no algorithm for cut detection will ever be able to detect all cuts with certainty, unless it is provided with powerful artificial intelligence.

While most algorithms achieve good results with hard cuts, many fail with recognizing soft cuts. Hard cuts usually go together with sudden and extensive changes in the visual content while soft cuts feature slow and gradual changes. A human being can compensate this lack of visual diversity with understanding the meaning of a scene. While a computer assumes a black line wiping a shot away to be "just another regular object moving slowly through the on-going scene", a person understands that the scene ends and is replaced by a black screen.

Methods

Each method for cut detection works on a two-phase-principle:
# Scoring. Each pair of consecutive frames of a digital video is given a certain score that represents the probability that between these two frames a cut is located.
# Thresholding. All pairs of frames are filtered with a certain threshold value: Each pair with a score higher than the threshold is considered a cut.

This principle is error prone. First, because even minor exceedings of the threshold value produce a hit, it must be ensured that phase one scatters values widely to maximize the average difference between the score for "cut" and "no cut". Second, the threshold must be chosen with care; usually useful values can be gained with statistical methods.

Scoring

Developing algorithms that produce valuable scorings is no easy task. Some of the most common are:
* Sum of absolute differences (SAD). This is both the most obvious and most simple algorithm of all: The two consecutive frames are compared pixel by pixel, summing up the absolute values of the differences of each two corresponding pixels. The result is a positive number that is used as the score. SAD reacts very sensitively to even minor changes within a scene: fast movements of the camera, explosions or the simple switching on of a light in a previously dark scene result in false hits. On the other hand, SAD hardly reacts to soft cuts at all. Yet, SAD is used often to produce a basic set of "possible hits" as it detects all visible hard cuts with utmost probability.
* Histogram differences (HD). Histogram differences is very similar to Sum of absolute differences. The difference is that HD computes the difference between the histograms of two consecutive frames; a histogram is a table that contains for each color within a frame the number of pixels that are shaded in that color. HD is not as sensitive to minor changes within a scene as SAD and thus produces less false hits. One major problem of HD is that two images can have exactly the same histograms while the shown content differs extremely, e. g. a picture of the sea and a beach can have the same histogram as one of a corn field and the sky. HD offers no guarantee that it recognizes hard cuts.
* Edge change ratio (ECR). The ECR attempts to compare the actual content of two frames. It transforms both frames to "edge pictures", i. e. it extracts the probable outlines of objects within the pictures (see edge detection for details). Afterwards it compares these edge pictures using dilatation to compute a probability that the second frame contains the same objects as the first frame. The ECR is one of the best performing algorithms for scoring. It reacts very sensitively to hard cuts and can detect many soft cuts by nature. In its basic form even ECR cannot detect soft cuts such as wipes as it considers the fading-in objects as regular objects moving through the scene. Yet, ECR can be extended manually to recognize special forms of soft cuts.

Finally, a combination of two or more of these algorithms can improve the performance.

Cost

All of the above algorithms complete in O(n) — that is to say they run in linear time — where "n" is the number of frames in the input video. The algorithms differ in a constant factor that is determined mostly by the image resolution of the video.

Measures for quality

Usually the following three measures are used to measure the quality of a cut detection algorithm:
* Precision is the probability, that an assumed cut in fact is cut:
$P\; =\; \{C\; over\; C\; +\; F\}$

* Recall is the probability that an existing cut will be detected:
$V\; =\; \{C\; over\; C\; +\; M\}$

* F1 is a combined measure that results in high value if, and only if, both precision "and" recall result in high values:
$F1=\{2\; *\; P\; *\; V\; over\; P\; +\; V\}$
The symbols stand for: C, the number of correctly detected cuts ("correct hits"), M, the number of not detected cuts ("missed hits") and F, the number of falsely detected cuts ("false hits"). All of these measures are mathematical measures, i. e. they deliver values in between 0 and 1. The basic rule is: the higher the value, the better performs the algorithm.

Sample implementations

A sample tool for Mac OS X that utilizes thresholding for shot detection is available from http://www.scene-detector.com