Audio timescale-pitch modification

Time stretching is the process of changing the speed or duration of an audio signal without affecting its pitch.Pitch scaling or pitch shifting is the reverse: the process of changing the pitch without affecting the speed. There are also more advanced methods used to change speed, pitch, or both at once, as a function of time.

These processes are used, for instance, to match the pitches and tempos of two pre-recorded clips for mixing when the clips cannot be reperformed or resampled. (A drum track could be moderately resampled for tempo without adverse effects, but a pitched track could not). They are also used to create effects such as increasing the range of an instrument (like pitch shifting a guitar down an octave).

Resampling

The simplest way to change the duration or pitch of a digital audio clip is to resample it. This is a mathematical operation that effectively rebuilds a continuous waveform from its samples and then samples that waveform again at a different rate. When the new samples are played at the original sampling frequency, the audio clip sounds faster or slower. Unfortunately, the frequencies in the sample are always scaled at the same rate as the speed. In other words, slowing down the recording lowers the pitch, speeding it up raises the pitch, and the two effects cannot be separated. This is analogous to speeding up or slowing down an analog recording, like a phonograph record or tape, creating the chipmunk effect.

Phase vocoder

One way of stretching the length of a signal without affecting the pitch is to build a phase vocoder after Flanagan, Golden, and Portnoff.

Basic steps:
#compute the instantaneous frequency/amplitude relationship of the signal using the STFT, which is the discrete Fourier transform of a short, overlapping and smoothly windowed block of samples;
#apply some processing to the Fourier transform magnitudes and phases (like resampling the FFT blocks); and
#perform an inverse STFT by taking the inverse Fourier transform on each chunk and adding the resulting waveform chunks.

The phase vocoder handles sinusoid components well, but early implementations introduced considerable smearing on transient ("beat") waveforms at all non-integer compression/expansion rates, which renders the results phasey and diffuse. Recent improvements allow better quality results at all compression/expansion ratios but a residual smearing effect still remains.

The phase vocoder technique can also be used to perform pitch shifting, chorusing, timbre manipulation, harmonizing, and other unusual modifications, all of which can be changed as a function of time.

Time domain

Rabiner and Schafer in 1978 put forth an alternate solution that works in the time domain: attempt to find the period (or equivalently the fundamental frequency) of a given section of the wave using some pitch detection algorithm (commonly the peak of the signal's autocorrelation, or sometimes cepstral processing), and crossfade one period into another.This is called time domain harmonic scaling or the synchronized overlap-add method and performs somewhat faster than the phase vocoder on slower machines but fails when the autocorrelation mis-estimates the period of a signal with complicated harmonics (such as orchestral pieces).
Adobe Audition (formerly Cool Edit Pro) seems to solve this by looking for the period closest to a center period that the user specifies, which should be an integer multiple of the tempo, and between 30 Hz and the lowest bass frequency. For a 120 bpm tune, use 48 Hz because 48 Hz = 2,880 cycles/minute = 24 cycles/beat * 120 bpm.Fact|date=February 2007

This is much more limited in scope than the phase vocoder based processing, but can be made much less processor intensive, for real-time applications. It provides the most coherent results for single-pitched sounds like voice or musically monophonic instrument recordings.

High-end commercial audio processing packages either combine the two techniques (for example by separating the signal into sinusoid and transient waveforms), or use other techniques based on the wavelet transform, or artificial neural network processing, producing the highest-quality time stretching.

Sinusoidal/Spectral Modeling

Another alternative method for time stretching relies on a spectral model of the signal. In this method, peaks are identified in frames the STFT of the signal, and sinusoidal "tracks" are created by connecting peaks in adjacent frames. The tracks are then re-synthesized at a new time scale. This method can yield good results on both polyphonic and percussive material, especially when the signal is separated into sub-bands. However, this method is more computationally demanding than other methods.

Speed reading

Time stretching can be used with audio books and recorded lectures.Slowing down may improve comprehension of foreign languages [http://www.enounce.com/whatistsm.shtml] .

While one might expect speeding up to reduce comprehension,Herb Friedman says that "Experiments have shown that the brain works most efficiently if the information rate through the ears--via speech--is the "average" reading rate, which is about 200-300 wpm (words per minute), yet the average rate of speech is in the neighborhood of 100-150 wpm." [http://www.atarimagazines.com/creative/v9n7/122_Variable_speech.php ]

Speeding up audio is seen as the equivalent of "speed reading" [http://www.nevsblog.com/2006/06/23/listen-to-podcasts-in-half-the-time/ ] [http://cid.lib.byu.edu/?p=128 ] .

Other

Time stretching is often used to adjust Radio commercials [http://www.tvtechnology.com/features/audio_notes/f_audionotes.shtml] and the audio of Television advertisements [http://www.atarimagazines.com/creative/v9n7/122_Variable_speech.php] to fit exactly into the 30 or 60 seconds available.(A telecine pulldown pattern adjusts the video).

Pitch scaling

These techniques can also be used to transpose an audio sample while holding speed or duration constant. This may be accomplished by time stretching and then resampling back to the original length. Alternatively, the frequency of the sinusoids in a sinusoidal model may be altered directly, and the signal reconstructed at the appropriate time scale.

Transposing can be called pitch scaling or pitch shifting, depending on perspective.

For example, one could move the frequency of every note up by a perfect fifth, keeping the tempo the same.One can view this transposition as "pitch shifting", "shifting" each note up 7 keys on a piano keyboard, or adding a fixed amount on the Mel scale, or adding a fixed amount in linear pitch space.One can view the same transposition as "pitch scaling", "scaling" (multiplying) the frequency of every note by 3/2.

Musical transposition preserve the ratios of the harmonic frequencies that determine the sound's timbre, unlike the "frequency shift" performed by amplitude modulation, which adds a fixed frequency offset to the frequency of every note. (In theory one could perform a literal "pitch scaling" in which the musical pitch space location is scaled [a higher note would be shifted at a greater interval in linear pitch space than a lower note] , but that is highly unusual, and not musical).

Time domain processing works much better here, as smearing is less noticeable, but scaling vocal samples distorts the formants into a sort of Alvin and the Chipmunks-like effect, which may be desirable or undesirable.A process that preserves the formants and character of a voice involves analyzing the signal with a channel vocoder or LPC vocoder plus any of several pitch detection algorithms and then resynthesizing it at a different fundamental frequency.

ee also

* Audio signal processing
* Pitch control
* Sound effects

External links

* [http://sourceforge.net/projects/mffmtimescale/ MFFM Time Scale Modification for Audio] Implementation of the WSOLA algorithm for time scale modification of audio without artifact introduction
* [http://www.lownorth.nl/software/products/TimeToy.html Timetoy (Mac OS X)] A fun, easy-to-use timestretcher for Mac OS X by LowNorth
* [http://www.dspdimension.com/admin/time-pitch-overview/ Time Stretching and Pitch Shifting Overview] A comprehensive overview of current time and pitch modification techniques by Stephan Bernsee (Quite old, 1999)
* [http://www.dspdimension.com/admin/pitch-shifting-using-the-ft/ Stephan Bernsee's smbPitchShift C source code] C source code for doing frequency domain pitch manipulation
* [http://www.panix.com/~jens/pvoc-dolson.par The Phase Vocoder: A Tutorial] - A good description of the phase vocoder
* [http://www.ee.columbia.edu/~dpwe/papers/LaroD99-pvoc.pdf New Phase-Vocoder Techniques for Pitch-Shifting, Harmonizing and Other Exotic Effects]
* [http://www.ircam.fr/equipes/analyse-synthese/roebel/paper/dafx2003.pdf A new Approach to Transient Processing in the Phase Vocoder]
* [http://www.fon.hum.uva.nl/praat/manual/PSOLA.html PSOLA Synthesis] , [http://www.ee.columbia.edu/~dpwe/papers/HejMus91-solafs.pdf SOLAFS Synthesis] - Two specific methods of time domain TDHS or SOLA processing.
* [http://www.aes.org/ Audio Engineering Society]
*Original E2 article (http://everything2.com/index.pl?node_id=1074923)
* [http://www.dspdimension.com/data/html/dirac.html DSPdimension: DIRAC library]
* [http://www.time-stretching.com zplane.development: élastique SDKs]
*http://www.bdti.com/faq/dsp_faq.htm - comp.dsp FAQ
* [http://www.surina.net/soundtouch SoundTouch library] - An open-source implementation of time/pitch scaling algorithms. SoundStretch came from here. Used in the cross-platform Audacity editor.
* [http://keizai.yokkaichi-u.ac.jp/~ikeda/research/picola.html PICOLA and TDHS]
* [http://sourceforge.net/projects/wavmasher wavMasher] - Time and pitch scaling software
* [http://hypermammut.sourceforge.net/paulstretch/ PaulStretch] A program that works for extreme time stretching (like 50x), only
* [http://substance-night.it/4BPS/pitchshift.html 4 Band Shifter] An open source VST plugin based on Bernsee's code that shifts the pitch on 4 independent, user-definable frequency bands.
* [http://sourceforge.net/projects/sbsms/ sbsms] An open source sub-band-sinusoidal modeling library/tool for time-stretching/pitch-shifting