Media Foundation

Microsoft Media Foundation (MF) is a COM-based multimedia framework pipeline and infrastructure platform for digital media in Windows Vista, Windows 7 & Windows 8. It is the intended replacement for Microsoft DirectShow, Windows Media SDK, DirectX Media Objects (DMOs) and all other legacy multimedia APIs such as Audio Compression Manager (ACM) and Video for Windows (VfW). The existing DirectShow technology is intended to be replaced by Media Foundation step-by-step, starting with a few features. For some time there will be a co-existence of Media Foundation and DirectShow. Media Foundation will not be available for previous Windows versions, including Windows XP.

The first release, present in Windows Vista, focuses on audio and video playback quality, high-definition content (i.e. HDTV), content protection and a more unified approach for digital data access control for digital rights management (DRM) and its interoperability. It integrates DXVA 2.0 for offloading more of the video processing pipeline to hardware, for better performance. Videos are processed in the colorspace they were encoded in, and are handed off to the hardware, which composes the image in its native colorspace. This prevents intermediate colorspace conversions to improve performance. MF includes a new video renderer, called Enhanced Video Renderer (EVR), which is the next iteration of VMR 7 and 9. EVR has better support for playback timing and synchronization. It uses the Multimedia Class Scheduler Service (MMCSS), a new service that prioritizes real time multimedia processing, to reserve the resources required for the playback, without any tearing or glitches.

The second release included in Windows 7 introduces expanded media format support and DXVA HD for acceleration of HD content if WDDM 1.1 drivers are used.^[1]

Architecture

Media Foundation Architecture

The MF architecture is divided into the Control layer, Core Layer and the Platform layer. The core layer encapsulates most of the functionality of Media Foundation. It consists of the media foundation pipeline, which has three components: Media Source, Media Sink and Media Foundation Transforms (MFT). A media source is an object that acts as the source of multimedia data, either compressed or uncompressed. It can encapsulate various data sources, like a file, or a network server or even a camcorder, with source specific functionality abstracted by a common interface. A source object can use a source resolver object which creates a media source from an URI, file or bytestream. Support for non-standard protocols can be added by creating a source resolver for them. A source object can also use a sequencer object to use a sequence of sources (a playlist) or to coalesce multiple sources into single logical source. A media sink is the recipient of processed multimedia data. A media sink can either be a renderer sink, which renders the content on an output device, or an archive sink, which saves the content onto a persistent storage system such as a file. A renderer sink takes uncompressed data as input whereas an archive sink can take either compressed or uncompressed data, depending on the output type. The data from media sources to sinks are acted upon by MFTs; MFTs are certain functions which transform the data into another form. MFTs can include multiplexers and demultiplexers, codecs or DSP effects like reverb. The core layer uses services like file access and networking and clock synchronization to time the multimedia rendering. These are part of the Platform layer, which provides services necessary for accessing the source and sink byte streams, presentation clocks and an object model that lets the core layer components function asynchronously, and is generally implemented as OS services. Pausing, stopping, fast forward, reverse or time-compression can be achieved by controlling the presentation clock.

However, the media pipeline components are not connected; rather they are just presented as discrete components. An application running in the Control layer has to choose which source types, transforms and sinks are needed for the particular video processing task at hand, and set up the "connections" between the components (a topology) to complete the data flow pipeline. For example, to play back a compressed audio/video file, the pipeline will consist of a file source object, a demultiplexer for the specific file container format to split the audio and video streams, codecs to decompress the audio and video streams, DSP processors for audio and video effects and finally the EVR renderer, in sequence. Or for a video capture application, the camcorder will act as video and audio sources, on which codec MFTs will work to compress the data and feed to a multiplexer that coalesces the streams into a container; and finally a file sink or a network sink will write it to a file or stream over a network. The application also has to co-ordinate the flow of data between the pipeline components. The control layer has to "pull" (request) samples from one pipeline component and pass it onto the next component in order to achieve data flow within the pipeline. This is in contrast to DirectShow's "push" model where a pipeline component pushes data to the next component. Media Foundation allows content protection by hosting the pipeline within a protected execution environment, called the Protected Media Path. The control layer components are required to propagate the data through the pipeline at a rate that the rendering synchronizes with the presentation clock. The rate (or time) of rendering is embedded as a part of the multimedia stream as metadata. The source objects extract the metadata and pass it over. Metadata is of two types: coded metadata, which is information about bit rate and presentation timings, and descriptive metadata, like title and author names. Coded metadata is handed over to the object that controls the pipeline session, and descriptive metadata is exposed for the application to use if it chooses to.

Media Foundation provides a Media Session object that can be used to setup the topologies, and facilitate a data flow, without the application doing it explicitly. It exists in the control layer, and exposes a Topology loader object. The application specifies the required pipeline topology to the loader, which then creates the necessary connections between the components. The media session object manages the job of synchronizing with the presentation clock. It creates the presentation clock object, and passes a reference to it to the sink. It then uses the timer events from the clock to propagate data along the pipeline. It also changes the state of the clock to handle pause, stop or resume requests from the application.

Media Foundation Transform

Media Foundation Transforms (MFTs) represent a generic model for processing media data. They are used in Media Foundation primarily to implement decoders, encoders, mixers and digital signal processors (DSPs) – between media sources and media sinks. Media Foundation Transforms are an evolution of the transform model first introduced with DirectX Media Objects (DMOs). Their behaviors are more clearly specified. Hybrid DMO/MFT Objects can also be created. Applications can use MFTs inside the Media Foundation pipeline, or use them directly as stand-alone objects. MFTs can be any of the following type:

• Audio and video codecs
• Audio and video effects
• Multiplexers and demultiplexers
• Tees
• Color-space converters
• Sample-rate converters
• Video scalers

Microsoft recommends developers to write a Media Foundation Transform instead of a DirectShow filter, for Windows Vista, Windows 7 & Windows 8.^[2] For video editing and video capture, Microsoft recommends using DirectShow as they are not the primary focus of Media Foundation in Windows Vista. Starting with Windows 7, MFTs also support hardware-accelerated video processing, encoding and decoding for AVStream-based media devices. ^[3]

Enhanced Video Renderer

Media Foundation uses the Enhanced Video Renderer (EVR) for rendering video content, which acts as a mixer as well. It can mix up to 16 simultaneous streams, with the first stream being a reference stream. All but the reference stream can have per-pixel transparency information, as well as any specified z-order. The reference stream cannot have transparent pixels, and has a fixed z-order position, at the back of all streams. The final image is composited onto a single surface by coloring each pixel according to the color and transparency of the corresponding pixel in all streams.

Internally, the EVR uses a mixer object for mixing the streams. It can also deinterlace the output and apply color correction, if required. The composited frame is handed off to a presenter object, which schedules them for rendering onto a Direct3D device, which it shares with the DWM and other applications using the device. The frame rate of the output video is synchronized with the frame rate of the reference stream. If any of the other streams (called substreams) have a different frame rate, EVR discards the extra frames (if the substream has a higher frame rate), or uses the same frame more than once (if it has a lower frame rate).

Supported media formats

Windows Media Audio and Windows Media Video are the only default supported formats for encoding through Media Foundation in Windows Vista. For decoding, an MP3 file source is available in Windows Vista to read MP3 streams but an MP3 file sink to output MP3 is only available in Windows 7^[4]. Format support is extensible however; developers can add support for other formats by writing encoder/decoder MFTs and/or custom media sources/media sinks.

Windows 7 expands upon the codec support available in Windows Vista. It includes AVI, WAV, AAC/ADTS file sources to read the respective formats ^[4], an MPEG-4 file source to read MP4, M4A, M4V, MP4V, MOV and 3GP container formats ^[5] and an MPEG-4 file sink to output to MP4 format ^[6]. Windows 7 also includes a media source to read MPEG transport stream/BDAV MPEG-2 transport stream (M2TS, MTS, M2T and AVCHD) files.^{[citation needed]}

Similar to Windows Vista, transcoding (encoding) support is not exposed through any built-in Windows application but several codecs are included as Media Foundation Transforms (MFTs).^[4] In addition to Windows Media Audio and Windows Media Video encoders and decoders, and ASF file sink and file source introduced in Windows Vista ^[4], Windows 7 includes an H.264 encoder with Baseline profile level 3 and Main profile support ^[7] and an AAC Low Complexity (AAC-LC) profile encoder ^[8]

For playback of various media formats, Windows 7 also introduces an H.264 decoder with Baseline, Main, and High profile support, up to level 5.1 ^[9], AAC-LC and HE-AAC v1 (SBR) multichannel, HE-AAC v2 (PS) stereo decoders ^[10], MPEG-4 Part 2 Simple Profile and Advanced Simple Profile decoders ^[11] which includes decoding popular codec implementations such as DivX, Xvid and Nero Digital as well as MJPEG ^[4] and DV ^[12] MFT decoders for AVI. Windows Media Player 12 uses the built-in Media Foundation codecs to play these formats by default.

MIDI playback is also not yet supported using Media Foundation.

Benefits over DirectShow

Media Foundation offers the following benefits:

• Is scalable for high-definition content and DRM-protected content.
• Allows DirectX Video Acceleration to be used outside of the DirectShow infrastructure. DXVA 2.0 is available to user-mode components without using the DirectShow video renderer.
• Provides better resilience to CPU, I/O, and memory stress for low-latency glitch-free playback of audio and video. Video tearing has been minimized. The improved video processing support also enables high color spaces and enhanced full-screen playback. Enhanced video renderer (EVR) which is also available for DirectShow, offers better timing support and improved video processing.
• Media Foundation extensibility enables different content protection systems to operate together.
• Media Foundation uses the Multimedia Class Scheduler Service (MMCSS), a new system service in Windows Vista, Windows 7 & Windows 8. MMCSS enables multimedia applications to ensure that their time-sensitive processing receives prioritized access to CPU resources.

Application support

Media Foundation, for this initial release in Windows Vista and later release in Windows 7 & Windows 8, finds use in media playback applications. Until now, mainly internal or bundled Windows services and applications are using Media Foundation.

• Windows Media Player 11 in Windows Vista relies on Media Foundation for playing ASF (WMA and WMV) content and protected content, but can also use DirectShow or the Windows Media Format SDK instead. In the case of WMV9 playback, this also implies using DXVA 2.0 instead of DXVA 1.0 when the video hardware supports WMV9/VC-1 decoding acceleration.
• Windows Media Player 12 in Windows 7.
• Windows Media Player in Windows 8.
• Windows Media Center in Windows Vista, Windows 7, Windows 8 and later.
• Any application that uses Windows Protected Media Path (PMP), relies completely on Media Foundation.
• The updated Home Cinema fork of Media Player Classic supports EVR.
• GoldWave 5.60 and later relies on Media Foundation for importing and exporting audio.

References

1. ^ DXVA-HD
2. ^ Migrating from DirectShow to Media Foundation and comparison of the two
3. ^ Getting Started with Hardware Codec Support in AVStream
4. ^ ^{a b c d e} Supported Media Formats in Media Foundation
5. ^ MPEG-4 File Source
6. ^ MPEG-4 File Sink
7. ^ H.264 Video Encoder
8. ^ AAC Encoder
9. ^ H.264 Video Decoder
10. ^ AAC Decoder
11. ^ MPEG4 Part 2 Video Decoder
12. ^ DV Video Decoder

External links

• Microsoft Media Foundation SDK
• Media Foundation Development Forum
• Media Foundation Team Blog (with samples)
• Media Source Metadata
• Media Foundation Pipeline
• Media Foundation Architecture
• About the Media Session
• About the Media Foundation SDK
• Enhanced Video Renderer