Memory-mapped file

Memory-mapped file

A memory-mapped file is a segment of virtual memory which has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource. This resource is typically a file that is physically present on-disk, but can also be a device, shared memory object, or other resource that the operating system can reference through a file descriptor. Once present, this correlation between the file and the memory space permits applications to treat the mapped portion as if it were primary memory.

Contents

Benefits

The primary benefit of memory mapping a file is increasing I/O performance, especially when used on large files. For small files, memory-mapped files can result in a waste of slack space[1] as memory maps are always aligned to the page size, which is mostly 4 KiB. Therefore a 5 KiB file will allocate 8 KiB and thus 3 KiB are wasted. Accessing memory mapped files is faster than using direct read and write operations for two reasons. Firstly, a system call is orders of magnitude slower than a simple change to a program's local memory. Secondly, in most operating systems the memory region mapped actually is the kernel's page cache (file cache), meaning that no copies need to be created in user space.

Certain application level memory-mapped file operations also perform better than their physical file counterparts. Applications can access and update data in the file directly and in-place, as opposed to seeking from the start of the file or rewriting the entire edited contents to a temporary location. Since the memory-mapped file is handled internally in pages, linear file access (as seen, for example, in flat file data storage or configuration files) requires disk access only when a new page boundary is crossed, and can write larger sections of the file to disk in a single operation.

A possible benefit of memory-mapped files is a "lazy loading", thus using small amounts of RAM even for a very large file. Trying to load the entire contents of a file that is significantly larger than the amount of memory available can cause severe thrashing as the operating system reads from disk into memory and simultaneously pages from memory back to disk. Memory-mapping may not only bypass the page file completely, but the system only needs to load the smaller page-sized sections as data is being edited, similarly to demand paging scheme used for programs.

The memory mapping process is handled by the virtual memory manager, which is the same subsystem responsible for dealing with the page file. Memory mapped files are loaded into memory one entire page at a time. The page size is selected by the operating system for maximum performance. Since page file management is one of the most critical elements of a virtual memory system, loading page sized sections of a file into physical memory is typically a very highly optimized system function.[2]

Drawbacks

The major reason to choose memory mapped file I/O is performance. Nevertheless there can be tradeoffs. The standard I/O approach is costly due to system call overhead and memory copying. The memory mapped approach has its cost in minor page faults - when a block of data is loaded in page cache, but is not yet mapped into the process's virtual memory space. In some circumstances, memory mapped file I/O can be substantially slower than standard file I/O.[3]

Another drawback of memory mapped files relates to a given architecture's address space: a file larger than the addressable space can have only portions mapped at a time, complicating reading it. For example, a 32-bit architecture such as Intel's IA-32 can only directly address 4 GiB or smaller portions of files. This drawback is avoided in the case of devices addressing memory when an IOMMU is present.

Common uses

Perhaps the most common use for a memory-mapped file is the process loader in most modern operating systems (including Microsoft Windows and Unix-like systems.) When a process is started, the operating system uses a memory mapped file to bring the executable file, along with any loadable modules, into memory for execution. Most memory-mapping systems use a technique called demand paging, where the file is loaded into physical memory in subsets (one page each), and only when that page is actually referenced.[4] In the specific case of executable files, this permits the OS to selectively load only those portions of a process image that actually need to execute.

Another common use for memory-mapped files is to share memory between multiple processes. In modern protected mode operating systems, processes are generally not permitted to access memory space that is allocated for use by another process. (A program's attempt to do so causes invalid page faults or segmentation violations.) There are a number of techniques available to safely share memory, and memory-mapped file I/O is one of the most popular. Two or more applications can simultaneously map a single physical file into memory and access this memory. For example, the Microsoft Windows operating system provides a mechanism for applications to memory-map a shared segment of the system's page file itself and share data via this section.

Platform support

Most modern operating systems or runtime environments support some form of memory-mapped file access. The function mmap()[5], which creates a mapping of a file given a file descriptor, starting location in the file, and a length, is part of the POSIX specification, so the wide variety of POSIX-compliant systems, such as UNIX, Linux, Mac OS X[6] or OpenVMS, support a common mechanism for memory mapping files. The Microsoft Windows operating systems also support a group of API functions for this purpose, such as CreateFileMapping().[7]

The Boost C++ Libraries provide a portable implementation of memory-mapped files for Microsoft Windows and POSIX-compliant platforms.[8]

The Java programming language provides classes and methods to access memory mapped files, such as FileChannel.

The D programming language supports memory mapped files in its standard library (std.mmfile module).[9]

Ruby has a gem (library) called Mmap, which implements memory-mapped file objects.

Since version 1.6, Python has included a mmap module in its Standard Library.[10] Details of the module vary according to whether the host platform is Windows or Unix-like.

For Perl there are a several modules available for memory mapping files on the CPAN, such as Sys::Mmap[11] and File::Map[12].

In the Microsoft .NET runtime, P/Invoke can be used to use memory mapped files directly through the Windows API. Native managed access to memory mapped files was introduced in version 4 of the runtime (see Memory-Mapped Files). For previous versions, there are third-party libraries which provide managed API's.[13]

PHP supported memory-mapping techniques in a number of native file access functions such as file_get_contents() but has removed this in 5.3 (see revision log).

References

  1. ^ http://www.devshed.com/c/a/BrainDump/Using-mmap-for-Advanced-File-IO/
  2. ^ http://msdn2.microsoft.com/en-us/library/ms810613.aspx, "What Do Memory-Mapped Files Have to Offer?".
  3. ^ http://lists.freebsd.org/pipermail/freebsd-questions/2004-June/050371.html, read vs. mmap (or io vs. page faults) by Matthew Dillon
  4. ^ http://www.linux-tutorial.info/modules.php?name=Tutorial&pageid=89, "Demand Paging"
  5. ^ Memory Mapped Files
  6. ^ Apple - Mac OS X Leopard - Technology - UNIX
  7. ^ CreateFileMapping Function (Windows)
  8. ^ http://www.boost.org/doc/libs/1_40_0/libs/iostreams/doc/classes/mapped_file.html
  9. ^ http://www.digitalmars.com/d/2.0/phobos/std_mmfile.html
  10. ^ ""New Modules in 1.6"". Archived from the original on 30 December 2006. http://web.archive.org/web/20061230034955/http://www.python.org/download/releases/1.6.1/#new-modules-in-1-6. Retrieved 23 December 2008. 
  11. ^ "Sys::Mmap Perl Module". http://search.cpan.org/perldoc?Sys::Mmap. 
  12. ^ "File::Map Perl Module". http://search.cpan.org/perldoc?File::Map. 
  13. ^ DotNet

External links


Wikimedia Foundation. 2010.

Игры ⚽ Поможем написать курсовую

Look at other dictionaries:

  • Memory-mapped I/O — For more generic meanings of input/output port, see Computer port (hardware). MMIO redirects here. For the airport serving Saltillo, Mexico, assigned the ICAO code MMIO, see Plan de Guadalupe International Airport. Memory mapped I/O (MMIO) and… …   Wikipedia

  • Memory mapping — In computing, memory mapping may refer to: Memory mapped file, also known as mmap() Memory mapped I/O, an alternative to port I/O; a communication between CPU and peripheral device using the same instructions, and same bus, as between CPU and… …   Wikipedia

  • Memory management unit — This 68451 MMU could be used with the Motorola 68010 A memory management unit (MMU), sometimes called paged memory management unit (PMMU), is a computer hardware component responsible for handling accesses to memory requested by the CPU. Its… …   Wikipedia

  • Conventional memory — Memory areas of the IBM PC family. In DOS memory management, conventional memory, also called base memory, is the first 640 kilobytes (640 × 1024 bytes) of the memory on IBM PC or compatible systems. It is the read write memory usable by the… …   Wikipedia

  • Memory address — A digital computer s memory, more specifically main memory, consists of many memory locations, each having a memory address, a number, analogous to a street address, at which computer programs store (except when the address is in mask ROM) and… …   Wikipedia

  • File system — For library and office filing systems, see Library classification. Further information: Filing cabinet A file system (or filesystem) is a means to organize data expected to be retained after a program terminates by providing procedures to store,… …   Wikipedia

  • Flash memory — Computer memory types Volatile RAM DRAM (e.g., DDR SDRAM) SRAM In development T RAM Z RAM TTRAM Historical Delay line memory Selectron tube Williams tube Non volatile …   Wikipedia

  • Virtual memory — This article is about the computational technique. For the TBN game show, see Virtual Memory (game show). Virtual memory combines active RAM and inactive memory in disk form into a large range of contiguous addresses. In computing, virtual memory …   Wikipedia

  • Write-only memory — is the antithesis of read only memory (ROM). By definition, a WOM is a memory device which can be written but never read. Since there seems to be no obvious utility for such a memory circuit, from which data cannot be retrieved, the concept is… …   Wikipedia

  • Common Log File System — (CLFS) is a general purpose logging subsystem that is accessible to both kernel mode as well as user mode applications for building high performance transaction logs. It was introduced with Windows Server 2003 R2 and included in later Windows OSs …   Wikipedia

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”