Sparse file

In computer science, a sparse file is a type of computer file that attempts to use file system space more efficiently when blocks allocated to the file are mostly empty. This is achieved by writing brief information (metadata) "representing" the empty blocks to disk instead of the the actual "empty" space which makes up the block, using less disk space. The full block size is written to disk at the actual size only when the block contains "real" (non-empty) data.

When reading sparse files, the file system transparently converts metadata representing empty blocks into "real" blocks filled with zero bytes at runtime. The application is unaware of this conversion.

Most modern file systems support sparse files, including most Unix variants and NTFS, but notably not Apple's HFS+. Sparse files are commonly used for disk images, database snapshots, log files and in scientific applications.

Creating sparse files in Unix

If executed manually, the Unix command: dd if=/dev/zero of=sparse-file bs=1 count=0 seek=5MWill create a file of five megabyte in size, but with no data stored on disk (only metadata).

Detecting sparse files in Unix

Sparse files have different apparent and actual file sizes. This can be detected by comparing the output of: du -s -B1 --apparent-size sparse-fileand: du -s -B1 sparse-file

Advantages

The advantage of sparse files is that storage is only allocated when actually needed: disk space is saved, and large files can be created even if there is insufficient free space on the file system.

Disadvantages

Disadvantages are that sparse files may become fragmented; file system free space reports may be misleading; filling up file systems containing sparse files can have unexpected effects; and copying a sparse file with a program that does not explicitly support them may copy the entire, uncompressed size of the file, including the sparse, mostly zero sections which are not on disk—losing the benefits of the sparse property in the file.

ee also

* Comparison of file systems
* Sparse matrix

References

* Giampaolo, Dominic (1999). " [http://www.nobius.org/~dbg/practical-file-system-design.pdf Practical File System Design with the Be File System] ". Morgan Kaufmann. ISBN 1-55860-497-9

External links

* [http://www.flexhex.com/docs/articles/sparse-files.phtml NTFS Sparse Files For Programmers]
* [http://technet2.microsoft.com/windowsserver/en/library/33dafa5a-5921-48ae-aff3-7cdc2d74cd201033.mspx?mfr=true Creating sparse files in Windows XP Professional using fsutil]
* [http://docs.sun.com/app/docs/doc/816-5166/6mbb1kq8r?a=view Creating sparse files in Solaris using mkfile]
* [http://msdn2.microsoft.com/en-us/library/ms175823.aspx Understanding Sparse File Sizes in Database Snapshots]
* [http://lwn.net/Articles/260795/ Detecting holes in sparse files]