tar (file format)

tar

GNU tar 1.23 showing three common types of Tarballs (shown in red).
Filename extension	.tar
Internet media type	application/x-tar
Uniform Type Identifier	public.tar-archive org.gnu.gnu-tar-archive
Magic number	u s t a r \0 0 0 at byte 257 ("ustar" followed by a null byte followed by two digits '0', 8 bytes in total)
Type of format	File archiver
Contained by	gzip, bzip2, lzip, lzma, xz, lzop, compress
Open format?	Yes

In computing, tar (derived from tape archive) is both a file format (in the form of a type of archive bitstream) and the name of a program used to handle such files. The format was created in the early days of Unix and standardized by POSIX.1-1988 and later POSIX.1-2001.

Initially developed to be written directly to sequential I/O devices for tape backup purposes, it is now commonly used to collect many files into one larger file for distribution or archiving, while preserving file system information such as user and group permissions, dates, and directory structures.

Compression and naming

"TGZ" redirects here. For other uses, see TGZ (disambiguation).

Conventionally, uncompressed tar archive files have names ending in ".tar". Unlike ZIP archives, tar files (somefile.tar) are commonly compressed as a whole rather than piecemeal. Applying a compression utility such as gzip, bzip2, xz, lzip, lzma, or compress to a tar file produces a compressed tar file, typically named with an extension indicating the type of compression (e.g., somefile.tar.gz).

Popular tar programs like the BSD and GNU versions of tar support the command line options -z (gzip), and -j (bzip2) to automatically compress or decompress the archive file it is currently working with. GNU tar from version 1.20 onwards also supports --lzma (LZMA). 1.21 also supports lzop via --lzop, 1.22 adds support for xz via --xz or -J, and 1.23 adds support for lzip via --lzip.

Tar archiving is often used together with a compression method, such as gzip, to create a compressed archive. As shown, the combination of the files in the archive are compressed as one unit.

MS-DOS's 8.3 filename limitations, and a desire for brevity, resulted in additional popular conventions for naming compressed tar archives, though this practice has declined with FAT offering long filenames.

• .tgz is equivalent to .tar.gz
• .tbz and .tb2 are equivalent to .tar.bz2
• .taz is equivalent to .tar.Z
• .tlz is equivalent to .tar.lz
• .txz is equivalent to .tar.xz

A tar file or compressed tar file is commonly referred to as a tarball.^[1]

Format details

A tar file is the concatenation of one or more files. Each file is preceded by a 512-byte header record. The file data is written unaltered except that its length is rounded up to a multiple of 512 bytes and the extra space is zero filled. The end of an archive is marked by at least two consecutive zero-filled records. (The origin of tar's record size appears to be the 512-byte disk sectors used in the Version 7 Unix file system.)

Many historic tape drives read and write variable-length data blocks, leaving significant wasted space on the tape between blocks (for the tape to physically start and stop moving). Some tape drives (and raw disks) only support fixed-length data blocks. Also, when writing to any medium such as a filesystem or network, it takes less time to write one large block than many small blocks. Therefore, the tar command writes data in blocks of many 512 byte records. The user can specify a blocking factor, which is the number of records per block; the default is 20, producing 10 kilobyte blocks (which was large when UNIX was invented, but now seems rather small). The final block of an archive is padded out to full length with zero bytes.

File header

The file header block contains metadata about a file. To ensure portability across different architectures with different byte orderings, the information in the header block is encoded in ASCII. Thus if all the files in an archive are text files, and have ASCII names, then the archive is essentially an ASCII text file (containing many NUL characters).

Subsequent extensions have diluted this original attribute (which was never particularly valuable, due to the presence of NULs, which are often poorly handled by text-manipulation programs).

The fields defined by the original Unix tar format are listed in the table below. The link indicator/file type table includes some modern extensions. When a field is unused it is filled with NUL bytes. The header is padded with NUL bytes to make it fill a 512 byte block.

Field Offset	Field Size	Field
0	100	File name
100	8	File mode
108	8	Owner's numeric user ID
116	8	Group's numeric user ID
124	12	File size in bytes
136	12	Last modification time in numeric Unix time format
148	8	Checksum for header block
156	1	Link indicator (file type)
157	100	Name of linked file

The Link indicator field can have the following values:

Link indicator field

Value	Meaning
'0'	Normal file
(ASCII NUL)	Normal file (now obsolete)
'1'	Hard link
'2'	Symbolic link
'3'	Character special
'4'	Block special
'5'	Directory
'6'	FIFO
'7'	Contiguous file

A directory is also indicated by having a trailing slash (/) in the name.

Numeric values are encoded in octal numbers using ASCII digits, with leading zeroes. For historical reasons, a final NUL or space character should be used. Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files. To overcome this limitation, star in 2001 introduced a base-256 coding that is indicated by setting the high-order bit of the leftmost byte of a numeric field. GNU-tar and BSD-tar followed this idea. Additionally, versions of tar from before the first POSIX standard from 1988 pad the values with spaces instead of zeroes.

The checksum is calculated by taking the sum of the unsigned byte values of the header block with the eight checksum bytes taken to be ascii spaces (decimal value 32). It is stored as a six digit octal number with leading zeroes followed by a NUL and then a space. Various implementations do not adhere to this, so relying on the first white space trimmed six digits for checksum yields better compatibility. In addition, some historic tar implementations treated bytes as signed. Users typically calculate the checksum both ways, and treat it as good if either the signed or unsigned sum matches the included checksum.

Unix filesystems support multiple links (names) for the same file. If several such files appear in a tar archive, only the first one is archived as a normal file; the rest are archived as hard links, with the "name of linked file" field set to the first one's name. On extraction, such hard links should be recreated in the file system.

UStar format

Most modern tar programs read and write archives in the UStar (Uniform Standard Tape Archive) format, introduced by the POSIX IEEE P1003.1 standard from 1988. It introduced additional header fields. Older tar programs will ignore the extra information, while newer programs will test for the presence of the "ustar" string to determine if the new format is in use. The UStar format allows for longer file names and stores extra information about each file.

Field Offset	Field Size	Field
0	156	(as in old format)
156	1	Type flag
157	100	(as in old format)
257	6	UStar indicator "ustar"
263	2	UStar version "00"
265	32	Owner user name
297	32	Owner group name
329	8	Device major number
337	8	Device minor number
345	155	Filename prefix

Uses

Tarpipe

Many old tar implementations (such as GNU tar) do not record extended attributes (xattrs) or ACLs.

In 2001, star introduced support for ACLs and extended attributes. Later, major Linux distributions created their own patched versions of GNU tar that fully support ACL.

A tarpipe is the process of creating a tar archive on stdout and then, in another directory, extracting the tar file from the piped stdin. This is one way to copy directories and subdirectories, even if the directories contain special files, such as symlinks, and character or block devices. Example Tarpipe:

 tar -c "$srcdir" | tar -C "$destdir" -xv

Tarpipes are inefficient because of the overhead of the second tar invocation (which is especially expensive on DOS/Windows when using command.com, as it is buffered to a file. But that isn't common. Since cmd.exe was introduced, parallel pipe are supported). Also, pipes may cause problems in automated scripts, as the success or failure of each command cannot be verified easily.

Software distribution

The tar format continues to be used extensively in Linux and Windows, as it is part of the GNU project and comes as part of every distribution of GNU/Linux. Linux versions use features prominently in various software distributions, with most software source code made available in .tar.gz (compressed tar archives), and its use featuring as a basis of other types of containment and distribution, such as in Debian's .DEB archives, which are also used in Ubuntu.

Problems and limitations

The original tar format was created in the early days of UNIX, and despite current widespread use, many of its design features are considered dated.^[2]

In 1997, a method for adding unlimited extensions to the tar format has been proposed by Sun and later accepted for the POSIX.1-2001 standard. This format is known as extended tar-format or pax-format. The new tar format allows to add any type of vendor tagged vendor specific enhancements. The following enhancement tags are defined by the POSIX standard:

• all three time stamps of a file in arbitrary resolution (most implementations use nanosecond granularity)
• path names of unlimited length and characterset coding
• symlink target names of unlimited length and characterset coding
• user and group names of unlimited length and character set coding
• files with unlimited size (the historic tar format is limited to 8 GB)
• userid and groupid without limitation (this historic tar format was is limited to a max. id of 2097151)
• a characterset definition for path names and user/group names

Star was the first tar implementation to support this enhanced archive format in 2001, GNU tar added support in 2004.

Other formats have been created to address the shortcomings of tar, such as DAR (Disk Archiver) and rdiff-backup (see Duplicity branch of the Savannah software site), these formats are however not part of any official standard.

Operating system support

Unix-like operating systems usually include tools to support tar files, as well as utilities commonly used to compress them, such as gzip and bzip2. In contrast, Microsoft Windows includes neither graphical nor command-line tools for reading or creating these formats, requiring the use of third-party tools like 7-Zip.

Tarbomb

A tarbomb is derogatory hacker slang used to refer to a tarball that does not follow the usual conventions, i.e. it contains many files that extract into the working directory. Such a tarball can create problems by overwriting files of the same name in the working directory, or mixing one project's files into another. It is almost always an inconvenience to the user, who is obliged to identify and delete a number of files scattered throughout the directory's contents. Such behavior is considered bad etiquette on the part of the archive's creator.

A related problem is the use of absolute paths or parent directory references when creating tarballs. Files extracted from such tarballs will often be created in unusual locations outside the working directory and, like a tarbomb, have the potential to overwrite existing files. GNU tar by default refuses to create or extract absolute paths, but is still vulnerable to parent-directory references. (The bsdtar program, which is available on a number of operating systems and is the default "tar" on Mac OS X v10.6, also ordinarily refuses to follow parent-directory references or symlinks.^[3])

A command line user can avoid both of these problems by first examining a tar file with a command like tar -tf archive.tar. This does not extract any files, but displays the names of all files in the archive. If any are problematic, the user can create a new empty directory and extract the tarball into it—or avoid the tarball entirely. Most graphical tools can display the contents of the archive before extracting them.

Random access

Another weakness of the tar format compared to other archive formats is that there is no centralized location for the information about the contents of the file (a "table of contents" of sorts). So to list the names of the files that are in the archive, one must read through the entire archive and look for places where files start. Also, to extract one small file from the archive, instead of being able to lookup the offset in a table and go directly to that location, like other archive formats, with tar, one has to read through the entire archive, looking for the place where the desired file starts. For large tar archives, this causes a big performance penalty, making tar archives unsuitable for situations that often require random access of individual files.

The possible reason for not using a centralized location of information is rooted in the fact that tar was originally meant for tapes, which are bad at random access anyway: if the TOC were at the start of the archive, creating it would mean to first calculate all the positions of all files, which either needs doubled work, a big cache, or rewinding the tape after writing everything to write the TOC. On the other hand, if the TOC were at the end-of-file (as is the case with ZIP files, for example), reading the TOC would require that the tape be wound to the end, also taking up time and degrading the tape by excessive wear and tear. Compression further complicates matters, as calculating compressed positions for a TOC at the start would need compression of everything before writing the TOC, a TOC with uncompressed positions is not really useful (since you have to decompress everything anyway to get the right positions) and decompressing a TOC at the end of the file might require decompressing the whole file anyway, too.

Key implementations

There have been historically many implementations of tar, and many general file archivers have at least partial support for tar (often using one of the implementations below). Most tar implementations can also read and create cpio and pax (the latter actually is a tar-format with POSIX-2001-extensions).

• FreeBSD tar (also BSD tar) is the default tar on most Berkeley Software Distribution-based operating systems including Mac OS X. The core functionality is available as libarchive for inclusion in other applications.
• GNU tar is the default on most Linux distributions. It is based on the public domain implementation pdtar which started in 1987. It can use various formats, including ustar, pax, GNU and v7 formats.
• Solaris tar is based on the original UNIX V7 tar and is the default on the Solaris operating system.
• star (unique standard tape archiver) was created in 1982 by Jörg Schilling and is published under the CDDL-license. In 1999 was found to be fastest tar implementation.^[4]

Additionally, most pax implementations can read and create many types of tar files.

Adoption rates

Some tar implementations are installed by default on most UNIX-like operating systems and often required for system operation. For example 99.99% of all Ubuntu Linux computers have GNU tar installed,^[5] with BSD tar and star at 0.05% and 0.04%. libarchive, the reusable part of BSD tar, is installed on 76.4% of Ubuntu computers,^[5] since it is used by various applications including parts of GNOME, the default desktop environment on Ubuntu. On Debian, libarchive1 is installed on 35.8% of computers.^[6] On BSD and Solaris operating systems, these numbers are expected to look completely different since they include a different tar by default. As such, the numbers mostly reflect the defaults used by these operating systems and desktop environments.

Since most general file archives on other platforms, including Windows, also support the tar archive format, this format can be considered universally usable on any computer platform.

See also

• List of archive formats
• Comparison of file archivers
• List of Unix programs

References

1. ^ Jargon File, tarball
2. ^ Proposed format to replace tar, by the Duplicity utility's developers.
3. ^ Man page for "bsdtar", as provided by Apple.
4. ^ Preston, W. Curtis (December 15, 1999). "Free Backup Utilities". Unix Backup and Recovery. Sebastopol, CA: O'Reilly Media. pp. 142–143. ISBN 1-56592-642-0. http://books.google.com/books?id=_i1sO47qNnMC&pg=PA142. Retrieved May 28, 2011. 
5. ^ ^{a b} Ubuntu Popularity Contest
6. ^ Debian Popularity Contest

External links

• "star(4) documentation of the archive formats". http://cdrecord.berlios.de/private/man/star/star.4.html.  Documentation summary for many formats with references to the standard
• "tar(5): format of tape archive files". FreeBSD Manual Pages. FreeBSD Project. http://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5&manpath=FreeBSD+8-current.  Includes documentation on how different implementations store various types of information and specialize headers.
• "libarchive - BSD-licensed library with tar file support". http://code.google.com/p/libarchive/. 
• "The standard tar command". The Open Group. http://www.opengroup.org/onlinepubs/7990989799/xcu/tar.html. 
• "The tar Command". The Linux Information Project (LINFO). http://www.linfo.org/tar.html. 
• "Detailed information on Tar and USTAR file headers". http://www.mkssoftware.com/docs/man4/tar.4.asp. 
• "tar(1) man page". OpenBSD Manual Pages. OpenBSD Project. http://www.openbsd.org/cgi-bin/man.cgi?query=tar. 
• "Tar — GNU Project". Free Software Foundation (FSF). http://www.gnu.org/software/tar/. 
• Java Tar API
• Java Tar API With Gnutar Support
• More about tarbomb