Hierarchical Data Format

Infobox file format
name = Hierarchical Data Format
icon =



caption =
extension = hdf, h4, hdf4, h5, hdf5, he4, he5
mime =
type code =
uniform type =
magic =
owner =
released =
latest release version =
latest release date =
genre = scientific data format
container for =
contained by =
extended from =
extended to =
standard =
url = http://www.hdfgroup.org

Hierarchical Data Format, commonly abbreviated HDF, HDF4, or HDF5 is a library and multi-object file format for the transfer of graphical and numerical data between computers. It was created by the NCSA, but is currently maintained by [http://www.hdfgroup.org The HDF Group] . The freely available HDF distribution consists of the library, command-line utilities, test suite source, Java interface, and [http://www.hdfgroup.org/hdf-java-html/hdfview/index.html the Java-based HDF Viewer (HDFView)] .

HDF supports several different data models, including multidimensional arrays, raster images, and tables. Each defines a specific aggregate data type and provides an API for reading, writing, and organizing the data and metadata. New data models can be added by the HDF developers or users.

HDF is self-describing, allowing an application to interpret the structure and contents of a file without any outside information. One HDF file can hold a mixture of related objects which can be accessed as a group or as individual objects. Users can create their own grouping structures called "vgroups."

Tools

* [http://www.hdfgroup.org/hdf-java-html/hdfview/index.html HDFView] A browser and editor for HDF files

HDF5

The HDF5 format is designed to address some of the limitations of the HDF4 library, and to address current and anticipated requirements of modern systems and applications. In 2002 it won an R&D 100 award. [ [http://www.rdmag.com/rd100ach/RD100SearchResults.aspx?strProduct=hdf5&Type=P R&D 100 Awards Archives] ] [ [http://www.hdfgroup.com/HDF5/RD100-2002/ HDF5 Wins R&D 100 Award] ] .

The next version of NetCDF, version 4, is based on HDF5.

Because it uses B-trees to index table objects, it works well for Time series data like stock market ticks or network monitoring data. The bulk of the data goes into straightforward arrays (the table objects) that can be accessed much more quickly than the rows of a SQL database. But you still have B-Tree access for non-array data. If you find yourself designing a Star schema to fit your data into SQL, then you might want to investigate HDF5 as a simpler, faster alternative storage mechanism.

Interfaces

Low-level APIs

* C
* C++
* Fortran
* F90
* Java
* Perl
* Matlab
* IDL
* PyTables – an interface for Python

High-level APIs

* [http://www.hdfgroup.org/HDF5/Tutor/h5lite.html HDF5 Lite] (H5LT) – a light-weight interface for C
* [http://www.hdfgroup.org/HDF5/Tutor/h5image.html HDF5 Image] (H5IM) – a C interface for images or rasters
* [http://www.hdfgroup.org/HDF5/Tutor/h5table.html HDF5 Table] (H5TB) – a C interface for tables
* [http://www.hdfgroup.org/HDF5/Tutor/h5packet.html HDF5 Packet Table] (H5PT) – interfaces for C and [C++] to handle "packet" data, accessed at high-speeds
* [http://www.hdfgroup.org/HDF5/Tutor/h5dimscale.html HDF5 Dimension Scale (H5DS)] – allows dimension scales to be added to HDF5; to be introduced in the HDF5-1.8 release
* Mathematica [ [http://reference.wolfram.com/mathematica/ref/format/HDF.html HDF Import and Export] Mathematica documentation] immediate analysis of HDF and HDF5 data

ee also

* Common Data Format (CDF)
* NetCDF
* FITS, a data format used in astronomy
* GRIB (GRIdded Binary), a data format used in meteorology
* Q5cost a FORTRAN API to use hdf5 in quantum chemistry

External links

* [http://www.hdfgroup.org/ HDF home page]
* [http://www.hdfgroup.org/products/hdf5/index.html HDF5 home page]
* [http://www.hdfgroup.org/HDF5/whatishdf5.html What is HDF5?]
* [http://www.xi-advies.nl/downloads/AnIntroductionToDistributedVisualization.pdf "An Introduction to Distributed Visualization"] ; section 4.2 contains a comparison of CDF, HDF, and netCDF.
* [http://www2.fci.unibo.it/~amonari/talk_834.pdf A presentation on how to handle large datasets in Quantum Chemistry using hdf5]

References