HDF5 documents and links 
Introduction to HDF5 
 HDF5 User's Guide 
HDF5 Reference Manual 
HDF5 Application Developer's Guide 

New Features in HDF5 Release 1.8.0 and Format Compatibility Considerations

 

1. Introduction

This document discusses backward/forward file format compatibility, features introduced in HDF5 Release 1.8.0, and corresponding APIs that may trigger these features and corresponding incompatibility.

Forward compatibility relates to the ability of an older HDF5 Library to read files created by a newer HDF5 Library. An older library should be able to read objects that are known to that library in a file created by Release 1.8.0. This document includes a list of features that are designed to create no forward compatibility problems; a forward compatibility failure with one of these features is a bug.

Backward compatibility relates to the ability of a newer HDF5 Library to read files created by an older HDF5 Library. E.g., if HDF5 Release 1.8.0 is unable to read files created by Release 1.6.5, that is a backward compatibility failure. An HDF5 design criterion is that the HDF5 Library is always backwardly compatible. A backward compatibilty failure is a bug.

The table below lists features introduced in HDF5 Release 1.8.0 and the objects they may act on, then indicates combinations in which compatibility conflicts might arise. Some features will always create objects or write data that will be incompatible with older versions of the library; others will do so only under certain circumstances.

Related document

API Compatibility Macros in HDF5 discusses configurable function macros that were introduced in HDF5 Release 1.8.0 to facilitate the adaptation of applications to that and future releases, criteria for selecting specific configurations, and steps required to set them up.

Whereas New Features in HDF5 Release 1.8.0 and Format Compatibility Considerations discusses format compatibility issues to consider when applications using HDF5 Release 1.8.0 (or a later release) will create files that might have to be read using older versions of the HDF5 Library, API Compatibility Macros in HDF5 discusses an approach to resolving API compatibility issues when moving an application to that release (or a later release).

Reading the table

New features in HDF5 Release 1.8.0 are listed across the top of the table.

Objects in an HDF5 file that features might affect format compatibility are listed on the left-hand side. If the object is printed in bold, the symbols across the row indicate the effect that the feature has when it acts directly on that object. Associated with several objects are one or more objects indented to the right and printed in normal typeface; ripple effects of a feature may affect these objects when the feature acts on the related bold-faced object.

Consider the intersection of the Attribute lines and Attribute and object headers features. The symbols on the Attribute line indicate the effect of the feature when it acts on an attribute; the symbols on the Object line, immediately below, indicate the effect of the feature on an HDF5 object when the feature acts on an attribute attached to that object. For example, the table indicates that attribute creation order can be tracked and indexed with the HDF5 Release 1.8.0 Library only with new-format attributes, and those attr ibutes will not be readable by older libraries. The table further indicates that an object to which such attributes are attached (a dataset, for example) will therefore be unreadable by an older HDF5 library.

How to read symbolic key to table

 

2. A Summary Table
New Features in HDF5 Release 1.8.0 and Their Impact on Format Compatibility

The table below lists new features in HDF5 Release 1.8.0 that may create forward-compatibility conflicts. Objects created or modified with any of these features may be inaccessible via HDF5 Libraries prior to Release 1.8.0.

See section 3, “Features that may create (or are known to create) compatibility concerns,” for further explanation of these functions and their format compatiblity implications.

Key for reading table

Table of features that can create compatibility concerns

Table footnotes:

  1. Under the new implementation, links may be stored as messages in an object header or in a type-2 B-tree in the group's local heap.
  2. By default, a datatype is stored as a message in the object header; a named datatype, which may be shared by several datasets, is stored as an independent object in the file.
  3. Several new features do not involve new or changed APIs.
  4. The following elements are stored (or can be stored; see notes 2 and 3) as messages in an object header.
  5. These APIs create coordinated sets of objects and metadata. Though individual elements are quite likely to be accessible through an older HDF5 library, that library will have no means of understanding the relationships among the various pieces.
  6. Compatibility problems will arise only if an application or the HDF5 Library crashed without properly closing a file with which this functionality was being used.

An expanded table [PDF, large format (11x17 inches)] is available, listing all new features in Release 1.8.0, including features that create no format compatibility issues. These additional features are described more fully in section 4, “Features that are designed to create no compatibility concerns.”

 

3. Features that may create (or are known to create) compatibility concerns

The following HDF5 Release 1.8.0 features may create new HDF5 objects that cannot be read by an older HDF5 Library or modify existing objects such that they cannot be read by an older HDF5 Library.
General features
Object creation using latest available format, specified via the H5Pset_libver_bounds function.
When H5Pset_libver_bounds is called with the low argument equal to HDF_LIBVER_LATEST, new objects are created using the latest available format versions.

This is a file access property, so this object creation behavior can be set for any existing file, and can vary according to application needs each time a file is opened.

Functions:
H5Pset_libver_bounds( hid_t fapl_id, H5F_libver_t low, H5F_libver_t high )
H5Pget_libver_bounds( hid_t fapl_id, H5F_libver_t* low, H5F_libver_t* high )

Default behavior: If H5Pset_libver_bounds is not called with low equal to HDF_LIBVER_LATEST, then the HDF5 Library provides the greatest-possible format compatibility. It does this by creating objects with the earliest possible format that will handle the data being stored and accommodate the action being taken.

Groups and links
Configurable compact-or-indexed link storage (compact and large groups; new group implementation)
Compact-or-indexed groups enable much-compressed link storage for groups with very few members and improved efficiency and performance for groups with very large numbers of members. The efficiency and performance impacts are most noticeable at the extremes: all unnecessary overhead is eliminated for groups with zero members; groups with tens of thousands of members may see as much as a 100-fold performance gain.

This new group implementation also enables user-defined and external links.

User-defined (UD) and external links
User-defined links enable fully-customizable linking in an HDF5 file. External links enable the insertion of a link into an HDF5 group in one file that points to an HDF5 object in a different HDF5 file. In part to provide a model for application developers, external links are implemented as an example of a user-defined link.

Link creation order tracking and indexing
Links can now be tracked and indexed by creation order. Links can be accessed according to an index by creation order or an index by name.

Note: Link indexing by name is inherent to the HDF5 implementation and requires no special setting.

Attributes and object headers
Faster access to large numbers of attributes
Attribute handling is improved for large numbers of attributes attached to a single object.

Large attributes (over ~64k)
Very large attributes (generally larger than 64KB) can be attached to an object.

Attribute creation order tracking and indexing
Attributes can now be tracked and indexed by creation order. Attributes can be accessed according to an index by creation order or an index by name.

(Note: Attribute indexing by name is inherent to the HDF5 implementation and requires no special setting.)

Shared object header messages (SOHM)
This feature enables the sharing of object header messages (attributes, datatypes, fill values, simple dataspaces, filter pipelines) when they are identical across multiple objects. For example, if a large text attribute is attached to many datasets or groups, that attribute can be saved once in the file with only a pointer from each dataset or group.

Miscellaneous
UNICODE support (UTF-8; non-ASCII character set encoding)
Non-ASCII character sets are supported for character strings and link names.

The current implementation includes the ASCII and UTF-8 standard character sets.

Object creation or copying with create intermediate groups property
Use of the create intermediate groups property enables the creation of any groups in the specified path that do not yet exist in the file when creating a new object or copying an existing object in an HDF5 file.

Note that when this property is set, any missing intermediate groups are created with the properties of the lowest-level, previously existing existing group. I.e., if the object elf is being created at FILE:/a1/b543/cde/d12/elf, and only groups a1 and b543 in that hierarchy exist in FILE, groups cde and d12 will be created with the properties of the group b543.

Anonymous object creation
Objects can be created anonymously in a file, i.e., without a location or name, with H5Dcreate_anon, H5Gcreate_anon, or H5Tcommit_anon.

If the object is to be preserved in the file, it must then be linked to a location in the file with H5Llink before the file is closed.

Compression filters: N-bit and scale-offset
Two new compression filters are provided: an n-bit filter and a scale-offset filter.

The n-bit filter compresses data from the in-memory datatype to an n-bit datatype in the file. Since data is normally byte- or word-aligned in memory, this can result in substantial savings in the size of stored data.

The scale-offset filter performs a scale and/or offset operation on each data value and truncates the resulting value to a minimum number of bits before storing it.

NULL dataspaces (H5Screate) (3)
The H5S_NULL dataspace class allows an application to use H5Screate to define a dataset or attribute dataspace with no elements.

This feature is implemented through the addition of a NULL dataspace, H5S_NULL, to the set of valid dataspace classes.

 

4. Features that are designed to create no compatibility concerns

The following HDF5 Release 1.8.0 features are designed to create no forward compatibilty conflicts; an older HDF5 Library should always be able to read and manipulate objects that they create or modify.

Groups and links
Link APIs (H5L) unrelated to user-defined (UD) and external links

Attributes and object headers
Enhanced attribute handling
Enhanced local heap size management (meta data for group)

Miscellaneous
Object API (H5O)
Object copy properties
Improved object information retrieval
Name of referenced object
New-style object open and object creation functions
User-defined identifiers
Other identifier API (H5I) enhancements
User-defined datatype conversion callback functions
Datatype compiler conversion check
Integer-to-floating-point conversion support (H5Tconvert) (3)
Dataset array size reduction (H5Dset_extent)
Dataspace equivalence (H5Sextent_equal)
Direct I/O, bypassing system cache
Parallel I/O optimization (collective chunk I/O)
Enhanced error handling (H5E)
Meta data cache management (6)
File read/write status (H5Fget_intent)
Arithmetic data transform on I/O
Dataspace and datatype serial conversion

High-level APIs
Two-way conversion between datatype and text description of datatype (H5LT)
New attribute datatype: long long (H5LT)
Dimension Scale API (H5DS) (5)
Packet Table API (H5PT) (5)
Fortran APIs for HDF5 Light, Image, and Table interfaces (H5LT, H5IM, and H5TB)

 

HDF5 documents and links 
Introduction to HDF5 
 HDF5 User's Guide 
HDF5 Reference Manual 
HDF5 Application Developer's Guide 
The HDF Group Help Desk:
Describes HDF5 Release 1.8.7, targeted for May 2011.
Last modified: 23 October 2007