HDF5-HL Packet Table
Use Cases

General Use Case:

1. Create/Open HDF5 File
2. Create/Open packet table
• Create fixed-length packet table
• Create variable-length packet table
• Open packet table
3. Perform operations on packet table (as many as desired)
• Append Packets
• Read Packets
• Read packets starting from index
• Set/reset index
• Read packets from any location
• Query packet table
• Query whether packet table identifier is valid
• Query whether packet table is variable-length or not
• Query number of packets in packet table
4. Close packet table
    - If packet table is variable-length, first free any buffers used
5. Close HDF5 File

Writing one kind of data to a fixed-length packet table

An experimenter wants to store data from a single kind of sensor in a Packet Table. First, she must create or load an HDF5 datatype representing a "packet" of her data. This can be any valid HDF5 datatype, though it is likely to be a compound datatype, since each packet corresponds to a horizontal entry in a table. If the experimenter has used HDF5 in the past, she may have already created a datatype representing her data and can simply load it from a file. This step is covered in more detail in the HDF5 User's Guide.
The experimenter then creates the packet table. She must specify the location (file or group identifier), name, and datatype of the new packet table. Her function call might look like this:

hid_t ptable_id= H5PTcreate_fl( file_id, "/trial1", datatype_id, 500 );

The last argument is the "chunk size," which affects how HDF5 stores data in the file; the experimenter may fine-tune this number later. Chunked storage is discussed in more detail in the HDF5 User's Guide.
Now the experimenter starts the experiment itself. Each time a sensor takes a reading, the data is stored in a buffer, which the experimenter uses to create an entry in the packet table:

ret_value= H5PTappend( ptable_id, 1, &(dataBuffer) );

The above statement executes inside a loop and is called to add a new packet every time a new sensor reading is placed in the data buffer.
When the experiment is over, the experimenter closes the packet table.

ret_value= H5PTclose( ptable_id );

She may also wish the close the HDF5 file if she is done accessing it.

Reading data from a fixed-length packet table

An analyst wants to read from a fixed-length packet table like the one created by the experimenter in the previous use case.
First, he must open the packet table. To do this, he needs to know its location and name.

hid_t ptable_id= H5PTopen( file_id, "/trial1" );

Once the packet table is open, the analyst wants to discover how many packets it contains.

hsize_t num_packets;
ret_value= H5PTget_num_packets( ptable_id, &num_packets );

The analyst then begins to read records one-by-one from the beginning of the packet table. He sets the packet table's index to the first packet.

ret_value= H5PTcreate_index( ptable_id );

Then, he creates a for-loop to read each packet into a buffer and perform his analysis on it.

for(int x=0; x<num_packets; x++) {
   ret_value= H5PTget_next( ptable_id, 1, &(readBuffer) );
   // Perform analysis on readBuffer
}

The analyst may want to read several packets into a larger buffer, and start from an arbitrary index without losing his place in the table. To read twenty packets starting from packet number 400, he uses the following call:

ret_value= H5PTread_packets( ptable_id, 400, 20, &(bigReadBuffer) );

Finally, the analyst closes the packet table.

ret_value= H5PTclose(ptable_id);

He may also wish to close the HDF5 file, or he may go on to examine other objects in the same file.

Writing data from multiple sources to a variable-length packet table

An experimenter has data from a number of sources to store in a packet table. Since there are different kinds of data to be stored together, he cannot describe them with a single datatype. Instead, he opens an HDF5 file and creates a packet table to store variable-length packets:

ptable_id= H5PTcreate_vl( file_id, "\trial2", 1000 );

The last value is the "chunk size" and affects how HDF5 stores data in the file. For more information on chunked storage, see the HDF5 User's Guide.
The experimenter then begins his experiment. When a sensor reading is taken, the data is stored in a buffer. The experimenter allocates space for HDF5's hvl_t struct and populates it with a pointer to the data and the length of the data in bytes. He then stores the hvl_t as a packet in his packet table:

hvl_t data_hvl_t;
data_hvl_t.p = &dataBuffer;
data_hvl_t.len = sizeof( dataBuffer );
ret_value= H5PTappend(ptable_id, 1, &(data_hvl_t) );

He repeats this procedure for each sensor reading that arrives, packaging each data buffer with its length and appending it to the packet table.
When the experiment is over, the experimenter closes the packet table.

ret_value= H5PTclose(ptable_id);

When he is done accessing it, he also closes the HDF5 file.

Reading data from a variable-length packet table

An analyst needs to read data from a variable-length packet table like the one created in the previous use case.
First, she opens the packet table. This step is identical to opening a fixed-length packet table.

hid_t ptable_id= H5PTopen( file_id, "/trial1" );

She finds the number of packets it contains:

hsize_t num_packets;
ret_value= H5PTget_num_packets(ptable_id, &num_packets);

Now she wishes to read packets from the packet table. She begins by iterating through all packets in order. She sets the index of the packet table to the first packet:

ret_value= H5PTcreate_index(ptable_id);

Then she reads each packet. Since the packets are of variable-length, her buffer will actually be filled with a hvl_t struct containing a pointer to the data and its length.

hvl_t readBuffer;
ret_value= H5PTget_next( ptable_id, 1, &(readBuffer) );
size_t dataLength= readBuffer.len;
(void *) data= readBuffer.p;

When she is finished accessing that packet, she must be careful to free the memory HDF5 allocated to hold it.

ret_value= H5PTfree_vlen_readbuff( ptable_id, 1, &(readBuffer) );

She can continue to read single packets and free them until has read all the packets in the packet table. She can also read a number of packets at once using an array of hvl_t structs.

hvl_t bigReadBuffer[20];
ret_value= H5PTread_packets( ptable_id, 0, 20, bigReadBuffer );

Again, she must be careful to free the memory HDF5 allocated during the read operation.

ret_value= H5PTfree_vlen_readbuff( ptable_id, 1, &(readBuffer) );

She can then safely close the packet table and the HDF5 file.

ret_value= H5PTclose(ptable_id);

For more examples of the use of hvl_t structs, see the HDF5 User's Guide or the Packet Table examples.

Handling an unknown packet table identifier

A function receives an identifier from another location in the program and the programmer wishes to double-check that it is the packet table she is expecting.
First, she checks that it is in fact a packet table and that it has been opened.

ret_value= H5PTis_valid( ptable_id );

If the identifier does not refer to an open packet table, the return value will be negative.

Next, she makes sure that it is the right kind of packet table:

ret_value= H5PTis_varlen( ptable_id );

Again, the return value will indicate whether the packet table is fixed-length or variable-length.
The programmer may need to remember to close the packet table, depending on the semantics of the application.