Projects:hdf5 - MdsWiki
Navigation
Personal tools

From MdsWiki

Jump to: navigation, search

HDF5 Tools

The HDF5 file format was developed by The National Center for Supercomputing Applications at University of Illinois at Urbana-Champaign for storing scientific data.

MDSplus includes tools for accessing and translating HDF5 files. These tools were developed with support from Tech-X Corporation. The translator is called hdf5ToMds. This converter will create an MDSplus tree that reflects the structure of the input HDF5 file and contains its data. The accessor/gateway is a set of tdi functions hdfopen, hdflist, hdfread, and hdfclose. These functions allow MDSplus aware applications to access HDF5 files directly. When coupled with mdsip (mdsconnect) they provide remote access to data stored in HDF5 files.

HDF5 datasets do not have restrictions on names whereas MDSplus node names are limited to 12 alphanumeric case insensitive characters. If the HDF5 name is a valid MDSplus node name, this routine will attempt to add it as is. If not, the non-alphanumeric characters will be replaced by underscores and the name will be truncated. If this fixup results in a duplicate node name, the last characters of the name will be replaced with 1,2,3,4,.... until a unique name is found. An additional node will be added under each node called: "HDF5NAME" and the original HDF5 dataset name will be loaded into this node.

HDF5 attributes and datasets are treated identically by the converter. The converter supports scalars and multidimentional arrays of:

Integer
unsigned byte signed byte
unsigned word signed word
unsigned long signed long
unsigned quad signed quad
Floating
single
double
String

The HDF5 types which are not supported are HT5_COMPOUND, H5T_TIME, H5T_BITFIELD, H5T_OPAQUE, H5T_ARRAY and H5T_VLEN.

Converting HDF5 to MDSplus

The HDF5 to MDSplus converter is called hdf5ToMds. It takes the following form:

hdf5Tomds hdf-file-spec treename shotnumber

Note: the tree must be 'defined' prior invoking the converter. This is done by creating an environment variable 'treename_path' containing the name of the directory this trees file will be stored in. For example:

$ mkdir /trees
$ mkdir /trees/hdf
$ export mdshdf_path=/trees/hdf

Now the converter can be run to create an MDSplus pulse from an hdf5 file.

$ hdf5ToMds my-hdf-file.h5 mdshdf 1

This will take the contents of the hdf5 file called my-hdf-file.h5 and translate them into an mdsplus tree which is shot 1 of the tree called mdshdf. This tree can be accessed using any of the standard MDSplus tools and APIs.

HDF5 datasets do not have restrictions on names whereas MDSplus node names are limited to 12 alphanumeric case insensitive characters. If the HDF5 name is a valid MDSplus node name, this routine will attempt to add it as is. If not, the non-alphanumeric characters will be replaced by underscores and the name will be truncated. If this fixup results in a duplicate node name, the last characters of the name will be replaced with 1,2,3,4,.... until a unique name is found. An additional node will be added under each node called: "HDF5NAME" and the original HDF5 dataset name will be loaded into this node.

Note: In order for the converter to operate, the hdf5 libraries must be in your LD_LIBRARY_PATH.

Accessing HDF5 files from MDSplus applications

The HDF5 accessor functions allow MDSplus aware applications to access HDF5 files directly. There are four functions:

  1. hdf5open(filename)
  2. hdf5list()
  3. hdf5read(name)
  4. hdf5close()

The examples are given here in IDL for brevity, but could be done using any of the MDSplus APIs, or applications.

hdf5Open

hdf5Open opens an existing HDF5 file for read access. It iterates of the all of the datasets and attributes in it, saving away names and linkages. From IDL it can be called:

IDL> status=mdsvalue('hdf5open("/hdffiles/phi.h5")')
IDL> if (status lt 0) then  print, "could not open HDF5 file"

hdf5List

hdf5list returns an array of strings containing the names of all of the data sets and attributes found the currently open HDF5 file.

IDL> names = mdsvalue('hdf5list()')
IDL> help,names
NAMES           STRING    = Array[3851]
IDL> for i=20,30 do begin print,i,' ',names(i)
      20 /GRID/j grid/units
      21 /GRID/phi
      22 /GRID/phi/units
      23 /step_0000000/B/Step number
      24 /step_0000000/B/time
      25 /step_0000000/B/time units
      26 /step_0000000/B/X
      27 /step_0000000/B/X/independent variables
      28 /step_0000000/B/X/units
      29 /step_0000000/B/Y
      30 /step_0000000/B/Z

hdf5Read

hdf5Read returns the data from one data set of attribute from the currently open hdf5 file.

IDL> units = mdsvalue('hdf5Read("/GRID/phi/units")')
IDL> help, units
UNITS           STRING    = 'radians'

IDL> x = mdsvalue('hdf5Read("/step_0000000/B/X")')
IDL> help, x
X               FLOAT     = Array[101, 61, 101]

hdf5Close

hdf5Close closes the currently open hdf5 file.

IDL> status = mdsvalue('hdf5Close()')
IDL> help, status
STATUS          LONG      =            0

remote access

If and mdsip server is running on the machine which has access to the files, suitably authorized users can execute an 'mdsconnect' call before issuing the commands above, and gain access to the file remotely.

IDL> mdsconnect, 'hdf-file-server.corp.org'
IDL> status = mdsvalue("hdf5Open('hdf-file.h5')")
IDL> etc...

Building the HDF5 tools

The hdf5 tools are provided as part of the standard MDSplus distribution. They are built by the top level makefile if the libraries and includes can be found at 'configure' time. By default the MDSplus configure script decides to build the HDF5 tools if the file hdf5.h can be found in the system include directories. This can be overridden to compile against uninstalled HDF5 distributions by defining the environment variable HDF5_DIR prior to running configure at the top of the MDSplus source tree.

$ export HDF5_DIR=/usr/local/hdf5-1.6.1
$ cd mdsplus
$ ./configure
$ cd hdf5
$ make

As mentioned above, if the includes and libraries are installed in the default places, no extra environment variables are needed.