From MdsWiki

Handling large signals in MDSplus

In the previous section we have seen how creating pulse files and filling them with data. Data may represent a variety of formats, from scalars to multidimensional arrays and complex types. In particular, we have seen how the "signal" data type is very useful to represent the time evolution of a given quantity. There are however some limitations in the signal usage:

• The number of samples is a signal cannot exceed few billion (~4 GSamples) because the length of arrays handled in MDSplus is stored in a 32 variable
• In practice the maximum number of manageable samples in a signal is even smaller because of the memory requirement and the long access time that will make a program very likely crash when accessing a very large signal, or in any case data access would take an unacceptable time.

MDSplus provides the concept of segmented data for handling large signals. When a signal is stored in segments, there is no limit in its dimension and data readout is efficiently managed. Basically, a signal is stored in a segmented node in chunks (aka segments). At any time it is possible to add a new chunk, that is, to enlarge the signal by adding new samples. This feature is useful for long lasting experiment because in this way, the signal samples acquired so far are accessible, even if the signal is still growing.

When reading a segmented node, the inner layers of MDSplus will stick segments together in order to return a signal data type composed of all the signal sample and the associated timebase. However, if the number of samples actually stored in the segmented node exceeds the maximum number of samples in a MDSplus array, signal readout will fail and, again, even if the total number of samples is less, the time and the required memory resources for reading the large signal would be unacceptable.

To overcome this limitation there are two possible solutions:

• Read each segment using method getSegment(segmentNumber)
• use method setTimeContext to read (a portion of) the (possibly resampled) signal.

Using the first solution, the signal corresponding to the given segment is returned. It is however necessary to handle portions of signal, possibly making the program more complicated.
Using the second solution, the program is the same as for traditional signals and methods getData() and data() will return the desired samples, leaving the inner data access layers of MDSplus handle the join of different segments and resampling. The definition