From MdsWiki
There are several items that you should consider when configuring your MDSplus installation that could have a drastic affects on the performance of your system.
Contents |
Location of MDSplus data files
Network File Systems
MDSplus provides the ability of having multiple concurrent writers and readers to the MDSplus data files. This enables event driven applications to access the data immediately after each measurement as been acquired and stored in the datafile even as other data is being stored in the same files. Also multiple data acquisition applications can be writing to the same datafile as they read each measurement. This functionality is quite powerful but does require the use of file locking to ensure that processes writing data do not corrupt data actively being written by other processes and that readers of the data do not attempt to read data when the data is in the process of being written. This file locking does slow down the data acquisition process somewhat but when it occurs on local disk devices the performance degradation is minimal. On the other hand, if the files reside on a network file system the cost of file locking can be substantial! Avoid placing your active data files on any network file system such as NFS or cifs/smb. Doing so could slow both reading and writing performance drastically! If you need to put your data files on a device not connected directly to the server then it recommended that you use technologies such as iscsi or fiber channel instead. These perform much better since the operating systems access these devices essentially the same as if they were locally attached drives thus handling the file locking considerably more efficiently than networked file systems.
Distribution of data files into multiple directories
During the course of running experiments with many pulses you may end up with a very large number of data files. If you configured your system using environment variables such as:
mytree_path=/mydata/mytree
then the /mydata/mytree directory will soon contain a large number of files (3 model files + 3 files per pulse x number of pulses). This will work but after a while performance will degrade as the operating system struggles to cache directories and find files in the huge directories.
A much better solution is to distribute your archived files to many directories based on the shot number. MDSplus provides the ability to define path definition with placeholders for digits from the shot number. Using these placeholders in the directory specification will let you distribute the files into multiple directories. You will need to develop scripts to move your files to these directories during an archive operation. For example, on the MIT C-Mod operation the tree path definitions look something like:
cmod_path=alcdata-new::/cmod/trees/new/~t/;
alcdata-models::/cmod/trees/models/~t/;
alcdata-archives::/cmod/trees/archives/~i~h/~g~f/~e~d/~t
where the definition was broken into multiple lines only to facilitate display on the web page. Note the use of the placeholders in the directory specification. The ~t placeholder is replaced by the tree name. The special placeholders ~a through ~i are replaced with the decimal digits of the shot number: ~a by the ones digit, ~b by the tens digit, ~c by the hundreds digits etc. Also note that the path environment variable lists three different directory specification. This definition uses the distributed access capability discussed in the next main section of this page. The tree models are kept in a dedicated directory on the alcdata-models host, new pulse files created by the TCL> CREATE PULSE command are created in the first directory of specified in the path definition. In this case the /cmod/trees/new/cmod directory on the alcdata-new host. Each night a script runs on the MIT server which copies the new pulse files created that day to a tape library archive and then moves the files to a directory on the alcdata-archives host as specified in the mask creating the directory tree if necessary. This configuration enables them to put the new shots on a fast server with high speed disk media to boost performance of most recent pulses while the older shots can be moved to less expensive servers and media. Distributing the archived files to this directory structure improves the performance of accessing the vast number of files representing all the data acquired from the C-Mod experiment over a span of more than 20 years.
Thin versus Distibuted versus Thick Client Data Access
MDSplus provides basically three ways for programs to access MDSplus data which resides on a remote system: Thin Client, Distributed Client and Thick Client. In this discussion a remote system may be a server on a wide area network or a local area network.
Thin Client Access
Clients may access remote servers using the connect operation in the data access API's. The non-object oriented API's provide an MDSCONNECT('host[:port') call while the object oriented API's provide a Connection object. When a client performs a connection, the client asks the server to run an mdsip application on the server which accesses the data by evaluating expressions sent by the client on the server system and then sends the result of that evaluation back to the client. Using this method, all the computation done based on the expression provided is performed by the server system. The client merely sends an expression and any arguments to the server, the server evaluates the expression and the server sends the answer back to the client.
Distributed Client Access
Clients can access MDSplus data files on remote servers transparently as if they were located locally on the clients hard drives by using this method of access. This is done by using the host[:port]::/directory specification of the tree path. When the client opens an MDSplus tree using this method, instead of opening the file on the local system, the client sends a message to an mdsip process on the remote server asking the server to open the file on the clients behalf. Each I/O to the file by the client results in a message to the server requesting that mdsip process on the server to perform that I/O on the clients behalf. Using this method, MDSplus datafiles could be distributed among many different servers and still the client software behaves as if the files were on a local disk. As the client opens a tree it will continue to use tree path environment variables to locate any subtrees referenced in the main tree being opened. This access method allows trees and subtrees to be distributed among many servers.
Since MDSplus permits multiple concurrent readers and writers to each tree it uses file level locking when accessing the files. For this reason accessing MDSplus tree files using mdsip connections to a server process is much more efficient that attempting to access those same files using a network file system such as NFS or cifs.
Thick Client Access
Clients can access MDSplus data files on remote servers transparently much like Distributed Client Access using Thick Client Access. This is done by using the host[:port:]:: specification of the tree path. This differs from Distributed Client Access in that using this method the remote server opens the entire tree including subtrees in the context of the server mdsip process. The client does not need to known where the files for the main tree or the subtrees live as it is the job of the mdsip process on the remote server to find and open the files on behalf of the client.
Which method is best?
Each access method has its advantages and disadvantages based on several factors including network latency between the client and the servers and subsequent number of I/O's required to retrieve the data, computation requirements needed to evaluate the expressions, and availability of libraries and functions used by the expressions and which tools you wish to use to access the data.
If the client is accessing data from a server which is located a great distance from the client it is often advantageous to reduce the number of network transactions required to retrieve data. Thin client access may be more suited for this situation since the transaction would consist of send the expression to be evaluated and retrieving the answer. If distributed access was used instead, the client might end up doing hundreds or more network transactions depending on the complexity of the expression.
If the client is on a high speed local area network the distributed client method is often the better choice since the transaction delays for I/O is minimal and the computation load can be distributed among the client systems. In this case the server is only providing disk I/O functionality on behalf of the client much the same a distributed file server.
In some cases the expressions that the client wishes to evaluate may include functions that are either only available on the client system or only available on the server system. For example, MDSplus might be used for remote access to data stored in a legacy data system on the server. The access routines to that data may exist in shared libraries only on the server so attempting to evaluate expressions on the client that reference those routines would fail since the routines are not present on the client system. In this case, the client must use the thin client method so that the server does the expression evaluation on behalf of the client. Just the opposite situation may arise however. Perhaps the client is working on developing some tdi functions on his local system and want to test it using data on the server. In this case the client must use the distributed access method to cause expression evaluation to take place on the client system while doing data file I/O on the server system on behalf of the client.
The performance of distributed and thick client are similar except that all data transfers are done with a single server with thick client while with distributed client access to the main tree and subtrees might end up communicating with several remote servers. Distributed client method also permits the tree path environment variables to include a list of servers and/or directories on those servers where tree files might be located. When using the thick client method, the remote system server process uses its own tree path environment variables to locate the files.
Some applications developed for manipulating data may only support one method but not the other. Applications like dwscope or traverser cannot use thin client mode unless you provide mdsconnect and mdsvalue calls in the expression you give them to evaluate.
Sometimes it is not at all clear which method is best so you may need to do some experimentation to see which method provides you with the best overall performance. In some cases you may find that it may be best to do a mix of the two methods in the same application, perhaps using mdsconnect/mdsvalue functions in some expressions while not in others. Just try and see what works best for your situation.
MDSplus Event System
The MDSplus event system provides a mechanism to broadcast named occurrences to applications which have established their interest in being notified of these events. The broadcast mechanism can use two different technologies; udp multicast or tcp client /server communication. For local area network configurations, the udp multicast method is very efficient and tends to be more robust than the client/server method. To use this method the participating applications should all have the following environment variables defined the same:
UDP_EVENTS=yes (required to use udp event mechanism)
mdsevent_port=nnn (where nnn is a port number to use for the event.
Default is 4000 if mdsevent_port not defined.
Use different port numbers to segregate experiments
on the same LAN)
mdsevent_address=nnn.nnn.nnn.nnn ( ip address for multicast broadcase.
Default is 224.1.0.1)
The UDP_EVENTS=yes definition is currently required to use udp multicast events. The mdsevent_port definition is optional and specifies the port number to use for the multicast messages. If not provided it will use port 4000. You may want to segregate events from different experiments on the same local area network by using different port numbers for each experiment. The mdsevent_address definition is optional and specifies the ip address to use for the multicast messages. It defaults to 224.1.0.1. Only the first three parts are significant as the messages may be delivered using any of 0-255 for the final part.
If a remote system wishes to participate in event notification and the network configuration prevents delivery or reception of multicast messages from the other systems it must instead use the client/server method. To use this the remote application can define the following environment variables:
mdsevent_server=host[:port] mdsevent_target=host[:port]
The mdsevent_server specifies which host to connect to in order to receive event notifications. When an application registers interest in an event, the application makes a connection to an mdsip process running on the server and asks the mdsip process to listen for events on behalf of the client. Similarly if mdsevent_target is specified, when the client issues an MDSplus event, the application makes a connection to a mdsip process running on the server and asks that mdsip process to generate the event on behalf of the client.
By defining the mdsevent_server and mdsevent_target environment variables, clients on the wide area network can participate in the event notification mechanism even though they cannot receive or send multicast messages to the remote servers.