Documentation:Tutorial:DataAccess - MdsWiki
Navigation
Personal tools

From MdsWiki

Jump to: navigation, search

Contents

Accessing MDSplus data in Fortran, IDL, MATLAB and C (old style)

In the first lesson we have seen how to create and fill a MDSplus tree. It is worth noting that in the real world most data in the tree will be written by data acquisition or elaboration programs. Only configuration data and set-up parameters will be stored in the tree using TCL or jTraverser.

In this lesson, we'll learn:

  1. How to export your MDSplus trees to the world;
  2. How to read/write data into MDSPlus trees using Matlab, IDL, Fortran and C.

IMPORTANT: please note the this section describes the old data access interface of MDSplus. C++, Java and Python programmers are recommended to use the Object Oriented interface of MDSplus described in the next section. MATLAB programmers may decide to use either the interface described in this chapter or to take advantage of the Java Object Oriented interface since MATLAB allows a direct interface to Java objects (more on this in the next section). There are no plans to port the IDL interface to the new one (IDL does not support a true OO interface), while volunteers are expected to start the porting of the new interface to F90.


For most users, accessing MDSplus data represents all the required knowledge for using MDSplus. In fact, except for people working in data acquisition, looking at acquired waveforms and developing or running analysis programs are the operations routinely executed by physicists and engineers.


Exporting MDSplus trees to the world

One of the key features of MDSplus is the ability of exporting the contents of MDSplus trees (and not only) over the network in a transparent way. So, you can see your data on any computer connected to the network in quite the same way you would do if you were working on the computer hosting the tree.

Remote data access is achieved via the mdsip protocol, a TCP/IP based protocol which allows remote expression evaluation. As you already know, in MDSPlus everything is an expression. Therefore, to access data, it suffices to evaluate an expression consisting in the reference to a tree node. You can do more, of course, such as evaluating an arbitrary expression involving more than one tree node.

Being based on TCP/IP, the mdsip protocol uses one port number. The default port for mdsip is 8000, but is can be changed in the case that port is used for some other service.

To export data, you need to start a mdsip server on the computer hosting the MDSPlus tree. If you installed MDSplus on a Windows system, all you need is to press Start->Programs->MDSplus->DataServer->Install mdsip service. At this point the mdsip server is activated as a Windows service, and you can forget about it.

You can activate the mdsip server manually, provided your PATH environment variable includes also the directory containing the program mdsip (for Windows usually Program Files\MDSplus, on Unix usually /usr/local/mdsplus/bin), with the command:

mdsip -p 8000 -m -h mdsip.hosts

where flag -p indicates the port number to which the mdsip server listens to, and flag -h indicates the protection file. The default mdsip.hosts file, provided with the MDSplus distribution allows access to everybody from everywhere. You can define some restrictions in data access by changing the content of the mdsip.hosts file (the default mdsip.hosts file includes a brief description of the required syntax). For example,

  • jas@oak.pfc.mit.edu | MDSPLUS
    indicates that user jas has access permission from node oak.pfc.mit.edu and the same local permission of local user MDSPLUS
  • *@132.169.8.* | nobody
    indicates that every user from subnet 132.169.8 has access with the same local permission of user nobody.

In general, the name to the left of @ indicates the remote user, the name to the right of @ and to the left of | indicates the IP address of the remote node (wildchars can be used for both), and the name to the right of | indicates the privileges on the server, taken from that of the specified username (this last feature is available only under UNIX).

Note that the remote user name is provided by the connecting client and not verified, so you can not use it for authentication.

To connect to a mdsip server, MDSplus clients need to specify the IP address (and optionally the port) of the mdsip server. We shall see in the rest of this lesson how to access remote or local data through jScope or through a programming language


Reading and writing data in MDSplus trees in IDL and Matlab

So far, you have seen of jScope can be connected to a mdsip server for accessing data. In this case data is accessed only for display, but in a more general situation you may wish to access MDSplus data for running some kind of analysis program, possibly with a graphical interface for data display. MDSplus allows data access from two tools for interactive data analysis: IDL and Matlab. We shall see that the two interfaces are quite similar, both requiring as first step the connection to a mdsip data server.

Currently, there is a slight difference between the way data is accessed via IDL and Matlab: while in IDL it is possible to read also local data (i.e. not connecting to a mdsip server, but directly using the data access routines of MDSplus), in Matlab it is only possible to access MDSplus data via a mdsip server.

Let's start with IDL, making local data access to the sample MDSplus tree used in the previous section.
Once MDSplus has been installed, all you need to make it work under IDL is to make sure that the IDL_PATH environment variable includes also the directory containing the MDSPlus interfaces for IDL (on Unix, usually /usr/local/mdsplus/idl).

Once IDL has been started, it is possible to evaluate TDI expressions using function mdsvalue, as in the following example:

IDL> a = mdsvalue('2+3')
IDL> print,a
5

function mdsvalue accepts a string argument which represents the expression to be evaluated, and returns the result of such evaluation.
We already know that data references are always seen as expressions, so this mechanism is valid also for data access. However, the following statement (recall that :SIGNAL1 is the name of a node of my_tree containing a signal):

IDL> a = mdsvalue(':SIGNAL1')

will generate an error message. In fact, before accessing data in a MDSplus tree, we need to open the tree first. The correct sequence is (in the example we are working with the experiment model, labelled by shot number -1):

IDL> mdsopen, 'my_tree', -1
IDL> a = mdsvalue(':SIGNAL1')

At this point the IDL variable a contains the result of the evaluation of the expression :SIGNAL1. Being such expression composed of a reference to a node of my_tree, the result of its evaluation is the content of the node (an array in our example). More generally, the result of the evaluation of a TDI expression may be a scalar, and array (both integer or float) or a text string.

Recalling the structure of my_tree, node :SIGNAL has been designed to containing a Signal data type, i.e. a MDSplus data type defining values for both X and Y axis. However, the result of its evaluation is a single array (of Y values), stored then in IDL variable a. This means that there is still information to be retrieved from node :SIGNAL1, i.e. the values of the X axis (usually representing the times associated with the samples of the signal). To retrieve this information, we need to evaluate another TDI expression, as shown in the complete example below:

IDL> mdsopen, 'my_tree', -1
IDL> y = mdsvalue(':SIGNAL1')
IDL> x = mdsvalue('DIM_OF(:SIGNAL1)')

The TDI operator DIM_OF returns in fact the values of the X axis of the argument, assumed to be a signal.

We can also write into MDSPlus trees from IDL, using procedure mdsput, as in the following example in which we write an array into node :NUM1.

IDL> mdsopen, 'my_tree', -1
IDL> mdsput, ':NODE1', '[1,2,3,4,5]'

The first argument of procedure mdsput is the name of the node, expressed as a string. You can use both path names and (more likely) tag names.
The second argument is a string representing the TDI expression to be stored. Recall that, while reading data expressions are evaluated (i.e. the returned values are always numbers or text strings), when writing data into trees, the expression is only compiled, i.e. transformed into the corresponding MDSplus data type prior to be stored. This difference may seem subtle, but it important and worth to be understood. Consider the following code snippet:

IDL> mdsput,':NUM1', '2*:NUM2'
IDL> mdsput,':NUM2', '1'
IDL> a = mdsvalue(':NUM1')
IDL> print,a
2
IDL> mdsput,':NUM2', '100'
IDL> a = mdsvalue(':NUM1')
IDL> print,a
200

Why we retrieve two different values from node :NUM1? The reason is that node :NUM1 contains a TDI expression. Such expression is evaluated when calling mdsvalue, yielding two different results as the value of node :NUM2 is changed between the two data accesses.

Be aware of the usage defined for the node into which we write data. The MDSplus data access layer performs in fact some checks on the kind of data being inserted in the MDSplus tree. Try for example to write a string in node :NUM1 (whose usage has been declared as numeric) and you will see something like this:

IDL> mdsput,':NUM2', ' "This is a string" '
% MDSPUT: %TREE-E-TreeINVDTPUSG, Attempt to store datatype which conflicts with the designated usage of this node

At this point we may think to have in hand all the tools required for making MDSplus data computation and visualization in IDL, but this is not true in practice. In fact, when we perform some kind of computations, we store results in some IDL variables, so we need a mechanism for transferring the content of IDL variables into a MDSPlus tree.
The following example stores a floating point array into IDL variable a and then stores such array in tree node :NUM1

IDL> a = sin(findgen(100)/30.)
IDL> mdsput,':NUM1', '$1', a

The mechanism is almost the same of the previous examples, but in this case the TDI expression is composed of symbol $1, meaning that it will be replaced by the contents of the first argument after the TDI expression in the mdsput argument list.
This mechanism is of course more general, and you can define an arbitrary TDI expression containing symbols $1, $2, ...$n, followed by n arguments whose content will replace those symbols in the expression which will be written into the MDSplus tree.
It is often convenient to store the results of a computation in a Signal structure, so that the same node contains both Y and X information and can be displayed, for example, by jScope without the need of a separate definition of the X axis. Suppose we have the Y and X values in IDL variables y and x, respectively. The following command stores X and Y information into node :SIGNAL1:

IDL> mdsput,':SIGNAL1', 'BUILD_SIGNAL($1,,$2)',y,x

Recall that the TDI operator BUILD_SIGNAL defines a Signal MDSplus data structure, taking the Y and X axis as first and third arguments, respectively.

Finally, the currently open MDSplus tree can be closed with the command:

IDL> mdsclose

Now we are really able to perform data analysis in MDSplus, so let's take a step further, and move to the management of remote MDSplus trees.
Remote data access, both for reading and writing, is the most likely scenario in practice. In fact, even if you are working in your lab, accessing data of your experiment, such data will be likely stored in a machine different from the machine on which you are performing tour computation. In this case, unless you are using a distributed file system (so that you see remote data as local), you need to connect to the mdsip server exporting MDSplus data before working on it.

Working with remote MDSplus trees is as simple as working with local trees. In fact all you need is to call once the procedure mdsconnect, '<IP of the node running mdsip>:<port>'.
For example, if your mdsip server runs on node at IP 150.178.3.101 using port 8000, before any other data access operation (opening trees, reading and writing data), you need to give the following command:

IDL> mdsconnect,'150.178.3.101:8000'

You can also connect to a different mdsip server within the same IDL session by calling mdsdisconnect, and mdsconnect again.

Reading and writing data in MDSplus trees in Matlab

Data access from Matlab is very similar to IDL, and we shall briefly revise the above examples in the Matlab environment.
Observe first that the matlab interface is not directly created by the make procedure of MDSplus. The following steps for the installation are required within Matlab:

  • Include the path of the MDSplus-Matlab interface directory in the search path list of Matlab. In Matlab 6 and following, this requires selecting option File->Set Path... and then adding the directory containing the MDSplus interfaces files in the Set Path form. On windows such directory is usually \Program Files\mdsplus\matlab, while on Unix is usually /usr/local/mdsplus/matlab;
  • Make sure that environment variable MDSPLUS_DIR is defined as the root directory of the MDSplus distribution (e.g. /usr/local/mdsplus);

The following commands read the Y and X axis of node :SIGNAL1 into Matlab variables y and x.

>> mdsopen('my_tree', -1)
>> [y,status] = mdsvalue(':SIGNAL1')
>> [x,status] = mdsvalue('DIM_OF(:SIGNAL1)')

Note that the function returns both the result of the evaluation of the passed TDI expression and an integer variable containing the status for the operation (assumed to be successful if odd).

In the following example an array is written into node :NUM1

>> a=sin((0:100)/10.)
>> mdsput(':NUM2', '$1', a)

And finally, let's write a sinusoidal waveform in :SIGNAL1

>> x = (0:1000)/50.
>> y = sin(x)
>> mdsput(':signal1','BUILD_SIGNAL($1,,$2)',y,x)


Reading and Writing MDSplus data in C

Several data access libraries are available in MDSplus. In particular, one library, called mdslib, is intended to provide easy data access for MDSPlus data. As usual, data access corresponds to the evaluation of a TDI expression, which will specify in most cases the name of a node of a MDSplus tree. In this section we shall present two examples for reading and writing data, respectively. The complete source for both read and write examples is available here and a more complete description of the mdslib library is available here. We shall assume also that the reader is familiar with the C programming language.

As we have already seen in Matlab and IDL, the steps for reading/writing in MDSplus trees are:

  1. Establish a connection with a mdsip server (in the case remote data access is performed)
  2. Open a tree
  3. Read and write data
  4. Close the tree
  5. Disconnect from the mdsip server (for remote access)

When programming in C (and Fortran) things are a bit more complicated than in IDL and Matlab. In the last two languages, in fact, a variable can contain data of any supported type, while in C a variable can only be declared of a given type. As a tree node may contain data of an arbitrary type (string, scalar, array, integer, float etc.), we need a mechanism for mapping the returned data type into the receiving variable. This is achieved by the usage of descriptors. A descriptor specified the kind of variable used for receiving MDSplus data. In particular, a descriptor will specify:

  1. The type of the receiving variable
  2. Its dimension
  3. Its starting address

Before reading MDSplus data, we need therefore to specify a descriptor, and pass its reference to the data access routine. The desc() function is used to create a descriptor, whose prototype is:

int descr (int *dtype, void *data, int *dim1, ...);

where:

  • dtype: specifies the type of the associated variable, and can be any of:
    DTYPE_UCHAR
    DTYPE_USHORT
    DTYPE_ULONG
    DTYPE_ULONGLONG
    DTYPE_CHAR
    DTYPE_SHORT
    DTYPE_LONG
    DTYPE_LONGLONG
    DTYPE_FLOAT
    DTYPE_DOUBLE
    DTYPE_CSTRING
  • data: is the starting address of the associated variable, which may be a scalar or an array;
  • dim1: is the dimension (in elements) of the associated variable

It is possible to define additional dimensions for multidimensional arrays (supported by MDSplus, but not described in this tutorial), and in any case it is necessary to finish the argument list with a pointer to a null integer.

The prototypes of the routines of the mdslib interface are

  • int MdsConnect( const char *server )
    For connecting to a mdsip server. The string argument specifies the IP (and optionally the port) of the machine running the mdsip server.
  • int MdsOpen( const char *treename, int *shotnumber )
    For opening a MDSplus tree.
  • int MdsValue( char *expression, int *ptr_to_descriptor1, ... int *ptr_to_zero, int *ptr_to_len )
    For evaluating a TDI expression. The expression is specified as a string in the first argument. The first argument is followed by a sequence of one or more references to descriptors. The last one will specify the variable which will receive the result of the evaluation of the expression. The other optional descriptors (seldomly used) will define variables whose contents will substitute the $n symbols in the TDI expression (it is the same mechanism we have already seen for writing data in IDL and Matlab). A pointer to a null integer will indicate the end of the description list. Finally, the last argument is a pointer to an integer variable, which will contain the number of samples effectively read. (define that pointer as NULL if you are not interested in this information).
  • int MdsPut( char *node, char *expression, int *ptr_to_descriptor1, ... int *ptr_to_zero )
    For writing an expression in a tree node. The first argument specifies the path name (or tag name) of the node. The second argument specifies the TDI expression to be stored, whose symbols $n will be replaced by the contents of the variables associated with the descriptors of the following list terminated by a pointer to a null integer.
    Be aware that tag names (specified in the node name or in the expression) begins with a backslash (\) which within the specification of C strings needs to be duplicated (\\).
  • int MdsClose( char *treename, int *shot )
    For closing a MDSplus tree.

All the MDSplus routines return an integer status indicating whether the operation has been successful. The usual MDSplus convention is that an odd status value means success, while even values describe some sort of failure condition. It is possible to transform the coded status into a string description using the following routine:

char * MdsGetMsg(int status)

which returns the pointer to a string containing a description of the condition described by the passed status.

We are now ready for our example.Consider the [Documentation:Tutorial:MdsLibC|readExample] routine first, which reads the X and Y ales of signal :SIGNAL1 in tree MY_TREE, shot -1:

#include <mdslib.h>
#include <mdsshr.h>

mdslib.h and mdsshr.h contain the prototypes of the used MDSplus routines.

#define statusOk(status) ((status) & 1)

As we have seen, an odd status means success, so the above macro can be used to make code more readable.

socket = MdsConnect("150.178.3.101:8000");
if ( socket == -1 )
{
fprintf(stderr,"Error connecting to mdsip server.\n");
return -1;
}

Mdsip connection establishment. Routine MdsConnect returns the handle of the used communication socket. In our code we simply check that is different from -1, and then we forget about it.

status = MdsOpen("my_tree", &shot);
if ( !statusOk(status) )
{
fprintf(stderr,"Error opening tree for shot %d: %s.\n",shot, MdsGetMsg(status));
return -1;
}

Open my_tree for shot -1

size = getSize(":SIGNAL1");
if ( size < 1 )
{
fprintf(stderr,"Error retrieving length of signal\n");
return -1;
}

As we do not know in advance the number of samples stored in :SIGNAL1, we need to retrieve this information, in order to reserve the right amount of memory to the arrays which will contain the X and Y axis of the signal. Routine getSignal is described later.

sigDesc = descr(&dtypeFloat, data, &size, &null);
timeDesc = descr(&dtypeFloat, timebase, &size, &null);

Create the two descriptor for arrays data and timebase, which will contain the Y and X axis of the signal, respectively.

status = MdsValue(":SIGNAL1", &sigDesc, &null, &retLen );
if ( !statusOk(status) )
{
fprintf(stderr,"Error retrieving signal: %s\n", MdsGetMsg(status));
return -1;
}
status = MdsValue("DIM_OF(:SIGNAL1)", &timeDesc, &null, &retLen);
if ( !statusOk(status) )
{
fprintf(stderr,"Error retrieving timebase: %s\n", MdsGetMsg(status));
free( (void *)data );
free( (void *)timebase );
return -1;
}

Now everything is ready for calling routine MdsValue, a first time for getting the Y samples, and a second time for getting the X samples, which will be stored in arrays data and timebase, respectively. Variable retLen will contain the number of transferred samples, which in this example as to be equal to variable size.

In the above example we have used a support function getSize() to which we passed the name of the node :SIGNAL1. This routine performs in turn the evaluation of the TDI expression SIZE(:SIGNAL1), which returns the number of samples stored in :SIGNAL1.
Being in this case the information to be retrieved composed of a scalar integer value, fewer arguments are passed to function descr:

 lenDescr = descr(&dtypeLong, &retSize, &null);

In fact it is not required now to specify the number of elements of the associated variable (it is a scalar). Observe that the teminator argument is always required.

Routine writeExample should be now easy to understand. In this case we build the two descriptors for the arrays data and timebase containing the Y and X values of the signal to be written before calling routine MdsPut()

 dataDesc = descr(&dtypeFloat, data, &len, &null);
timeDesc = descr(&dtypeFloat, timebase, &len, &null);

status = MdsPut(":SIGNAL1", "BUILD_SIGNAL($1,,$2)", &dataDesc, &timeDesc, &null);

You can use the following commands to compile and link a program using the MDSplus libraries (we assume here that the mdsplus libraries are in /usr/local/mdsplus):

cc -c -I /usr/local/mdsplus/include <my_program>.c
cc -o <my_program> -L/usr/local/mdsplus/lib <my_program>.o -lMdsLib -lMdsShr -lc

Flag -I indicates the directory from which the include files mdslib.h and mdsshr.h are located
Flag -L instructs the linker to search the shared libraries in /usr/local/mdsplus/lib


Reading and Writing MDSplus data in Fortran

An interface somewhat similar to that in C is available for the Fortran programming language. Still in this case it is necessary to specify the kind of variable which is going to receive the results of a read operation, or which contains the data which will replace the $n symbols when writing a TDI expression. In this case, however, the variable is not directly "attached" to the descriptor, which will only contain the information about the type and size of the variable, and which will be followed by the variable in the argument list.
The steps required for accessing data are the same as in C: connecting to a mdsip server (if doing remote data access); opening the tree; reading/writing data; closing the tree.

The following example illustrates how to read the content of node :SIGNAL1 in my_tree.

     PROGRAM read_test
Include 'mdslib.inc'
REAL*4 current(20000)
REAL*4 times(20000)
INTEGER*4 samples_returned
INTEGER*4 status
INTEGER*4 socket

socket = MdsConnect('150.178.3.101'//char(0))
if(socket .eq. -1) then
write(6,*) 'Error connecting to mdsip server'
stop
endif
status = MdsOpen('my_tree'//char(0), -1)
if(mod(status,2) .eq. 0) then
write(6,*) 'Error opening tree'
stop
endif
status = MdsValue2(':SIGNAL1'//char(0),
+ descr2(IDTYPE_FLOAT,20000,0),
+ current, 0, samples_returned)
if(mod(status,2) .eq. 0) then
write(6,*) 'Error reading data'
stop
endif
write(6,*) samples_returned
status = MdsClose('my_tree'//char(0), -1)

stop
end

Being not possible in Fortran to dynamically allocate arrays, it is necessary to declare in advance enough room for the variables which will contain the Y and X values of :SIGNAL1.
Another difference is that in the string arguments for the IP address, the TDI expression and the node name, we need to add explicitly the terminator character, not supported by default in Fortran. Moreover, the names of the MdsValue, MdsPut, and descr are now changed to MdsValue2, MdsPut2, descr2 (other functions are available, but these are the suggested ones).
The arguments passed to function MdsValue2 differ somewhat from the C interface: in the descriptor, created by function descr2 (specifying in this case that the associated variable if of type REAL*4 and is an array with 20000 elements) is followed by the variable itself. Still in this case a null argument terminates the list of (descriptor, variable) pairs (mostly a single pair), and the last argument will contain the number of samples actually read (possibly less than the dimension of the passed array).

Still in this case the control of the returned status is done by checking whether it is odd or even.

In the following example, a new waveform is written in node :SIGNAL1

  
PROGRAM write_test
Include 'mdslib.inc'
REAL*4 samples(10000)
REAL*4 times(10000)
INTEGER*4 samples_returned
INTEGER*4 status
INTEGER*4 socket

C Build sample Y and X values
do 10 i = 1, 10000
samples(i) = sin(exp(i/3000.))
times(i) = i/10000.
10 continue
socket = MdsConnect('150.178.3.101'//char(0))
if(socket .eq. -1) then
write(6,*) 'Error connecting to mdsip server'
stop
endif
status = MdsOpen('my_tree'//char(0), -1)
if(mod(status,2) .eq. 0) then
write(6,*) 'Error opening tree'
stop
endif
status = MdsPut2(':SIGNAL1'//char(0),
+ ' BUILD_SIGNAL($1,$2)'//char(0),
+ descr2(IDTYPE_FLOAT,10000,0), samples,
+ descr2(IDTYPE_FLOAT,10000,0), times, 0)
if(mod(status,2) .eq. 0) then
write(6,*) 'Error writing data'
stop
endif
status = MdsClose('my_tree'//char(0), -1)
stop
end

In this example, two array are defined for the Y and X axis of the signal to write in the tree. The two variables are then passed to function MdsPut2, together with the corresponding descriptor, specifying their number of elements and type.

To compile and link a Fortran program (e.g. using g77 on a Linux system in which MDSplus is at /usr/local/mdsplus) give the following command:

g77 -o prova prova.f -I/usr/local/mdsplus/include -L/usr/local/mdsplus/lib -lMdsLib

as in the cc command, flags -I and -L specify the libraries where the include file (mdslib.inc) and the shared libraries reside.