From MdsWiki
Contents |
Setting up an Experimental Sequence
Introduction to Action Nodes
Up to now, we have handled data in MDSplus trees. In this tutorial, we shall introduce new data types which do not define straight data, but actions. An action node is an element of the tree that does not store data, but stores information about a task that has to be carried out. More concretely, it specifies what has to be done during an experiment, when the operation has to be carried out during the sequence, and which actor will actually carry out the specified action.
Let's concentrate first on what has to be done during an experiment. Usually a set of electronic devices has to be properly configured when preparing the pulse session, and most of these devices will acquire data during the plasma discharge. Afterwards, acquired data will need to be read from the devices and stored into the pulse file. Moreover, data analysis routines will be executed as soon as the required data is available, possibly storing the result of the computation in the pulse file, which typically will contain both raw and processed data.
Basically, all the operations described before consist in either the activation of a routine from some library, or the execution of a program. Therefore, to specify what has to be done, it suffices to specify either the name of a program or a pair (library name, routine name), possibly with one or more arguments.
As stated before, an action specifies also when the corresponding operation has to be done during the experiment sequence. It makes no sense to initialize a device after the discharge, or to call a data acquisition routine before data is available. Therefore it is essential that the operation is executed at the right time. There will be sets of actions which can be executed regardless their respective order, but these will possibly require to be executed before or after another set of actions. To describe such kind of dependence, it is useful to divide the experiment sequence into a set of phases, and to define a sequence number within each phase. Typically, phases INIT, PULSE_ON, and STORE are always defined, but it is possible to define others, in case a finer granularity is required in the definition of the experiment sequence. Sequence numbers define a partial ordering among actions for a given phase: every action defined for sequence n will be executed after actions defined for sequence number n-1 and before actions defined for sequence number n+1.
The last aspect of an action is the definition of the actor which will actually carry out the specified operation at the specified time. In practice, actors will be processes running on one or more machines. We shall see that a name can be associated with each actor process, and therefore it suffices to specify such name in the action definition.
Creating Action Nodes
When you create an action node in jTraverser and select "Modify Data" in the context menu, you will see the following dialog:
In theory you could specify the entire action in form of a TDI expression. However, it is much more convenient to select "Action" in the drop-down menu. This will result in the following dialog:
On the left of this panel you specify when the action has to be executed (i.e. the dispatching information), and on the right you specify the kind of operation to be performed.
Dispatching Information
The choices displayed in the left menu include Sequential and Conditional. The Conditional dispatching mode represents an advanced organization in which actions can be scheduled for execution after the execution of a given set of action. We shall ignore here this option (seldomly used) and we select the Sequential mode, defining sequence numbers. After selecting this mode we get a new form defining the following fields:
- Ident: the name associated with the server process which will actually perform the operation during the sequence;
- Phase: the name of the phase in which the action will be executed. Although you can specify an arbitrary string here, you have to make sure that you actually enter this phase during when you dispatch actions (more about this below).;
- Sequence: the sequence number for that action;
- Completion (optional): the name of the MDSplus event to be generated when the action completes (see the tutorial section on MDSplus events);
- Notify: not used yet.
Which computer is used to carry out a particular action is determined by the 'ident' field of the action node. This field contains a name, and is used to find out the corresponding server. Some actions may be executed by any available server, such as programs doing data analysis, but others cannot. For example, a CompactPCI device initialization can only be executed by the computer running the Control CPU for that CompactPCI crate.
Action Information
Consider now the other option menu on the right. This menu defines the possible operations which can be associated to an action, and allows the following choices:
- Routine
- Procedure
- Program
- Expression
- Method
For all choices except expression, you can define the action timeout (in seconds). If a value is assigned to the timeout and if the operation does not terminate within the specified time, the executing thread is discarded and the action is assumed to be aborted, so that the sequence can continue with the following actions.
Routine
When a routine is defined, the form is changed to show the following fields:
- Image: the name of the library containing the routine. On Linux this will be a .so file, whose directory has to be included in the LD_LIBRARY_PATH environment variable. On Windows this will be a .dll file, whose directory has to be included in the PATH environment variable;
- Routine: the name of the routine.
- Arguments: The list of optional arguments which are passed to the routine.
Procedure
When a procedure is defined, the following fields are displayed:
- Procedure: the name of the command procedure to be executed;
- Language: the language of the procedure (e.g. IDL)
- Arguments: the arguments to be passed to the procedure.
Program
When a program is defined, the following field is displayed:
- Program: the filename of the program to be launched.
Expression
When an expression is defined, the corresponding operation is the evaluation of the specified expression. Recall that expressions are very flexible, and can be composed of a sequence of statements, thus behaving as a program written in the TDI language.
Method
Finally it is possible to define a Method. Methods are operations associated with the devices involved in the experiment and represent in practice the most common type of operation executed during the shot sequence.
Usually you will not need to add actions calling device methods yourself. Instead, Traverser (and jTraverser) allow you to automatically create the correct action nodes for most supported devices. When you select the Add device item in Traverser (the tree has to be opened in edit mode), Traverser will ask you for the type of the device and then create the appropriate node structure. If you look into the new branch, you will see that this already includes action nodes that call the methods specific to this device. You may, however, still have to adjust the phases, sequence numbers or ident fields.
Dispatching Actions
Using mdstcl and mdsip
The simplest way to run an experimental sequence is to start mdsip action servers on the computers that should execute actions, and then use the mdstcl shell to control the dispatching.
The command for running an action server is the following:
mdsip -p <server port number> -s -h <hosts file location>
The next step is to let mdstcl know about the mapping from ident fields in action nodes to hostnames running mdsip action servers. This is done by means of environment variables of the form
server_name = <server IP>:<server port>
Once you have set up the environment variables and started the mdsip servers, you start the TCL shell with mdstcl. Then you have to lead the experimental model
TCL> set tree <tree_name> /shot=-1
Increment the current shot number and create the new pulse ('0' always stands for the current pulse, while '-1' stands for the experimental model)
TCL> set current <tree_name> /increment TCL> create pulse 0
Finally, open the new shot and build the dispatching table
TCL> set tree <tree_name> /shot=0 TCL> dispatch /build
Afterwards, you can cycle through the phases of your experiment with the 'dispatch' command, which dispatches the actions for the phase to the appropriate mdsip servers"
TCL> dispatch/phase=<phase_name>
The command will block until all actions have been executed. Note that if your experiment requires to stay in a specific phase for a minimum number of time (e.g. so that the digitizers can record enough data) you have to manually "sleep" long enough before issuing the next dispatch command.
Once the sequence is complete, you should close the tree with
TCL> close tree
Although you can rely solely on mdstcl and mdsip for dispatching actions, there is also a much more advanced framework available. It has the following additional features:
- The possibility of running multiple servers associated with the same name
- Dynamic load balancing
- The possibility of aborting actions
- A Graphical User interface
- More sophisticated dispatching definitions (not explained in this tutorial)
jServer
The first step to using the advanced framework is to use jServer processes instead of mdsip action servers.
To start a jServer, all you have to do is execute
# jServer [port]
This will start a jServer instance that listens on the given port. Note that jServer will accept all connections, so you should configure your firewall to reject all connections to this port that do not come from trusted machines. Note that jServer will have to access the tree to carry out any actions, so the environment variable <treename>_path has to be set correctly.
jDispatcher
Instead of dispatching directly from mdstcl, we now use a dedicated program called jDispatcher. The jDispatcher program runs on only one computer and is in control of the experimental sequence. Whenever the experiment switches into a new phase, jDispatcher determines which actions have to be carried out and assigns them to the available jServers. The jServer tasks will carry out the scheduled actions, and will communicate to jDispatcher the completion (successful or unsuccessful) of the action, so that the jDispatcher can proceed with the following actions. If more than one server is defined, it is possible to have several actions concurrently in execution (i.e. actions with the same sequence number within the same phase).
To start a jDispatcher, execute
# jDispatcherIp [tree_name]
This will start a jDispatcher instance for the given tree. Note that jDispatcher will accept all connections, so you should configure your firewall to reject all connections to this port that do not come from trusted machines.
jDispatcher requires a property file called jDispatcher.properties in the current directory. This file specifies the ports to use and the mapping from 'ident names in the tree to the hostnames running jServer processes. A typical properties file looks like this:
#The port at which jDispatcherIp listens to incoming commands jDispatcher.port = 8001 #server classes and addresses jDispatcher.server_1.class = SERVER_1 jDispatcher.server_1.address=localhost:8002 jDispatcher.server_2.class = SERVER_2 jDispatcher.server_2.address=localhost:8003 #default server id: used by jDispatcher when an unknown server is found jDispatcher.default_server_idx = 1 #phase names and corresponding identifiers jDispatcher.phase_1.id = 1 jDispatcher.phase_1.name = INIT jDispatcher.phase_2.id = 2 jDispatcher.phase_2.name = PULSE_ON jDispatcher.phase_3.id = 3 jDispatcher.phase_3.name = STORE # The ports used by jDispatcher to export information to jDispatchMonitor jDispatcher.monitor_1.port = 8010 jDispatcher.info_port = 8011
The properties file contains also the definition of the phases used in the sequence, and an associated id: normally you write this once to include the phases you have defined and then forget about it.
A bit more explanation is required to fully understand the sequential semantics in jDispatcher. If an action is declared as sequential, jDispatcher ensures that it will be executed before all sequential actions with a larger sequence number and after all sequential actions with a lower sequence number within the same server class. Sometimes, however, it is necessary to define some global synchronization which span over different server classes. This is possible by declaring in jDispatcher.properties a (list of) synchronization sequence numbers for a given phase with the property
jDispatcher.phase_<n>.synch_seq_numbers
For every synchronization number (if any) defined in this sequence, jDispatcher ensures than in all server classes sequential actions with larger sequence number are executed after actions with a sequence number which is lower or equal to the synchronization number.
jDispatchMonitor
The last two lines in the properties file are required by jDispatcherMonitor, which is the Graphical interface of jDispatcher. jDispatchMonitor can run on a different machine, and uses TCP/IP to get information from jDispatcher. If you are running jDispatchMonitor on a different machine, recall that a copy of file jDispatcher.properties is required in the working directory of jDispatchMonitor.
To start a jDispatchMonitor that shows the status of the jDispatcher running on localhost and exporting information to port 8010, execute
jDispatchMonitor localhost:8010
Controlling jDispatcher
Once the jDispatcher instance is running, you can send it commands using mdstcl. The syntax is
TCL> dispatch/command/server=<jDispatcher IP><jDispatcher port> <Dispatcher-Command>
So to have jDispatcher initialize the dispatching, you would execute
TCL> dispatch/command/server=<jDispatcher IP><jDispatcher port> set tree <tree_name> TCL> dispatch/command/server=<jDispatcher IP><jDispatcher port> increment current TCL> dispatch/command/server=<jDispatcher IP><jDispatcher port> create pulse 0 TCL> dispatch/command/server=<jDispatcher IP><jDispatcher port> dispatch /build
This assumes that you already have a "current" shot number that can be incremented. You can set the current shot with
TCL> set current <tree> <shotno>
To cycle through the experimental phases, you use the following command:
TCL> dispatch/command/server=<jDispatcher IP><jDispatcher port> dispatch/phase=<phase_name>
Once the sequence is complete, you would issue
TCL> dispatch/command/server=<jDispatcher IP><jDispatcher port> close tree
Note that these are the same commands that we could give directly to mdstcl. But by sending them to jDispatcher instead, we are able to make use of jDispatcher's more advanced facilities and can e.g. track the progress with jDispatchMonitor.
Example
Defining Action Nodes
Let's extend our MY_TREE experiment model by adding three actions: ACTION_1, ACTION_2 and ACTION_3 with the following configuration:
- ACTION_1
ACTION_1 defines an action composed of a TDI expression, which gets evaluated at the time this action is executed. The TDI expression is composed of two TDI statements: the first one writes a message, and the second one represents the value which is returned by the evaluation of the expression (odd status in MDSplus means success). ACTION_1 will be executed in phase INIT, with a sequence number of 1, by server SERVER_1.
- ACTION_2
The configuration of ACTION_2 is quite similar to that of ACTION_1. Only the message , the sequence number and the server are changed. This mean that ACTION_2 will be executed after ACTION_1 by another server.
- ACTION_3
In this case ACTION_3 executes a program. The path name of the program is provided in field Program The sequence number is the same as ACTION_2, so we cannot state which action will be executed first (both however after ACTION_1). The server is SERVER_2, and therefore this action will be executed by the same server of ACTION_2.
Let's also add a device (with the tree open in edit mode) by adding device MY_ADC , by selecting the Add Device popup item. Once the item has been selected the following window is displayed.
The first field defines the kind of device being added (DEMOADC this is a demo device, available in the current MDSplus distribution, which defines two dummy methods: INIT and STORE.) The second field defines the name of the device, actually the name of the root of the subtree containing the data items associated with the device.
Once the device has been added you can see that a subtree has been created, including two actions called INIT and STORE. If, for example you look at the INIT action, you will see the following configuration:
This action specifies method INIT (for device type DEMOADC) applied to the device instance called MY_DEVICE. The method will be invoked during phase INIT at sequence number 50 (therefore after ACTION_1, ACTION_2 and ACTION_3) by the server named CAMAC_SERVER.
The task associated with ACTION_1 and ACTION_2 is a TDI expression which, when evaluated, prints a message. The actions coming with device MY_DEVICE (of type DEMO_ADC) define methods INIT and STORE, respectively, which, besides making other operations (see the next tutorial), print a message.
If you want to have a look at the resulting TDI expressions in the action nodes read [this].
Writing Actions
We are now almost ready to run the experiment MY_TREE. As ACTION_3 defines a user program, we need to provide first an implementation for program demo, such as the following one (in demo.c):
public void main(int argc, char *argv[]) { printf("Hello, I am a C program"); }
Compile and link demo.c and make sure that the environment variable PATH contains its directory.
Starting Server Processes
The jDispatcher.properties file for this demo application looks as follows:
#The port at which jDispatcherIp listens to incoming commands jDispatcher.port = 8001 #server classes and addresses jDispatcher.server_1.class = SERVER_1 jDispatcher.server_1.address=localhost:8002 jDispatcher.server_2.class = SERVER_2 jDispatcher.server_2.address=localhost:8003 #default server id: used by jDispatcher when an unknown server is found jDispatcher.default_server_idx = 1 #phase names and corresponding identifiers jDispatcher.phase_1.id = 1 jDispatcher.phase_1.name = INIT jDispatcher.phase_2.id = 2 jDispatcher.phase_2.name = PULSE_ON jDispatcher.phase_3.id = 3 jDispatcher.phase_3.name = STORE # The ports used by jDispatcher to export information to jDispatchMonitor jDispatcher.monitor_1.port = 8010 jDispatcher.info_port = 8011
In the listed configuration, two jServer tasks are declared, both running on the same machine, and listening at port 8002 and 8003, respectively. The associated names are SERVER_1 and SERVER_2, the same names we declared in the ident field for ACTION_1 and ACTION_2. Observe that the default ident content for actions INIT and STORE, belonging to the subtree associated with device MY_DEVICE we added before, is CAMAC_SERVER. jDispatcher will dispatch such actions to SERVER_1, because this server is defined as the default server.
Finally, we are defining three experimental phases: INIT, PULSE_ON and STORE, corresponding to the phases that we specified in the action nodes themselves.
Running the Experiment
To run the sample sequence defined in experiment MY_TREE, we need to do:
- Start two jServer applications (on two separate terminal window), by issuing the commands
java jServer 8002
and
java jServer 8003
- In the same directory containing jDispatcher.properties, start jDispatcher by issuing the command
java jDispatcherIp my_tree
(the name jDispatcherIp comes from the fact that the main class for the dispatcher handles mdsip messages)
- Start the GUI for jDispatcher by issuing the command:
java jDispatchMonitor localhost:8010
in the case jDispatchMonitor is running on a separate mahine, specify the IP address of jDispatcher in the command line.
- Prepare the following script files for TCL (just to avoid typing long commands):
init.tcl
dispatch/command/server=localhost:8001 set tree MY_TREE dispatch/command/server=localhost:8001 increment current dispatch/command/server=localhost:8001 create pulse 0 dispatch/command/server=localhost:8001 dispatch /build dispatch/command/server=localhost:8001 dispatch /phase INIT
pon.tcl
dispatch/command/server=localhost:8001 dispatch /phase PULSE_ON
store.tcl
dispatch/command/server=localhost:8001 dispatch /phase STORE dispatch/command/server=localhost:8001 close tree
and issue the following commands from TCL (in another terminal window):
TCL> @init TCL> @pon TCL> @store
If everything works, you will see the messages on the jServer terminals. At the same time you will see the sequence information in the graphical user interface of jDispatchMonitor. If things do not work......check better what you have done: be careful, you are running a Nuclear Fusion Experiment!
Variations
Now let's make some variations. First of all let's change the expression representing the task for ACTION_2 by adding a wait command and another print command, as follows:
write(*, "Hi, I am ACTION_2"); wait(10.); write(*, "ACTION_2 finished"); 1;
Now ACTION_2, defined to be run at phase INIT with sequence number 2 will take more time to be executed. ACTION_3 is also defined to be run at phase INIT with sequence number 50, so there is a chance that the latter action needs to for wait the termination of ACTION_2 before being executed. Actions with the same sequence number may be in fact dispatched in any order. To make sure that both actions are executed soon, we can add a third jServer task, listening, say, at port 8004. So we add the following lines in jDispatcher.properties:
jDispatcher.server_3.class = SERVER_2 jDispatcher.server_3.address=localhost:8004
Let's repeat the shot sequence, after re-launching the dispatcher and the dispatch monitor: this time the two actions are both executed soon, because the dispatcher has the chance of associating a separate (equivalent) server with ACTION_2 and ACTION_3.
Now let's kill the jServer we just launched. A message in the jDispatcher terminal will warn us that a server died, but no worry, jDispatcher is fault tolerant, and knows what to do, provided at least one instance of the default server is running. Note that you can run a subset of the servers declared in jDispatcher.properties. A list of declared and running servers is displayed by jDispatchMonitor by selecting Info->Show server option. At any time a new jServer task can join jDispatcher, provided it has been declared in the properties file.
Another useful feature of jDispatcher is the possibility to abort an action which, for some reason hangs. To exercise this, let's change the wait time in the TDI program associated with ACTION2, and port it to, say, 1000 seconds. When you execute the INIT phase, the whole system will wait the termination of such action. To abort it, select the action in jDispatchMonitor GUI and then click the popup option "Abort". In response, jServer will discard the thread actually executing the action, so that jDispatcher can proceed with the other actions.
You can also manually re-dispatch an executed actions, once the phase has terminated, via the popup option "Redispatch".
What's next
While in the previous tutorials we learned how to access MDSplus data, we have learned here how to perform data acquisition using MDSplus. A further step is however required: when acquiring data we use some kind of hardware device, requiring some kind of programming. Several devices are already supported in the current MDSplus distribution, and, if you are lucky, the device you are going to use is already available. More probably, you will need to integrate device support code into MDSplus. The next tutorial will show you how to do it.
Control Scripts
Having to type a TCL commands for every phase whenever you run your experiment can get quite boring. For that reason, most people write additional dispatch control scripts that provide the user with a GUI to run the experiment and automatically send the correct commands. An example for such a script can be found here: take_shot.py