Scripting:Requirements

Overview

Users, developers, and managers (oh my) have requested a scripting interface to ParaQ. The ParaQ Scripting Group is responsible for capturing the requirements for scripting, and developing a design document to guide ParaQ development.

We must strike a balance between supporting scripting, and limiting the scope of work required by scripting.

Goals

The goals of ParaQ scripting are:

• To simplify developer testing through scripting.
• To enable users to automate repetative offline tasks.

Requirements

1. Ability to apply a script to multiple different input files and save an image/movie of the visualization from each input file
2. Scripting language is backwards compatible. Scripts written in a previous version of ParaQ must work in newer versions.
3. Script files saved from ParaQ must work in the batch/command line program.
4. Script files must be understandable by humans without needing 2 Tylenol.
5. Either the scripting language or the pvCommandLine application must provide looping and flow-of-control.
6. (Optional?) When a user determines that they like a set of scripted operations, they can attach the script to a button in ParaQ. This gets into the question of what exactly the button does. If you just attach the script, it will load the original dataset and apply all the operations. The user might have wanted all the operations after the data was loaded. If this is implemented, there needs to be some way for the user to edit the script that is attached to the button from within ParaQ.

Use Cases

1. A user makes a visualization in ParaQ and saves out a script file. The user then open the script file in pvCommandLine, twiddles some values and saves the image/movie of the visualization. Some values include filenames, variable names, values of variables.
2. A user opens a toy dataset that models an extremely large dataset. The user creates a visualization and saves a script. The user then opens pvCommandLine and applies the script to the extremely large dataset
3. An expert user can copy and paste from their scripts to new scripts to add steps.
4. An expert user can set some reader parameters for one or possibly several pipeline elements, then re-run the script for different filenmanes ending up with an interactive interface to continue investigating the data (e.g. setting the metadatafile name or loading each sideset in a different pipeline element).
5. A user makes a visualization in ParaQ and saves out a script file. The user then repeats the script for different input files (note: this case could be implemented by state files instead of editable script files.
6. An expert user can edit a script to add mathematical (calculator) functions to create new variables and either save a new data file or proceed to an interactive visualization.
7. (Low priority) A user opens a saved script file with ParaQ and steps through, visualizing what each command actually does to their data.

Current Capabilities

ParaView

ParaView currently uses or enables scripting in the following ways:

1. Wrapping: ParaView classes can be automatically wrapped in a variety of scripting languages. Tcl is the default scripting language. Note that both GUI and server manager (SM) classes are wrapped and can be scripted. Tests and demos use this scripting language to test GUI and SM.
2. Trace file: ParaView saves out a trace file, which is a Tcl/Tk script, that records every action taken by the user. The trace script manipulates both GUI and SM objects but it is mostly at GUI level. This trace file can be 'replayed' in ParaView, restoring the application to the final state of the script. In addition, this script can be hand-edited by expert users, who can make the script do just about anything. Hand-editing the scripts is difficult because they access internal GUI API that is not documented well. Backwards compatibility is poor since GUI API changes impact scripts. Trace file is loaded by the full application.
3. State file: This is a Tcl/Tk script that captures the current state of the application. This is very similar to trace files (with all the same pros and cons) but does not store intermediate steps. State file is loaded by the full application
4. Batch file: This is a Tcl (no Tk) script that is SM level only. Batch file is loaded by a separate application (pvbatch). pvbatch is very similar to the python SM application we have. pvbatch tends to be better at backwards compatibility (i.e. older scripts can be loaded by newer pvbatch). Some SM changes (specially render modules) have broken compatibility but we should be able to avoid these. Batch files are written out by the GUI classes not the SM.
5. SM state file: Server manager is capable of dumping out it's entire state as an XML file. Currently, this file is only used by ParaView Enterprise Edition (PVEE). It is almost identical to the Tcl batch file but we have not had time to merge the two together. This file is very verbose and contains the entire SM state.

Ensight

Ensight currently uses or enables scripting in the following ways:

1. External API: Ensight has an API that can be called by a variety of scripting languages. Metadata support scripts use this API to support metadata. The API is accessed by opening a socket connection to EnSight. The current implementation is hokey because you have to open the command window in EnSight and explicitly tell it to listen for a connection. IMO it should always listen and just pop-up a yes/no dialog box with the remote host if someone tries to connect.
2. Context file: Ensight can save a context file that contains the current state of EnSight. This file file can be loaded in EnSight, restoring the application to the final state of the file. In addition, this files can be hand-edited by expert users, who can make the script do just about anything. Backwards compatibility is good.
3. Archive file: This is a file script that captures the current state of the application. This is very similar to context files (with all the same pros and cons) but it includes binary information about the current state. This means that archive files load significantly faster than context files. Archive file is loaded by the full application. Archive files are not compatible between versions.
4. Command file: This is a file that records all the commands entered into the application in this session. It can be "replayed" to repeat all the commands. Expert users can edit this file to do anything they want. It is somewhat easier to edit than a context file and is even more compatible between versions. It is slower however as it does all the commands in the session even if they have no impact on the final result.
5. Scenario file: EnSight can save a scenario file, which is much like a context file except that it is used only by the standalone EnLiten application to view the created scene.
6. Language Notes: The EnSight command language is very straightforward and reasonably documented. It's error handling is minimal since it was meant as a journal language and not meant for users to write their own scripts. It is fairly well documented so writing scripts is not overly difficult. The biggest problem is debugging since there is basically only one error message for all script errors and it gives no clue as to what the error is.

Cubit

Cubit currently uses or enables scripting in the following ways:

1. Wrapping: SWIG is used to wrap a set of query functions in Cubit (mesh & geometry queries) and the Cubit command interface. SWIG is also used to wrap helper functions that interface with journaling of commands, playback of journals, loading/managing Qt extension modules, etc...
2. Extension Modules:PyQt can optionally be loaded to add to the GUI. C++ extensions wrapped with SWIG can optionally be loaded to add features.
3. Interactive Python prompt:A python prompt is provided to general functionality. In python, Cubit can be queried for objects and state. String based commands can be fed through Cubit to do desired operations. Step-by-Step playback through a python script and breakpoints are supported.

Notes from Scripting Group telecon

• Requirement: Multiple clients can connect to a single ParaView Server Manager
• Design: All functionality (such as linking, etc.) that is to be available across clients must reside in the PSM
• Requirement[Low priority]: State should be loadable and editable without loading data or connecting to a server
• Requirement: Get reasonable client-side data to a script
• Requirement: Scripts play well with modules.

Design

A draft of scripting design based on these requirements can be found here.