Subgrouping Parallel Communicators

This page discusses the design for the general functionality of subgrouping a vtkCommunicator. That is, add the ability to create a new communicator that comprises some subset of the original communicator is possibly a new order. Currently this is possible with vtkMPIGroup and the MPI version of the communicator/controller, but that requires code to check for that type of parallel API, which is often further complicated by the fact that MPI support might not be available at compile time. The class vtkSubGroup also allows you to establish a (limited) group of processes and then perform collective operations within this class, but the implementation is a bit ugly and it is not integrated well back into the communicator and controller classes. The functionality discussed here will subsume and depreciate these two classes.

Defining Process Groups

To make subsetting communicators possible, we first need a mechanism to define a group of processes to use as the subset. The name of this class shall be vtkProcessGroup. It will behave mostly like vtkMPIGroup with the following exceptions.

• vtkProcessGroup will have reliance on MPI or any of the VTK/MPI classes (hence the name change). The initialize function will thus take a generic vtkMultiProcessController or vtkCommunicator.
• vtkProcessGroup may be initialized with a communicator rather than a controller.
• In vtkMPIGroup, the initial processes the group is based on is a bit ambiguous. vtkProcessGroup will keep around a reference to the communicator it was initialized with, and the group will be defined based on that communicator's processes.
• There will be minor API changes.

Creating Sub-Communicators

Once a group of processes are defined, it is important to be able to use this group to create a new communicator. This will be implemented in a new method of vtkMultiProcessController named CreateSubController and takes a vtkProcessGroup object. This method will create a new controller object that is defined over the group it is given. It is important that this method be on vtkMultiProcessController rather than vtkController so that it has the chance of creating a different type of controller depending on the original controller it was based on.

The default CreateSubController method creates a special communicator named vtkSubCommunicator. This communicator keeps around the group object, and for every communication request uses the group object to translate the process ids and then delegates the control to the original communicator. The vtkMPIControler will override the behavior of CreateSubController to use the underlying MPI mechanisms to create a new communicator, thus ensuring that the high performance MPI mechanisms are always used.