<HTML>
<HEAD>
<TITLE>Re: [Paraview] Problems with Paraview 3.6.1 on cluster (client - server mode)</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Luis,<BR>
<BR>
I saw your email, but I have been on travel and don’t have a quick answer for your problem, so I have not had a chance to respond. This is why it is always a good idea to cc the ParaView mailing list.<BR>
<BR>
Offhand I do not know what the problem might be. The first thing I would check would be to make sure that none of the processes has crashed. You are loading a lot of data, so if it is not partitioned in a way that allows it to be load balanced well you could be running out of memory. If you find that all the processes are doing something, then it would be helpful to know exactly where they are spending a lot of time.<BR>
<BR>
-Ken<BR>
<BR>
<BR>
On 12/7/09 12:47 AM, "myaccountmail9 myaccountmail9" <<a href="myaccountmail9@gmail.com">myaccountmail9@gmail.com</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hi Ken, did you received my last e-mail (on 1st december)? I write you again...<BR>
<BR>
<BR>
It seems that we have finally resolved the problem regarding OSMesa.<BR>
<BR>
Now can you help us to configure paraview options to improve the performances on our cluster (8 nodes dual quad core -> 8 core per node = 64 cores; no GPU; 10 GB ram per node)?<BR>
<BR>
If we run pvserver vith 64 processes (8 processes per node with: mpirun -np 64 -machinefile list pvserver --use-offscreen-rendering)<BR>
we have problems to load and/or execute operations with big models (20*10^6 cells) exported from CFD++ in Ensight binary mode (it seems that Paraview loop endless).<BR>
<BR>
Our client configuration is (version 3.6.1):<BR>
<BR>
edit -> settings -> render view:<BR>
- use immediate mode rendering: checked<BR>
- use triangle strips: unchecked<BR>
- LOD threshold: unchecked<BR>
- outline threshold: 500 MCells<BR>
- allow rendering interrups: unchecked<BR>
- enable deept pelling: checked<BR>
- number of peels: 4<BR>
- use offscreen rendering for screenshot: checked<BR>
<BR>
edit -> settings -> server:<BR>
- remote rendering threshold: checked, 0 MBytes<BR>
- suppress ordered compositing: checked<BR>
- subsample rate: checked, 4 pixels<BR>
- squirt compression: checked, 10Bits<BR>
- still subsample rate: checked, 4 pixels<BR>
- client collect: checked, 97 MBytes<BR>
- compositing threshold: unchecked<BR>
<BR>
<BR>
Can you help us?<BR>
Thanks again<BR>
Luis<BR>
<BR>
<BR>
2009/11/10 Moreland, Kenneth <<a href="kmorel@sandia.gov">kmorel@sandia.gov</a>><BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>If you are using OSMesa, you should not need X running on any of the nodes. It should be sufficient to have the OSMesa libraries and associated Mesa GL libraries installed on a shared folder. However, make sure that folder is actually mounted everywhere (a common failure in my experience) and that they are the ones actually loaded for each node.<BR>
<BR>
-Ken<BR>
<BR>
<BR>
<BR>
On 11/10/09 2:31 AM, "myaccountmail9 myaccountmail9" <<a href="myaccountmail9@gmail.com">myaccountmail9@gmail.com</a> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> > wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>I'm sorry, I explained the problem very bad; we are using OSMesa library.<BR>
<BR>
If we run pvserver only on the master node:<BR>
<BR>
mpirun -np 2 pvserver --use-offscreen-rendering<BR>
<BR>
It's all ok (even if DISPLAY variable is not set: we don't know if it's necessary).<BR>
<BR>
<BR>
If we try to use 2 nodes (master and node1), running this command on the master node:<BR>
<BR>
mpirun -x LD_LIBRARY_PATH=/home/user/ParaView/ParaView-bin/bin/:/usr/lib/:/home/user/ParaView/ComponentiAggiuntivi/lib -np 2 -machinefile /home/user/ParaView/NodesList /home/user/ParaView/ParaView-bin/bin/pvserver --use-offscreen-rendering<BR>
<BR>
the error is:<BR>
<BR>
Listen on port: 11111<BR>
Waiting for client...<BR>
Client connected.<BR>
[node1:06211] *** Process received signal ***<BR>
[node1:06211] Signal: Segmentation fault (11)<BR>
[node1:06211] Signal code: Address not mapped (1)<BR>
[node1:06211] Failing at address: 0x21<BR>
mpirun noticed that job rank 0 with PID 22632 on node master exited on signal 15 (Terminated). <BR>
1 additional process aborted (not shown)<BR>
<BR>
<BR>
X is running on the second node (called node1); DISPLAY is not set neither on node master nor in the second node.<BR>
Must X server run on each node and must DISPLAY variable to be set also if we are using OSMesa library?<BR>
<BR>
Must we install OSMesa on each node or it's enough to have installed that library on a shared folder (/home/user/ParaView/Components/lib) in the master node?<BR>
<BR>
Have you any other idea?<BR>
<BR>
Thank you very much for your patience<BR>
Luis<BR>
<BR>
<BR>
2009/11/9 Moreland, Kenneth <<a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> ><BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>By “usual error” do you mean the crash in the GL or pthreads library? It could be that the environment in the non-master nodes is incorrect.<BR>
<BR>
-Ken<BR>
<BR>
<BR>
<BR>
On 11/6/09 9:26 AM, "myaccountmail9 myaccountmail9" <<a href="myaccountmail9@gmail.com">myaccountmail9@gmail.com</a> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> > wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hi ken, thanks again.<BR>
<BR>
we recompiled again OSMesa library (only on the master node of our cluster; the libraries file are generated on a shared folder - /home/user/ParaView/Components/lib - visible on each node), then we recompiled Paraview with these libraries (and setting shared libraries=off in ccmake menu); now we are able to connect the client with pvserver provided that pvserver runs only on the master node.<BR>
<BR>
If we run pvserver on two or more nodes with the command:<BR>
<BR>
mpirun -x LD_LIBRARY_PATH=/home/user/ParaView/Components/lib -np 2 -machinefile NodesList pvserver --use-offscreen-rendering<BR>
<BR>
we always get the usual error when we try to connect the client with the server<BR>
<BR>
Must X server run on each node? Currently X server is not running on any node (if we execute "pidof X" we obtain nothing) and DISPLAY variable is not set, but we can connect the client with the master node...<BR>
<BR>
Have you any idea? Can you help us?<BR>
<BR>
Thank you very much<BR>
Luis<BR>
<BR>
<BR>
<BR>
2009/10/29 Moreland, Kenneth <<a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> ><BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>(Responding back to the ParaView list.)<BR>
<BR>
These symbols should be defined in the GL library that is part of Mesa. You should have a library like /home/myuser/TempOSMesa/Mesa-7.6/lib/libGL.so. That is where the symbols should be defined. libOSMesa.so simply references those libraries. It is because the symbols cannot be found in the GL library that you are getting the error. You can use the ‘nm’ command to query what symbols are defined are what are required.<BR>
<BR>
-Ken<BR>
<BR>
<BR>
<BR>
On 10/28/09 2:16 AM, "myaccountmail9 myaccountmail9" <<a href="myaccountmail9@gmail.com">myaccountmail9@gmail.com</a> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> > wrote:<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>It sounds like you are still linking to the system OpenGL libraries somehow. Those symbols should be defined in the libGL.so that is built with Mesa (whereas the /usr/lib/libGL.so on your system probably does not have them). <BR>
</SPAN></FONT><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
Hi Ken,<BR>
thanks again.<BR>
<BR>
Looking the errors such as:<BR>
<BR>
"home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so: undefined reference to `_mesa_free_context_data'"<BR>
<BR>
I believe those symbols must be defined in libOSMesa.so library and not in /usr/lib/libGL.so library.<BR>
<BR>
In fact, if we install OSMessa using Debian packages we can compile ParaView, but we can't connect client and server; the error always is similar to:<BR>
<BR>
ParaView3/Servers/Common/vtkServerConnection.cxx, line 67<BR>
vtkServerConnection (0x3182220): Server Connection Closed!<BR>
<BR>
Executing with: 0<BR>
*** Process received signal ***<BR>
Signal: Segmentation fault (11)<BR>
Signal code: (128)<BR>
Failing at address: (nil)<BR>
[ 0] /lib/libpthread.so.0 [0x7f4fc6536a80]<BR>
[ 1] /usr/lib/libGL.so.1(glBlendFuncSeparateEXT+0x16) [0x7f4fc7bc12e6]<BR>
*** End of error message ***<BR>
Segmentation fault<BR>
<BR>
<BR>
If instead we compile OSMesa ourselves we can't compile ParaView: I believe that something went wrong compiling OSMesa (some simbols aren't present in libOSMesa.so created), but I don't know if compiling OSMesa ourselves can help us to resolve our problem.<BR>
<BR>
We also tried to install on our cluster ParaView 3.6.1already compiled (downloaded from <a href="http://www.paraview.org/paraview/resources/software.html">http://www.paraview.org/paraview/resources/software.html</a>) and it works: we can establish client-server connection.<BR>
<BR>
But if we try to run more 2 or more pvserver instances on the same node (mpirun -np 2 pvserver), we have this error:<BR>
<BR>
"vtkProcessModuleConnectionManager (0x23d6da0): Failed to set up server socket"<BR>
<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'> Did you do a clean build of ParaView (that is, delete the binary directory and start over from scratch)? If you just tried to change some CMake variables in an existing build, you may have some stale libraries built.<BR>
</SPAN></FONT><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
We did "make clean". <BR>
</SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
You should also check to make sure that the executables being built are linking to your compiled Mesa libraries instead of the system libraries. You can check this with the ldd command. You may have to set your LD_LIBRARY_PATH environment variable to point to /home/myuser/TempOSMesa/Mesa-7.6.<BR>
</SPAN></FONT><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
We also tried to set LD_LIBRARY_PATH.<BR>
<BR>
Can you help us?<BR>
Luis<BR>
<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
-Ken<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
On 10/23/09 3:27 AM, "myaccountmail9 myaccountmail9" <<a href="myaccountmail9@gmail.com">myaccountmail9@gmail.com</a> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> > wrote:<BR>
<BR>
Hi Ken,<BR>
<BR>
thank you.<BR>
We compiled Mesa with:<BR>
<BR>
./configure --enable-gl-osmesa --disable-gallium<BR>
make<BR>
make install<BR>
<BR>
and we configured Paraview with:<BR>
<BR>
OPENGL_INCLUDE_DIR=/home/myuser/TempOSMesa/Mesa-.6/include<BR>
OPENGL_gl_LIBRARY=/home/myuser/TempOSMesa/Mesa-7.6/lib/libGL.so<BR>
OPENGL_glu_LIBRARY=/home/myuser/TempOSMesa/Mesa-7.6/lib/libGLU.so<BR>
OPENGL_xmesa_INCLUDE_DIR=/home/myuser/TempOSMesa/Mesa-7.6/include<BR>
OSMESA_INCLUDE_DIR=/home/myuser/TempOSMesa/Mesa-7.6/include <BR>
OSMESA_LIBRARY=/home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so <BR>
<BR>
but we have many error compiling Paraview; for example:<BR>
<BR>
/home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so: undefined reference to `_mesa_free_context_data'<BR>
/home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so: undefined reference to `_mesa_get_current_context'<BR>
/home/myuser/TempOSMesa/Mesa-7.6/lib/libOSMesa.so: undefined reference to `_mesa_enable_2_1_extensions'<BR>
<BR>
Do you think there is something went wrong installing Mesa?<BR>
Can you help us?<BR>
<BR>
Thank you again.<BR>
Luis<BR>
<BR>
<BR>
<BR>
<BR>
2009/10/21 Moreland, Kenneth <<a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> ><BR>
</SPAN></FONT><BLOCKQUOTE><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Not really, but I would consider trying downloading the Mesa source and compiling it myself. Since the error is happening somewhere in the OpenGL libraries (and is not really being replicated on other user’s platforms), the only cause I can come up with is a problem with the libraries (and mismatched libraries is the most common problem). Compiling Mesa is actually pretty easy, so it is worth taking the effort to check.<BR>
<BR>
-Ken<BR>
<BR>
<BR>
<BR>
On 10/21/09 12:54 AM, "myaccountmail9 myaccountmail9" <<a href="myaccountmail9@gmail.com">myaccountmail9@gmail.com</a> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> > wrote:<BR>
<BR>
Hi Ken,<BR>
thank you for your reply.<BR>
No, we didn't compile OSMesa ourselves, but we installed OSMesa library on each node downloading the .deb packages with:<BR>
<BR>
apt-get install libosmesa6<BR>
apt-get install libosmesa6-dev<BR>
<BR>
Have you got any other idea?<BR>
Thank you again.<BR>
Luis<BR>
<BR>
<BR>
2009/10/20 Moreland, Kenneth <<a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> ><BR>
</SPAN></FONT><BLOCKQUOTE><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Could it be a problem with mismatched libraries? I notice in the error below that it occurs in /usr/lib/libGL.so.1. This is a system library. Did you compile OSMesa yourself? If so, then you should not be using the system GL libraries; you should be using the libraries that you compiled when you also generated the OSMesa library.<BR>
<BR>
-Ken<BR>
<BR>
<BR>
<BR>
On 10/20/09 1:56 AM, "myaccountmail9 myaccountmail9" <<a href="myaccountmail9@gmail.com">myaccountmail9@gmail.com</a> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> <<a href="http://myaccountmail9@gmail.com">http://myaccountmail9@gmail.com</a>> > wrote:<BR>
<BR>
Hi,<BR>
we have a Linux (Debian) cluster without gpu and we have a big problem using Paraview 3.6.1in client - server mode.<BR>
The server has been compiled with OSMesa library and following the instructions reported here (in particular the section "OSMesa support"):<BR>
<BR>
<a href="http://www.itk.org/Wiki/Setting_up_a_ParaView_Server">http://www.itk.org/Wiki/Setting_up_a_ParaView_Server</a><BR>
<BR>
If we try to connect the client started with the command:<BR>
<BR>
paraview<BR>
<BR>
and the server, started with the command:<BR>
<BR>
pvserver --use-offscreen-rendering<BR>
<BR>
we must disable the checkbox "Remote rendering threshold" (but this causes bad performances) in the client settings (Edit -> Settings -> Render view -> Server), othervise Paraview crashes after loading a big input or performing some operation such us changing "outline" to "wireframe"...<BR>
<BR>
The error always is (similar to):<BR>
<BR>
ParaView3/Servers/Common/vtkServerConnection.cxx, line 67<BR>
vtkServerConnection (0x3182220): Server Connection Closed!<BR>
<BR>
Executing with: 0<BR>
*** Process received signal ***<BR>
Signal: Segmentation fault (11)<BR>
Signal code: (128)<BR>
Failing at address: (nil)<BR>
[ 0] /lib/libpthread.so.0 [0x7f4fc6536a80]<BR>
[ 1] /usr/lib/libGL.so.1(glBlendFuncSeparateEXT+0x16) [0x7f4fc7bc12e6]<BR>
*** End of error message ***<BR>
Segmentation fault<BR>
<BR>
<BR>
This error also occurs if we start client and server logging directly (without any ssh connection) on the master node; on master node X is running and we can execute the test:<BR>
<BR>
mpirun -np 4 /usr/X11R6/bin/glxgears<BR>
<BR>
So, it doen't seem a problem regarding X connection and permission...<BR>
<BR>
Moreover, executing the test:<BR>
<BR>
/ParaView/ParaView-bin$ ctest -I ,,10<BR>
<BR>
we obtain some segmentation fault, but the log file can't give any useful information.<BR>
<BR>
111/475 Testing TestGradientBackground ***Exception: SegFault<BR>
121/475 Testing TestOSConeCxx ***Exception: SegFault<BR>
131/475 Testing TestTranslucentLUTDepthPeeling***Exception: SegFault<BR>
141/475 Testing BoxWidget ***Exception: SegFault<BR>
151/475 Testing TestParallelopipedWidget ***Exception: SegFault<BR>
<BR>
..<BR>
<BR>
On our local machine with gpu 100% test passed, and we haven't any problem to connect client and server on that machine.<BR>
<BR>
<BR>
MPI version is: mpirun (Open MPI) 1.2.7rc2<BR>
OSMesa version is: 7.0.3-7<BR>
<BR>
<BR>
We searched on google, on Paraview wiki... without any succes: can you help us please?<BR>
<BR>
If you need more information let us now<BR>
<BR>
Thank you very much<BR>
<BR>
Luis<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
</SPAN></FONT><FONT SIZE="2"><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'><BR>
**** Kenneth Moreland<BR>
*** Sandia National Laboratories<BR>
*********** <BR>
*** *** *** email: <a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <BR>
** *** ** phone: (505) 844-8919<BR>
*** web: <a href="http://www.cs.unm.edu/~kmorel">http://www.cs.unm.edu/~kmorel</a> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <BR>
</SPAN></FONT></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
<BR>
</SPAN></FONT><FONT SIZE="2"><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'><BR>
**** Kenneth Moreland<BR>
*** Sandia National Laboratories<BR>
*********** <BR>
*** *** *** email: <a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <BR>
** *** ** phone: (505) 844-8919<BR>
*** web: <a href="http://www.cs.unm.edu/~kmorel">http://www.cs.unm.edu/~kmorel</a> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <BR>
</SPAN></FONT></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
<BR>
</SPAN></FONT><FONT SIZE="2"><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'><BR>
**** Kenneth Moreland<BR>
*** Sandia National Laboratories<BR>
*********** <BR>
*** *** *** email: <a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <BR>
** *** ** phone: (505) 844-8919<BR>
*** web: <a href="http://www.cs.unm.edu/~kmorel">http://www.cs.unm.edu/~kmorel</a> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <BR>
</SPAN></FONT></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT><FONT SIZE="1"><FONT FACE="Monaco, Courier New"><SPAN STYLE='font-size:7pt'><BR>
**** Kenneth Moreland<BR>
*** Sandia National Laboratories<BR>
*********** <BR>
*** *** *** email: <a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <BR>
** *** ** phone: (505) 844-8919<BR>
*** web: <a href="http://www.cs.unm.edu/~kmorel">http://www.cs.unm.edu/~kmorel</a> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <BR>
</SPAN></FONT></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT><FONT SIZE="2"><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'><BR>
**** Kenneth Moreland<BR>
*** Sandia National Laboratories<BR>
*********** <BR>
*** *** *** email: <a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <BR>
** *** ** phone: (505) 844-8919<BR>
*** web: <a href="http://www.cs.unm.edu/~kmorel">http://www.cs.unm.edu/~kmorel</a> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <BR>
</SPAN></FONT></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT><FONT SIZE="2"><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'><BR>
**** Kenneth Moreland<BR>
*** Sandia National Laboratories<BR>
*********** <BR>
*** *** *** email: <a href="kmorel@sandia.gov">kmorel@sandia.gov</a> <<a href="http://kmorel@sandia.gov">http://kmorel@sandia.gov</a>> <BR>
** *** ** phone: (505) 844-8919<BR>
*** web: <a href="http://www.cs.unm.edu/~kmorel">http://www.cs.unm.edu/~kmorel</a> <<a href="http://www.cs.unm.edu/%7Ekmorel">http://www.cs.unm.edu/%7Ekmorel</a>> <BR>
</SPAN></FONT></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT><FONT SIZE="1"><FONT FACE="Monaco, Courier New"><SPAN STYLE='font-size:7.5pt'><BR>
**** Kenneth Moreland<BR>
*** Sandia National Laboratories<BR>
*********** <BR>
*** *** *** email: <a href="kmorel@sandia.gov">kmorel@sandia.gov</a><BR>
** *** ** phone: (505) 844-8919<BR>
*** web: <a href="http://www.cs.unm.edu/~kmorel">http://www.cs.unm.edu/~kmorel</a><BR>
</SPAN></FONT></FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
</SPAN></FONT>
</BODY>
</HTML>