[Paraview-developers] Performance on a large volume rendered job

[Scott, W Alan]

G'Day all,
I am looking at the performance of a very large volume rendered fire.  My user has been using lots of nodes, and we wanted to understand if these nodes are needed.  Here is my result:

Dataset: 1296 files, 1.3 GByte each, 18 TBytes total, 67 million cells, 468 timesteps.  These viz runs are taking around 12 hours with a reasonable number of nodes.  When hitting play, TimerLog says each frame uses the following time:

2 nodes - died
4 nodes/ 64 cores and ranks -  79 seconds/frame, node memory 65% full
8 nodes/ 128 cores and ranks -  64 seconds/frame, node memory 41% full
16 nodes/ 128 cores and ranks -  59 seconds/frame, node memory 23% full
24 nodes/ 192 cores and ranks - 53 seconds/frame, node memory unknown.

For 16 nodes, timer log says we spent:
37 seconds - Regenerate Kd-Tree
15 seconds - Redistributing data
7 seconds - OpenGL Rendering

For 8 nodes, timer log says we spent:
39 seconds - Regenerate Kd-Tree
17 seconds - Redistributing data
13 seconds - OpenGL Rendering

So, we are getting scaling with increased memory, but not increased cpu.  More specifically, the Regenerate Kd-Tree is not speeding up as you halve each nodes work (which is a surprise), and ditto redistributing data.  OpenGL Rendering is linearly changing speeds.  Last, he majority of our time is regenerating the Kd-Tree.  What is this tree, why do we need it, what does it do?  I have had this dataset in a debugger, and frankly, it was really hard to figure out where time was being spent.  Any ideas how to proceed?  Is there something we could do to increase information from the timer log?

Thanks team!

Alan


--------------------------------------------------------
W. Alan Scott
ParaView Support Manager

SAIC
Sandia National Laboratories, MS 0822
Org 9326 - Building 880 A1-K
(505) 284-0932   FAX (505) 284-5619

The most exciting phrase to hear in science
is not "Eureka!" but "That's funny..."  -- Isaac Asimov
---------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview-developers/attachments/20170510/cde54a90/attachment.html>