[Paraview] [EXTERNAL] Memory explosion and strange behavior -- Linux -- 'clip' -- 360 MB file needs 60 GB ??

[Robert.Atwood at diamond.ac.uk]

Aren't you glad I already reduced the data to 8bit grayscale and clipped out only 850x850x500 out of 4008x4008x2672 32-bit floating point values??? They keep building bigger cameras ...

Did I mention it is part of a time-series? 


>Some of the greatest minds in the world are still trying to figure out how to visualize data of this size.

Isn't that you guys? That's why I've tried Paraview! :-)  or is it Ajay?

There is indeed a cluster lurking, once we figure out how to configure it. It has several (50?)  GPU nodes too. That's how we get such images reconstructed :-) 



The work flow 
 downsample whole volume ->view downsample to choose VOI -> select VOI->process VOI->apply processing required on cluster to whole data->go home->render-> go home again 
is what I have in mind. 



 
 





________________________________________
From: Scott, W Alan [wascott at sandia.gov]
Sent: Monday, June 02, 2014 6:11 PM
To: Atwood, Robert (DLSLtd,RAL,SCI); paraview at paraview.org
Subject: RE: [EXTERNAL] [Paraview] Memory explosion and strange behavior -- Linux -- 'clip' -- 360 MB file needs 60 GB ??

Not replying to your specific questions below, but ... you have some pretty large data here!  It is amazing how a fairly small cell count on the edge of your data (for instance X axis) can become huge when you add a Y axis and Z axis.

You have about 375 million cells.  If you were able to volume render that on a local server, one machine, I believe you have the world's record!  Wow.

Generally speaking, I sort of tell people that anything around thousands of cells is a toy, a million is real data (and you should start thinking about using a cluster for a back end), and a billion is hero size and you want some heavy iron to deal with it.  Tens of billions, and especially trillions, pushes state of the art.  If you are volume rendering, divide any of these numbers by 10.

By the way, exascale will be in the trillions of cells.  Some of the greatest minds in the world are still trying to figure out how to visualize data of this size.

Alan

-----Original Message-----
From: ParaView [mailto:paraview-bounces at paraview.org] On Behalf Of Robert.Atwood at diamond.ac.uk
Sent: Monday, June 02, 2014 9:42 AM
To: paraview at paraview.org
Subject: [EXTERNAL] [Paraview] Memory explosion and strange behaviour -- Linux -- 'clip' -- 360 MB file needs 60 GB ??

Hi,
I like volume viewing in Paraview 4.1.0 so far!

But, when I try to use even what I consider a rather small subset of one image file that I work with normally, the large memory on my large memory workstation still gets used up.

I am using Red Hat 6 on a system with 100 GB of Ram, adn a Quadro 6000 graphics card. I have compiled the Paraview code that I downloaded on Friday.

The image I loaded is saved as raw unsinged 8-bit and is 860x872x501 voxels i.e. about 360 MB file. If I load it and then try to apply the 'clip' tool, the system is unresponsive for a few minutes. If I run 'top ' during this process I see the memory in use expand up to 60 GB , this seems excessive!

Then, the resulting display behaves oddly, the clipped view suddenly 'vanishes' after moving the view around ??



--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom




_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView

Follow this link to subscribe/unsubscribe:
http://www.paraview.org/mailman/listinfo/paraview

-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom