<div dir="ltr">Well the issue with HDF5 as a vis format is paraview does not seem to be able to load it in parallel. Instead, it seems that paraview loads the whole file on rank 0 and then tries to broadcast to other processors. This would significantly limit the size of vis file ...</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, May 3, 2014 at 7:06 AM, Moreland, Kenneth <span dir="ltr"><<a href="mailto:kmorel@sandia.gov" target="_blank">kmorel@sandia.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>Reason 1 applies just as much to Vis as restart. More so as you usually do Vis on a different number of processors than the sim. </div>
<div><br>
</div>
<div>You may want to rethink reason 2. It may not seem like much now, but restarts can take a significant proportion of the run time. And ironically the longer they take, the more you have to run them (there is a whole theory behind that). </div>
<div><br>
</div>
<div>At any rate, lots do smart people have worked (and continue to work) on making io libraries like HDF5 fast. In general, I would expect HDF5 to be much faster than the VTK formats. <div class=""><br>
<br>
<div>-Ken</div>
<div><br>
</div>
Sent from my iPad so blame autocorrect.</div></div><div><div class="h5">
<div><br>
On May 2, 2014, at 10:35 PM, "Mohammad Mirzadeh" <<a href="mailto:mirzadeh@gmail.com" target="_blank">mirzadeh@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">I don't have any specific plan for that but the rationale for using HDF5 for restart is twofold:
<div><br>
</div>
<div>1) The restart file could be read later by a different set of processors and preferably include useful meta information about the run (date/time, SHA1 of git commit, run parameters, etc)</div>
<div>2) The restart file is assumed to be written less frequently compared to vis and thus performance loss should not be a big issue (hopefully)</div>
<div><br>
</div>
<div>Also, parallel loading of vis file is necessary as ParaView seems to default to loading everything on rank 0 which would severely limit the size of vis file (I'd be happy to be proven wrong on this one). All that said, I'm willing to move away from HDF5
if that proves to be too costly for restart files as well. It just seems to me, after two days of searching online, that working with parallel HDF5 (and MPI-IO in general) is tricky and subject to performance loss and large number of processors. (Again I'd
be happy to learn from others' experience here)</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, May 2, 2014 at 7:12 PM, Moreland, Kenneth <span dir="ltr">
<<a href="mailto:kmorel@sandia.gov" target="_blank">kmorel@sandia.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
What are you doing for your restart files? You said those are HDF5 and they must be at least as large as anything you output for Vis. Presumably you got that working pretty well (or are committed to getting it to work well). Why not write the Vis output similarly?<br>
<br>
-Ken<br>
<br>
Sent from my iPad so blame autocorrect.<br>
<div>
<div><br>
> On May 2, 2014, at 6:50 PM, "Mohammad Mirzadeh" <<a href="mailto:mirzadeh@gmail.com" target="_blank">mirzadeh@gmail.com</a>> wrote:<br>
><br>
> Hi I am at a critical point in deciding I/O format for my application. So far my conclusion is to use parallel HDF5 for restart files as they are quite flexible and portable across systems.<br>
><br>
> When it comes to visualization, however, i'm not quite sure. Up until now I've been using pvtu along with vtu files and although they generally work fine, one easily gets in trouble when running big simulations on large number of processors as the number
of files can easily get out of control and even simplest utility commands (e.g. ls) takes minutes to finish!<br>
><br>
> After many thinking I've come to a point to decide between two strategies:<br>
><br>
> 1) Go with a single parallel HDF5 file that includes data for all time-steps. This makes it all nice and portable except there are two issues. i) It looks like doing MPI-IO might not be as efficient as separate POSIX IO, especially on large number of processors.
ii) ParaView does not seem to be able to read HDF5 files in parallel<br>
><br>
> 2) Go with the same pvtu+vtu strategy except take precautions to avoid file explosions. I can think of two strategies here: i) use nested folders to separate vtu files from pvtu and also each time step ii) create an IO group communicator with much less processors
that do the actual IO.<br>
><br>
> My questions are 1) Is the second approach necessarily more efficient than MPI-IO used in HDF5? and 2) Is there any plan to support parallel IO for HDF5 files in paraview?<br>
</div>
</div>
> _______________________________________________<br>
> Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
><br>
> Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
><br>
> Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">
http://paraview.org/Wiki/ParaView</a><br>
><br>
> Follow this link to subscribe/unsubscribe:<br>
> <a href="http://www.paraview.org/mailman/listinfo/paraview" target="_blank">http://www.paraview.org/mailman/listinfo/paraview</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div></div></div>
</blockquote></div><br></div>