Question: Which software do you use for data analysis?

Hey guys!

This is more like an "open" question to see how others analyse their data, which they get from DualSPHysics simulations. So basically some questions I find interesting would be:
  1. Do you use binary data directly or export to csv / vtk?
  2. Do you use spread sheet programs (Excel etc.) or something like Matlab/Python etc.?
  3. What is your biggest challenge with handling data? (Storage, reading speed etc. maybe something else?
I hope a lot of you, want to share a bit of their workflow, so I might be able to optimize my own a bit more. Personally I am still primarily using Excel and Paraview, but I feel there should be a more efficient way.

Kind regards


  • 1. I use VTK for visualization. If I sample certain data (such as velocity) then I export it as .csv. And then I use Paraview for VTK.
    2. I use mostly excel, since my .CSV files differ marginally each time. I could write a script, but deemed it not worthy (would not win any time).
    3. Setting up the sampling of the raw data, so slightly changing the .sh (script) file which handles the inbuilt functions such as IsoSurfaces and MeasureTool. And mostly modifying the supporting files such as Fileboxes.txt or VelocityPoints.txt (sampling locations).
  • Hello !
    Excellent question !

    1. I use .csv and .vtk with paraview for quick visualization and with blender for nice rendering.
    2. I switch from excel to Python recently. It makes all the difference for me. I tend to make a few simulations with only one parameter changing (flow for exemple) so the script approach is very usefull to treat the data and then do some science with it. But even with a single simulation, I would go with Python, because it is a better and more reliable way to treat or explore data (IMO) (also if you want to modify your work you have a history and can easily do so).
    3. Some measuretool export can be very long (more than an hour with i5-8600 (3.1 GHz))... Also, I still haven't managed to make my own export function for the binary data. I would want to add some visualization option...

    Kind regards
  • Thanks @bladomas1 and @TPouzol ! Very exciting to hear your perspectives, especially since you represent both sides regarding using spreadsheets or programming languages. I have some further questions to both of you which I hope you would like to answer:

    For bladomas1
    1. For your answer to question 3., could you explain what you mean with sampling? Are you talking about exporting csv files for specific locations inside your simulation domain? Like velocity at point x,y,z?
    2. If this is the case then; is the problem finding the exact location to probe?l
    For TPouzol
    1. How does the Python script you made work? Does it read all the arrays you want from the csv file and then you do the math you want to do?
    2. You say you are interested in making your own export of a binary file, what is your problem regarding this right now?
    Kind regards
  • 1. yes, I import .csv files and then do all the math I want
    2. I'd say time and "fear". As I'm self taught in code, it still is a bit daunting to compile my tool. However, I intend to remedy this problem soon.
  • Hey @Asalih3d,
    1. Exactly, setting up the specific locations for exporting data (I call it sampling). So for velocity I provide a list of points for data gathering (X,Y,Z). For measuretool boxes have to be drawn to gather the data (specify 8 points X Y Z).
    2. I have often slightly different setups (simulations) which means I cannot gather the data using the old probe locations, and need to write new ones. This costs each time some time. I did consider writing a python script that reads the Def.xml file and creates the appropriate probe locations, but since my project scope is quite limited it was not worth the effort.

    I would still agree with @TPouzol that Python is better, but only if you work on the project for a long time. Then using scripts can save you time. For me however, I wanted quick results and had varying simulations, so using Excel was optimal.
  • Thanks for the input guys! I am trying to develop a tool which makes it possible to only export the vtk's and extract data from them programmatically, instead of having to export csv - in this way lessening the storage space. If you had a tool which could programmatically load in like mass, acceleration, velocity etc. and then you could use these arrays for computation, would that be of interest? Without having the need of using CSV?

    Kind regards
Sign In or Register to comment.