How to use HPC resources

[szaghi]

Dear All,

it is time to decide how to use our grant.

As I said we have

• 250.000 core-hours on Fermi, CINECA, Italy, a BlueGene/Q system
• 100.000 core-hours on MareNostrum, BSC, Spain, a Intel Sandy Bridge cluster

We will have 6 months to use these resources (I think starting from the date when we activate our accounts).

I am preparing a 2D (hopefully 3D) FOODIE test that I hope will be able to exploit the above resources. I have not a direct experience on MareNostrum, whereas I have used Fermi: the minimum-size job is about thousand of cores (in debug-smallpar queues, that are suitable for our scaling tests, should be smaller), thus the tests could not be too small. In particular we 1GB/core of RAM on Fermi and [2,4,8]GB/core on MareNostrum.

I would like to know if you (all of you, not only the ones directly involved into the paper for example) have some proposal on how to use these resources at best.

In the following a brief summary of the tests I am preparing is reported, I am sorry, but my background is gas-dynamic, I am essentially an inviscid (sub)man 😄

2D Riemann Problem(s)

Classical tests, as reported in the interesting Kurganov and Tadmor paper, e.g.

Shock Diffraction

Very classical test, e.g. Mach 2 shock diffraction on 90° corner, e.g.

Wind Tunnel Step

Even more classical ( 😄 ), the Mach 3 flow into a stepped wind tunnel, e.g.

[zbeekman]

@szaghi: So you're proposing some gas-dynamic inviscid test cases? That sounds good to me! In my experience, the viscous terms do complicate the physics (turbulence etc.) but numerically, the inviscid terms are often the most difficult to compute (expense) and get right from a stability and boundary condition perspective.

Are you thinking of using OFF or some other finite volume code/framework? I don't think the curve-linear finite difference codes I use will be able to handle the sharp corners of the geometries you have proposed, so a FV code is probably better.

[szaghi]

@zbeekman
Yes, I am of the same opinion: inviscid cases should be sufficient for benchmarking performance without introducing other side effects 😄

No, OFF is currently freeze (Indeed I was warned of not wasting my time on it... I have only my free/launch break projects...). I am finishing a very simple Finite-Volume Structured Block framework that should be soon ready for the above tests. It is just an helper for FOODIE to handle finite volume mesh.

[milancurcic]

@szaghi @zbeekman I'm curious about the choice of the test problem. We want to test the scalability of the FOODIE implementation of dU/dt = R(U,t). I thought that we aim to minimize the workload and complexity in R(U,t) in order to maximize the workload in dU/dt, which is what FOODIE is implementing. Please correct me if I'm wrong - isn't it advantageous to select the simplest possible problem R(U,t), like a periodic 2-d burgers equation or just simply linearized advection of a Gaussian blob with periodic BCs?

My arguments for using the simplest possible problem are:

• Minimize implementation time for R(U,t) so we have more time for scaling analysis;
• Minimize computational load in R(U,t) so we can do more tests.

Or am I missing something? Maybe these tests you propose are very simple, they just don't look simple to me! :)

[zbeekman]

@milancurcic Yes, actually I was just starting to have similar thoughts after I posted my first hasty response. If we make the workload in R(U,t) too large, we'll miss out on the scaling properties of FOODIE itself. However, the large RHS type of work load is more representative of what we're building FOODIE for.

Here is what I propose: Let's start with some more minimalist RHS test cases like 2-D Burgers and linearized advection, as suggested by @milancurcic. If we are dissatisfied with those results we can then move to a more realistic and computationally intensive RHS, to argue that FOODIE, even though it may not scale as well as we had hoped (if that is the case) it is still of great utility because in a typical problem/workload the RHS takes up the largest fraction of clock cycles and the design pattern helps reduce the complexity and development time of the code, which are also expenses in and of themselves.

Let's start with small RHS workloads to examine the computational efficiency and scaling. Then if we have leftover hours, or if we find less than ideal scaling results, move to something beefier, like the compressible Euler equations.

[szaghi]

@milancurcic @zbeekman

I agree, but some of my arguments are the following:

• reducing the R(U,t) complexity too much could be not desiderable because the test becomes too far from a production case;
• yes, Compressible Euler is more complex than Burgers, but not so much, it should take me 10/15 lunch breaks to have a working 2D Riemann test of FOODIE, it depends on my actual hunger 😄
• I planned to complete such simulations few times, for benchmarking it could be enough few time steps integration.

This said, I totally agree to start from the simplest test. My Finite volume framework is 90% working and it is abstract 😄 it should be used with any conservative PDE systems (indeed, it started as a secret present for @milancurcic who asked for a BVP solver 😄 ) Burgers included.

At anyrate, feel free to propose any test/approach/implementation, indeed for me is more interesting to simulate something new with respect my background.

See you soon.

[rouson]

On Jan 22, 2016, at 11:55 AM, Stefano Zaghi [email protected] wrote:

This said, I totally agree to start from the simplest test. My Finite volume framework is 90% working and it is abstract it should be used with any conservative PDE systems (indeed, it started as a secret present for @milancurcic https://github.com/milancurcic who asked for a BVP solver ) Burgers included.

Are you referring to [OFF](https://github.com/szaghi/OFF <https://github.com/szaghi/OFF)? Does it solve 3D problems? the examples folder only mentions 2D.

I just added myself to the Watch list. I’m unlikely to make direct use of a GPL project, but I think it’s great that you’re doing it and I hope it’s successful. We need a modern Fortran competitor to OpenFOAM. Now the you’ve experienced coarray Fortran via FOODIE, I hope you’ll consider using it in OFF. It could be even more important there because the one-sided communication model (and other features we’re developing) might offer speedup — emphasizing “might” — depending on the MPI implementation and the hardware.

Damian

[milancurcic]

This said, I totally agree to start from the simplest test. My Finite volume framework is 90% working and it is abstract 😄 it should be used with any conservative PDE systems (indeed, it started as a secret present for @milancurcic who asked for a BVP solver 😄 ) Burgers included.

@szaghi You always make me 😄!

[szaghi]

@rouson

OFF is able to simulate 3D flow (I used it in production for the simulation of ignition transient of Solid Rocket Motors in the framework of my Ph.D., there are not 3D examples because I was lazy), but as I said I am not allowed to develop it (it is conflicting with my official work). I am currently allowed to spend only my free time on not so conflicting projects, thus I decided to face the problem (solving Navier-Stokes) from a different point of view: small, kiss libraries solving a small piece of the whole problem, e.g. FOODIE. So OFF development is currently freeze. Indeed I was very sad when my boss stped me, but as it often happens, from a bad thing new opportunities come, and now I feel that FOODIE & Co. are better than the monolitic OFF (with the good pace of my boss).

The framework I was talking about is FriVolous: this GitHub repository contains just a skeleton, my local version is very far from it. I essentially exploits your idea of ACP (I sorry, but your book change my nerdy life 😄 ): the framework helps you to solve BVP by a finite volume approach (3D general curvilinear metrics like OFF ones) where it handles the metrics for you. The key point is that you extends an abstract conservative type and its abstract conservative flux. It does for space what FOODIE does for time, but in this case I need 2 ADT not one. I hope to give you some example the next week.

See you soon.

P.S. @milancurcic pay attention on what you ask 😄

[rouson]

Sounds exciting. I look forward to seeing more. And I'm glad the book helped! :)

[milancurcic]

@rouson I think your book has attracted a cult following, myself included. I wish I learned about it sooner than few months ago!

[rouson]

You guys are making my day! Thanks tons for the great feedback. There is so much that I wish I could have done differently, but I wrote what I could at the time. I really hope to find the time to write another book and I imagine it being better in so many ways. It's really encouraging to know that the first one helped people.

[szaghi]

@rouson no doubt! Your book is a bomb: I am very glad to move my Institute to by a copy of it.

Your main merit has been demostrating the usefulness of OOD in scientific application. What we have done in FOODIE in so few time (really breaks lunch) could be very difficult and time consuming without abstraction in a monolitic-nonOO approach.

Be proud of your book!

[rouson]

👍

[zbeekman]

@rouson I completely agree. (Have been kicking myself for not violently
disagreeing with you when you made a comment about your book being "bad" or
"not that good" on our call earlier.) I am in the cult of Rouson too!
On Fri, Jan 22, 2016 at 5:14 PM Damian Rouson [email protected]
wrote:

[image: 👍]

—
Reply to this email directly or view it on GitHub
#31 (comment)
.

[rouson]

You guys rock!!

Today was an absolutely horrible day (I found out a roughly three-month effort to close a deal completely unrelated to Fortran or science is likely to fail), which made this one of the best possible days for you to write the things you wrote.

I was on the other end of this experience just a few weeks ago when I had the opportunity to do something nice for a former mentor and it turned out the timing was perfect because of challenges he was facing at the time. As the old folks used to say when I was growing up, "What goes around comes around." So maybe this is karma. 😄