Nix build for AdditiveFOAM

[wd15]

I have a Nix build for AdditiveFOAM and OpenFOAM 10 on this branch. It uses the ExaCA Nix build to build a combined AdditiveFOAM and ExaCA build. It has been tested for a number of AdditiveFOAM / ExaCA problems.

The current issue is that it is broken in parallel. I've had to substitute runApplication for runParellel in the test script.

Here is the log.additiveFoam file from running an example using runParallel. Can anyone give me some hints on how to debug this? My hunch is that OpenFOAM is not built correctly to run in parallel.

[wd15]

Current failure in parallel

--> FOAM FATAL ERROR: 
bool IPstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

    From function static bool Foam::UPstream::init(int&, char**&, bool)
    in file UPstream.C at line 108.

FOAM aborting

#0  Foam::error::printStack(Foam::Ostream&) at ??:?
#1  Foam::error::abort() at ??:?
#2  Foam::UPstream::init(int&, char**&, bool) at ??:?
#3  Foam::argList::argList(int&, char**&, bool, bool, bool) at ??:?
#4  ?addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
 addr2line failed
#5  ? in "/lib/x86_64-linux-gnu/libc.so.6"
#6  __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
#7  ?addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
 addr2line failed
[pippi:1197366] *** Process received signal ***
[pippi:1197366] Signal: Aborted (6)
[pippi:1197366] Signal code:  (-6)
[pippi:1197366] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7ffff4c42520]
[pippi:1197366] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7ffff4c969fc]
[pippi:1197366] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7ffff4c42476]
[pippi:1197366] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7ffff4c287f3]
[pippi:1197366] [ 4] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam5error5abortEv+0x2db)[0x7ffff56b9c9b]
[pippi:1197366] [ 5] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so(_ZN4Foam8UPstream4initERiRPPcb+0x3e9)[0x7ffff62efd19]
[pippi:1197366] [ 6] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7argListC1ERiRPPcbbb+0x758)[0x7ffff56d6248]
[pippi:1197366] [ 7] /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam(+0x408ce)[0x5555555948ce]
[pippi:1197366] [ 8] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7ffff4c29d90]
[pippi:1197366] [ 9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7ffff4c29e40]
[pippi:1197366] [10] /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam(+0x4df75)[0x5555555a1f75]
[pippi:1197366] *** End of error message ***


--> FOAM FATAL ERROR: 
bool IPstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

    From function static bool Foam::UPstream::init(int&, char**&, bool)
    in file UPstream.C at line 

--> FOAM FATAL ERROR: 
bool IPstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

    From function static bool Foam::UPstream::init(int&, char**&, bool)
    in file UPstream.C at line 108.

FOAM aborting

108.

FOAM aborting

#0  #0  Foam::error::printStack(Foam::Ostream&)Foam::error::printStack(Foam::Ostream&)

--> FOAM FATAL ERROR: 
bool IPstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

    From function static bool Foam::UPstream::init(int&, char**&, bool)
    in file UPstream.C at line 108.

FOAM aborting

#0  Foam::error::printStack(Foam::Ostream&)

--> FOAM FATAL ERROR: 
bool IPstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

    From function static bool Foam::UPstream::init(int&, char**&, bool)
    in file UPstream.C at line 108.

FOAM aborting

#0  Foam::error::printStack(Foam::Ostream&) at ??:?
#1  Foam::error::abort() at ??:?
#1  Foam::error::abort()

--> FOAM FATAL ERROR: 
bool IPstream::init(int& argc, char**& argv) : attempt to run parallel on 1 processor

    From function static bool Foam::UPstream::init(int&, char**&, bool)
    in file UPstream.C at line 108.

FOAM aborting

#0  Foam::error::printStack(Foam::Ostream&) at ??:?
#1  Foam::error::abort() at ??:?
#1  Foam::error::abort() at ??:?
#2  Foam::UPstream::init(int&, char**&, bool) at ??:?
#2  Foam::UPstream::init(int&, char**&, bool) at ??:?
#1  Foam::error::abort() at ??:?
#3  Foam::argList::argList(int&, char**&, bool, bool, bool) at ??:?
#3  Foam::argList::argList(int&, char**&, bool, bool, bool) at ??:?
#2  Foam::UPstream::init(int&, char**&, bool) at ??:?
#2  Foam::UPstream::init(int&, char**&, bool) at ??:?
#2  Foam::UPstream::init(int&, char**&, bool) at ??:?
#3  Foam::argList::argList(int&, char**&, bool, bool, bool) at ??:?
#3  Foam::argList::argList(int&, char**&, bool, bool, bool) at ??:?
#4  ? at ??:?
#4  ? at ??:?
#3  Foam::argList::argList(int&, char**&, bool, bool, bool)addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
 addr2line failed
#5  ?addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
 addr2line failed
#5  ? at ??:?
#4  ? at ??:?
#4  ? at ??:?
#4  ? in "/lib/x86_64-linux-gnu/libc.so.6"
#6  __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
#6  __libc_start_mainaddr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
 addr2line failed
#5  ? addr2line failed
#5  ?addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
 addr2line failed
#5  ? in "/lib/x86_64-linux-gnu/libc.so.6"
#7  ? in "/lib/x86_64-linux-gnu/libc.so.6"
#7  ? in "/lib/x86_64-linux-gnu/libc.so.6"
#6  __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
#6  __libc_start_mainaddr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
 in "/lib/x86_64-linux-gnu/libc.so.6"
 addr2line failed
#6  __libc_start_main[pippi:1197370] *** Process received signal ***
[pippi:1197370] Signal: Aborted (6)
[pippi:1197370] Signal code:  (-6)
[pippi:1197370] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7ffff4c42520]
[pippi:1197370] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7ffff4c969fc]
[pippi:1197370] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7ffff4c42476]
[pippi:1197370] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7ffff4c287f3]
[pippi:1197370] [ 4] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam5error5abortEv+0x2db)[0x7ffff56b9c9b]
[pippi:1197370] [ 5] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so(_ZN4Foam8UPstream4initERiRPPcb+0x3e9)[0x7ffff62efd19]
[pippi:1197370] [ 6] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7argListC1ERiRPPcbbb+0x758)[0x7ffff56d6248]
[pippi:1197370] [ 7] /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam(+0x408ce)[0x5555555948ce]
[pippi:1197370] [ 8] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7ffff4c29d90]
[pippi:1197370] [ 9] addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7ffff4c29e40]
[pippi:1197370] [10] /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam(+0x4df75)[0x5555555a1f75]
[pippi:1197370] *** End of error message ***
 addr2line failed
[pippi:1197369] *** Process received signal ***
[pippi:1197369] Signal: Aborted (6)
[pippi:1197369] Signal code:  (-6)
[pippi:1197369] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7ffff4c42520]
[pippi:1197369] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7ffff4c969fc]
[pippi:1197369] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7ffff4c42476]
[pippi:1197369] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7ffff4c287f3]
[pippi:1197369] [ 4] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam5error5abortEv+0x2db)[0x7ffff56b9c9b]
[pippi:1197369] [ 5] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so(_ZN4Foam8UPstream4initERiRPPcb+0x3e9)[0x7ffff62efd19]
[pippi:1197369] [ 6] /nix/store/a1zsvzszn5hanznf2yywckvq39sla0xk-openfoam-org-10/opt/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7argListC1ERiRPPcbbb+0x758)[0x7ffff56d6248]
[pippi:1197369] [ 7] /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam(+0x408ce)[0x5555555948ce]
[pippi:1197369] [ 8] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7ffff4c29d90]
[pippi:1197369] [ 9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7ffff4c29e40]
[pippi:1197369] [10] /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam(+0x4df75)[0x5555555a1f75]
[pippi:1197369] *** End of error message ***
 in "/lib/x86_64-linux-gnu/libc.so.6"
#7  ? in "/lib/x86_64-linux-gnu/libc.so.6"
#7  ? in "/lib/x86_64-linux-gnu/libc.so.6"
#7  ?addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
 addr2line failed
addr2line: /nix/store/y6vzcxr0cf8wqvf3lcrnawiaqqpj17x5-additivefoam-master/bin/additiveFoam: file format not recognized
[pippi:1197367] *** Process received signal ***
[pippi:1197367] Signal: Aborted (6)
[pippi:1197367] Signal code:  (-6)
[pippi:1197367] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7ffff4c42520]
[pippi:1197367] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7ffff4c969fc]
[pippi:1197367] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7ffff4c42476]
[pippi:1197367] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7ffff4c287f3]
[pippi:1197367] [ 4]  addr2line failed
[pippi:1197365] *** 

[gknapp1]

Hi Daniel, I was able to replace "runParallel" with "runApplication" and have the AMB2018 tutorial work without an error, so it may be something in the nix configuration.

A couple of ideas to troubleshoot:

1. Check that the OpenFOAM build is correct. From your nix build for OpenFOAM it appears to point to the OpenFOAM-12 repository, whereas it should point to the OpenFOAM-10 repo (https://github.com/OpenFOAM/OpenFOAM-10/).
2. If you are able to post the modified Allrun script that is being used, then that would be good to double check the replacements are behaving as expected. The error looks identical to when using "mpirun -np 1" for an OpenFOAM application with the "-parallel" flag. The number of processes has to be greater than one and equal to the number of subdomains specified in system/decomposeParDict. For example, to run the AMB20218 tutorial with MPI you would have to use:

mpirun -n 6 additiveFoam -parallel

[wd15]

Thanks @gknapp1. It actually is using OpenFOAM 10. It was just getting OpenFOAM 10 from the OpenFOAM 12 repository. I'm not sure why OpenFOAM uses separate repositories for major versions. Thanks for the subdomains reminder. That was causing some confusion. Anyway, it all seems to work now in serial and parallel. There were a number of issues that I fixed beyond those you raised. I've submitted the build as a PR.

[wd15]

I'm closing this as the Nix PR now works in parallel