我有以下设置，一个混合MPI / OpenMP代码，它运行M个MPI进程，每个进程有N个线程。 总共有MxN线程可用。
如果可能的话，我想做的是将线程仅分配给某些MPI进程而不是所有进程，我的代码会更高效，因为有些线程只是重复工作。
谢谢。
I have the following setup, a hybrid MPI/OpenMP code which runs M MPI processes with N threads each. In total there are MxN threads available.
What I would like to do, if possible, is to assign threads only to some MPI processes not to all of them, my code would be more efficient since some of the threads are just doing repetitive work.
Thanks.
最满意答案
你的问题是这个问题的一个通用版本。 至少有三种可能的解决方案。
对于大多数MPI实现，可以使用他们自己的环境（上下文）启动多个可执行文件作为相同MPI作业的一部分。 它被称为MPMD（多程序多数据）或MIMD（多指令多数据）模型。 语法通常包含:冒号）作为分隔符：
$ mpiexec <global parameters>
-n n1 <local parameters> executable_1 <args1> :
-n n2 <local parameters> executable_2 <args2> :
...
-n nk <local parameters> executable_k <argsk>
它使用命令行参数<args1>启动运行executable_1 n1等级，使用命令行参数<args1>启动运行executable_1 n2等等，依此类推。 总共n1 + n2 + ... + nk过程开始，并且线性分配等级：
Ranks (from .. to) | Executable
====================|=============
0 .. n1-1 | executable_1
n1 .. n1+n2-1 | executable_2
n1+n2 .. n1+n2+n3-1 | executable_3
... | ...
作为一个更窄的情况，同一个可执行文件可以指定k次，以便使用相同的可执行文件获得k个不同的上下文。 <local parameters>可能包括设置特定环境变量的值，例如在您的情况下可能是OMP_NUM_THREADS 。 指定环境的确切方法因实现而异。 通过Open MPI，人们可以这样做：
mpiexec --hostfile all_hosts \
-n 5 -x OMP_NUM_THREADS=2 myprog : \
-n 4 -x OMP_NUM_THREADS=4 myprog : \
-n 6 -x OMP_NUM_THREADS=1 myprog
这将在all_hosts指定的主机上启动15个MPI（全局参数），前5个使用两个OpenMP线程，接下来的四个OpenMP线程和最后六个按顺序运行。 在基于MPICH的实现中，命令会稍有不同：
mpiexec --hostfile all_hosts \
-n 5 -env OMP_NUM_THREADS 2 myprog : \
-n 4 -env OMP_NUM_THREADS 4 myprog : \
-n 6 -env OMP_NUM_THREADS 1 myprog
虽然得到广泛支持，但以前的方法有点不灵活。 如果人们想要例如所有的队伍，除了连续第10次跑？ 然后命令行变成：
mpiexec ...
-n 9 -x OMP_NUM_THREADS=1 myprog : \
-n 1 -x OMP_NUM_THREADS=N myprog : \
-n 9 -x OMP_NUM_THREADS=1 myprog : \
-n 1 -x OMP_NUM_THREADS=N myprog : \
...
更方便的解决方案是提供一个基于进程级别设置OMP_NUM_THREADS的包装器。 例如，Open MPI的这种包装看起来像：
#!/bin/bash
if [ $((($OMPI_COMM_WORLD_RANK + 1) % 10)) == 0 ]; then
export OMP_NUM_THREADS=N
else
export OMP_NUM_THREADS=1
fi
exec "$*"
并简单地用作：
mpiexec -n M ... mywrapper.sh myprog <args>
第三种也是最不灵活的选择是在MPI初始化之后但在任何并行区域之前从程序内简单地调用omp_set_num_threads() ，并根据等级设置不同数量的线程：
integer :: provided, rank, ierr
call MPI_INIT_THREAD(MPI_THREAD_FUNNELED, provided, ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
if (mod(rank, 10) == 0) then
call omp_set_num_threads(N)
else
call omp_set_num_threads(1)
end if
无论选择什么解决方案，进程和线程绑定都变得有点棘手，应该完全关闭。
Your question is a generalised version of this one. There are at least three possible solutions.
With most MPI implementations it is possible to start multiple executables with their own environments (contexts) as part of the same MPI job. It is called MPMD (Multiple Programs Multiple Data) or MIMD (Multiple Instructions Multiple Data) model. The syntax usually involves : (colon) as a separator:
$ mpiexec <global parameters>
-n n1 <local parameters> executable_1 <args1> :
-n n2 <local parameters> executable_2 <args2> :
...
-n nk <local parameters> executable_k <argsk>
It launches n1 ranks running executable_1 with command-line arguments <args1>, n2 ranks running executable_2 with command-line arguments <args2>, and so on. In total n1 + n2 + ... + nk processes are started and ranks are assigned linearly:
Ranks (from .. to) | Executable
====================|=============
0 .. n1-1 | executable_1
n1 .. n1+n2-1 | executable_2
n1+n2 .. n1+n2+n3-1 | executable_3
... | ...
As a more narrow case, the same executable could be specified k times in order to get k different contexts with the same executable. <local parameters> could include setting the values of specific environment variables, e.g. in your case that could be OMP_NUM_THREADS. The exact method to specify the environment differs from one implementation to another. With Open MPI, one would do:
mpiexec --hostfile all_hosts \
-n 5 -x OMP_NUM_THREADS=2 myprog : \
-n 4 -x OMP_NUM_THREADS=4 myprog : \
-n 6 -x OMP_NUM_THREADS=1 myprog
That will start 15 MPI ranks on the hosts specified in all_hosts (a global parameters) with the first five using two OpenMP threads, the next four - four OpenMP threads, and the last six running sequentially. With MPICH-based implementations the command would be slightly different:
mpiexec --hostfile all_hosts \
-n 5 -env OMP_NUM_THREADS 2 myprog : \
-n 4 -env OMP_NUM_THREADS 4 myprog : \
-n 6 -env OMP_NUM_THREADS 1 myprog
Although widely supported, the previous method is a bit inflexible. What if one would like e.g. all ranks except every 10-th run sequentially? Then the command line becomes:
mpiexec ...
-n 9 -x OMP_NUM_THREADS=1 myprog : \
-n 1 -x OMP_NUM_THREADS=N myprog : \
-n 9 -x OMP_NUM_THREADS=1 myprog : \
-n 1 -x OMP_NUM_THREADS=N myprog : \
...
A more convenient solution would be to provide a wrapper that sets OMP_NUM_THREADS based on the process rank. For example, such a wrapper for Open MPI looks like:
#!/bin/bash
if [ $((($OMPI_COMM_WORLD_RANK + 1) % 10)) == 0 ]; then
export OMP_NUM_THREADS=N
else
export OMP_NUM_THREADS=1
fi
exec "$*"
and is used simply as:
mpiexec -n M ... mywrapper.sh myprog <args>
The third and least flexible option is to simply call omp_set_num_threads() from within the program after MPI initialisation but before any parallel regions and set different number of threads based on the rank:
integer :: provided, rank, ierr
call MPI_INIT_THREAD(MPI_THREAD_FUNNELED, provided, ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
if (mod(rank, 10) == 0) then
call omp_set_num_threads(N)
else
call omp_set_num_threads(1)
end if
No matter what solution is chosen, process and thread binding becomes a bit tricky and should probably be switched off altogether.
更多推荐
发布评论