为什么Matlab parfor调度程序让工作人员闲置？(Why is the Matlab parfor scheduler leaving workers idle?)

为什么Matlab parfor调度程序让工作人员闲置？(Why is the Matlab parfor scheduler leaving workers idle?)

我有一个相当长时间运行的parfor循环（让我们说100,000次迭代，每次迭代大约需要一分钟），我运行36个核心。 我注意到，在工作即将结束时，大量核心处于闲置状态，而一些核心完成了我认为每个工作者多次迭代的工作。 这导致大量浪费的计算时间等待一名工人完成几项工作而其他人闲置。

以下脚本显示了该问题（使用文件交换实用程序Par.m ）：

% Set up parallel pool nLoop = 480; p1 = gcp; % Run a loop pclock = Par(nLoop); parfor iLoop = 1:nLoop Par.tic; pause(0.1); pclock(iLoop) = Par.toc; end stop(pclock); plot(pclock); % Process the timing info: runs = [[pclock.Worker]' [pclock.ItStart]' [pclock.ItStop]']; nRuns = arrayfun(@(x) sum(runs(:,1) == x), 1:max(runs)); starts = nan(max(nRuns), p1.NumWorkers); ends = nan(max(nRuns), p1.NumWorkers); for iS = 1:p1.NumWorkers starts(1:nRuns(iS), iS) = sort(runs(runs(:, 1) == iS, 2)); ends(1:nRuns(iS), iS) = sort(runs(runs(:, 1) == iS, 3)); end firstWorkerStops = min(max(ends)); badRuns = starts > firstWorkerStops; nBadRuns = sum(sum(badRuns)) - (p1.NumWorkers-1); fprintf('At least %d (%3.1f%%) iterations run inefficiently.\n', ... nBadRuns, nBadRuns/nLoop * 100);

我正在看它的方式，每个工人都应该忙，直到队列空了，之后所有工人都闲着。 但是在这里看起来并没有发生 - 通过480次迭代，我得到了6-20次迭代，这是在一名工人在一个不同的工作人员闲置一整个周期之后开始的 。 该数字似乎与循环迭代次数呈线性关系，接近总数的2％。 在有限的测试中，这似乎在Matlab 2016b和2014b中保持一致。

是否有任何理由认为这是预期的行为，或者这只是parfor实现中写得不好的调度程序？ 如果是这样的话，我该如何构建这个，所以我不会和闲散的工人坐在一起这么久？

I have a fairly long-running parfor loop (let's say 100,000 iterations where each iterations takes about a minute) that I'm running with 36 cores. I've noticed that near the end of the job, a large number of the cores go idle while a few finish what I think has to be multiple iterations per worker. This leads to a lot of wasted computing time waiting for one worker to finish several jobs while the others are sitting idle.

The following script shows the issue (using the File Exchange utility Par.m):

% Set up parallel pool nLoop = 480; p1 = gcp; % Run a loop pclock = Par(nLoop); parfor iLoop = 1:nLoop Par.tic; pause(0.1); pclock(iLoop) = Par.toc; end stop(pclock); plot(pclock); % Process the timing info: runs = [[pclock.Worker]' [pclock.ItStart]' [pclock.ItStop]']; nRuns = arrayfun(@(x) sum(runs(:,1) == x), 1:max(runs)); starts = nan(max(nRuns), p1.NumWorkers); ends = nan(max(nRuns), p1.NumWorkers); for iS = 1:p1.NumWorkers starts(1:nRuns(iS), iS) = sort(runs(runs(:, 1) == iS, 2)); ends(1:nRuns(iS), iS) = sort(runs(runs(:, 1) == iS, 3)); end firstWorkerStops = min(max(ends)); badRuns = starts > firstWorkerStops; nBadRuns = sum(sum(badRuns)) - (p1.NumWorkers-1); fprintf('At least %d (%3.1f%%) iterations run inefficiently.\n', ... nBadRuns, nBadRuns/nLoop * 100);

The way I'm looking at it, every worker should be busy until the queue is empty, after which all workers sit idle. But here it looks like that's not happening - with 480 iterations, I'm getting between 6-20 iterations that start on a worker after a different worker has been sitting idle for a full cycle. This number appears to scale linearly with the number of loop iterations coming in near 2% of the total. With limited testing this appears to be consistent across Matlab 2016b and 2014b.

Is there any reason that this is the expected behavior or is this simply a poorly written scheduler in the parfor implementation? If so, how can I structure this so I'm not sitting around with idle workers so long?

最满意答案

parfor调度程序尝试对循环进行负载平衡，其中迭代不会花费相同的时间。 不幸的是，正如您所观察到的，这可能导致工作人员在循环结束时变得闲置。 有了parfor ，你无法控制工作分工; 但你可以使用parfeval将你的工作分成均匀的块 - 这可能会让你更好地利用。 或者，您甚至可以将spmd与for-drange循环结合使用。

The parfor scheduler attempts to load-balance for loops where the iterations do not take a uniform amount of time. Unfortunately, as you observe, this can lead to workers becoming idle at the end of the loop. With parfor, you don't have control over the division of work; but you could use parfeval to divide your work up into even chunks - that might give you better utilisation. Or, you could even use spmd in conjunction with a for-drange loop.