Flink Job 子任务需要几分钟才能启动，而源会立即启动

本文介绍了Flink Job 子任务需要几分钟才能启动，而源会立即启动的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

限时送ChatGPT账号..

所以我最近将我的 flink 作业的并行度从 10 增加到 50.目前设置是 25 个任务管理器，每个任务管理器有 2 个任务槽，每个有 2 个 cpu.具有 5 个 CPU 的作业管理器.但是出于某种原因，当作业开始时，它能够立即启动两个源的所有子任务，但作业的其余部分需要一分钟或更长时间才能启动所有其他操作员的子任务.

So I recently increased the parallelism of my flink job from 10 to 50. Currently the setup is 25 taskmanagers with 2 task slots each, each with 2 cpu. Job manager with 5 cpus. For some reason though, when the job starts, it is able to start up all the sub tasks for the two sources immediately, but the rest of the job takes a minute or more to start all the other operators' subtasks.

我知道这不应该发生，但我不确定需要更改什么才能解决此问题.这是 ui 中任务管理器配置之一的屏幕截图.它也是使用rocksdb和s3进行检查点.

I know this shouldn't be happening, but I am not sure what needs to be changed to resolve this. Here is a screen shot of one of the task manager's configuration from the ui as well. Also it is checkpointing with rocksdb and using s3.

推荐答案

所以答案是它与基础设施设置有关.

So the answer to this is that it was related to the infrastructure setup.

tl;dr:RocksDB 位于通过网络共享的卷上.

The tl;dr: RocksDB was on a shared volume that was over network.

因此，要详细说明到底出了什么问题(并将深入解释归功于该项目的基础架构团队成员)，RocksDB 位于数据中心的共享卷上.在幕后，提供商将 NFS 用于共享卷，而 NFS 不太适合具有大量事务和 IO 的工作负载，例如 RocksDB 的工作负载.写入 NFS 卷需要锁定，多个 Pod 写入同一个 NFS 卷可能会在请求锁定的 Pod 之间发生争用.

So to break down a bit on exactly what was wrong (and credit for the in depth explanation goes to the infrastructure team member on the project) is the RocksDB was on a shared volume in the datacenter. And under the covers the provider uses NFS for shared volumes and NFS is not a great match for a workload with a high amount of transactions and IO such as that of the RocksDB. Writes to an NFS volume require locks and multiple pods writing to the same NFS volume may run into contention between the pods requesting a lock.

因此，我们不再使用 PVC，而是使用具有大小限制的 emptyDir 卷切换到每个 Pod 的临时存储，这样我们就不必担心 Pod 会因填充底层节点而意外导致中断.

So we moved away from using a PVC and instead switched to per-pod ephemeral storage using the emptyDir volume with a size limit so we don’t have to worry about a pod accidentally causing outages by filling up the underlying node.

这也解决了我在这个问题中的另一个问题，因为它基本上是同一问题的不同症状(Flink Checkpoint 完成时间增长，尽管没有处理新数据).

And this is also fixed my other problem in this question, as it was basically a different symptom of the same issue (Flink Checkpoint completion time growing despite no new data being processed).

不确定这是否应该有更多/不同的标签(或那些标签)，因为这不完全是 flink 问题.

Not sure if this should have more/different tags (or what those would be) since it wasn't exactly a flink issue.

这篇关于Flink Job 子任务需要几分钟才能启动，而源会立即启动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！