我正在使用Zeppelin v0.7.3笔记本运行Pyspark脚本.在一个段落中,我正在运行脚本以将数据从dataframe写入Blob文件夹中的parquet文件.文件按国家/地区分区.数据帧的行数为99,452,829.脚本到达1 hour时,遇到错误-
I'm using Zeppelin v0.7.3 notebook to run Pyspark scripts. In one paragraph, I am running script to write data from dataframe to a parquet file in a Blob folder. File is partitioned per country. Number of rows of dataframe is 99,452,829. When the script reaches 1 hour, an error is encountered -
400 StatusCode错误:要求失败:会话未完成 活跃.
Error with 400 StatusCode: "requirement failed: Session isn't active.
我的笔记本电脑默认解释器是jdbc.我已阅读有关timeoutlifecyclemanager的信息,并将其添加到解释器设置zeppelin.interpreter.lifecyclemanager.timeout.threshold中并将其设置为7200000,但在达到1小时运行时间(33%处理完成)后仍然遇到错误.
My default interpreter for the notebook is jdbc. I have read about timeoutlifecyclemanager and added in the interpreter setting zeppelin.interpreter.lifecyclemanager.timeout.threshold and set it to 7200000 but still encountered the error after it reaches 1 hour runtime at 33% processing completion.
在1个小时的超时后,我检查了Blob文件夹,并将镶木地板文件成功写入了Blob,这些文件确实按国家/地区划分了.
I checked the Blob folder after the 1 hr timeout and parquet files were successfully written to Blob which are indeed partitioned per country.
我正在运行的将DF写入镶木地板Blob的脚本如下:
The script I am running to write DF to parquet Blob is below:
trdpn_cntry_fct_denom_df.write.format("parquet").partitionBy("CNTRY_ID").mode("overwrite").save("wasbs://tradepanelpoc@blobasbackupx2066561.blob.core.windows/cbls/hdi/trdpn_cntry_fct_denom_df.parquet")这是Zeppelin超时问题吗?如何将其扩展超过1小时的运行时间?感谢您的帮助.
Is this Zeppelin timeout issue? How can it be extended for more than 1 hour runtime? Thanks for the help.
推荐答案来自这个对我有用的堆栈溢出问题的答案
从输出来看,如果您的应用程序未以FAILED状态完成,这听起来像是Livy超时错误:您的应用程序可能比Livy会话所定义的超时时间更长(默认为1h),所以即使尽管Spark应用程序成功运行,但如果该应用程序花费的时间比Livy会话的超时时间长,则您的笔记本仍会收到此错误.
Judging by the output, if your application is not finishing with a FAILED status, that sounds like a Livy timeout error: your application is likely taking longer than the defined timeout for a Livy session (which defaults to 1h), so even despite the Spark app succeeds your notebook will receive this error if the app takes longer than the Livy session's timeout.
如果是这种情况,请按以下步骤处理:
If that's the case, here's how to address it:
1. edit the /etc/livy/conf/livy.conf file (in the cluster's master node) 2. set the livy.server.session.timeout to a higher value, like 8h (or larger, depending on your app) 3. restart Livy to update the setting: sudo restart livy-server in the cluster's master 4. test your code again更多推荐
超时错误:400 StatusCode错误:“请求失败:会话未激活."
发布评论