本文介绍了设置templateLocation参数时,数据流作业运行失败的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
当我传递参数staging,temp&时,数据流作业失败,出现以下异常.输出GCS存储桶位置.
Dataflow job is failing with below exception when I pass parameters staging,temp & output GCS bucket locations.
Java代码:
final String[] used = Arrays.copyOf(args, args.length + 1); used[used.length - 1] = "--project=OVERWRITTEN"; final T options = PipelineOptionsFactory.fromArgs(used).withValidation().as(clazz); options.setProject(PROJECT_ID); options.setStagingLocation("gs://abc/staging/"); options.setTempLocation("gs://abc/temp"); options.setRunner(DataflowRunner.class); options.setGcpTempLocation("gs://abc");错误:
INFO: Staging pipeline description to gs://ups-heat-dev- tmp/mniazstaging_ingest_validation/staging/ May 10, 2018 11:56:35 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <42088 bytes, hash E7urYrjAOjwy6_5H-UoUxA> to gs://ups-heat-dev-tmp/mniazstaging_ingest_validation/staging/pipeline-E7urYrjAOjwy6_5H-UoUxA.pb Dataflow SDK version: 2.4.0 May 10, 2018 11:56:38 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Printed job specification to gs://ups-heat-dev-tmp/mniazstaging_ingest_validation/templates/DataValidationPipeline May 10, 2018 11:56:40 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Template successfully created. Exception in thread "main" java.lang.NullPointerException at org.apache.beam.runners.dataflow.DataflowPipelineJob.getJobWithRetries(DataflowPipelineJob.java:501) at org.apache.beam.runners.dataflow.DataflowPipelineJob.getStateWithRetries(DataflowPipelineJob.java:477) at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:312) at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:248) at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:202) at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:195) at com.example.DataValidationPipeline.main(DataValidationPipeline.java:66)推荐答案
我也遇到了同样的问题,错误发生在p.run().waitForFinish();.然后我尝试了以下代码
I was also facing the same issue, the error was throwing at p.run().waitForFinish();. Then I have tried following code
PipelineResult result = p.run(); System.out.println(result.getState().hasReplacementJob()); result.waitUntilFinish();这引发了以下异常
java.lang.UnsupportedOperationException: The result of template creation should not be used. at org.apache.beam.runners.dataflow.util.DataflowTemplateJob.getState (DataflowTemplateJob.java:67)然后要解决此问题,我使用了以下代码
Then to fix the issue I used the following code
PipelineResult result = pipeline.run(); try { result.getState(); result.waitUntilFinish(); } catch (UnsupportedOperationException e) { // do nothing } catch (Exception e) { e.printStackTrace(); }更多推荐
设置templateLocation参数时,数据流作业运行失败
发布评论