设置templateLocation参数时，数据流作业运行失败

本文介绍了设置templateLocation参数时，数据流作业运行失败的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

当我传递参数staging，temp&时，数据流作业失败，出现以下异常.输出GCS存储桶位置.

Dataflow job is failing with below exception when I pass parameters staging,temp & output GCS bucket locations.

Java代码:

final String[] used = Arrays.copyOf(args, args.length + 1); used[used.length - 1] = "--project=OVERWRITTEN"; final T options = PipelineOptionsFactory.fromArgs(used).withValidation().as(clazz); options.setProject(PROJECT_ID); options.setStagingLocation("gs://abc/staging/"); options.setTempLocation("gs://abc/temp"); options.setRunner(DataflowRunner.class); options.setGcpTempLocation("gs://abc");

错误:

INFO: Staging pipeline description to gs://ups-heat-dev- tmp/mniazstaging_ingest_validation/staging/ May 10, 2018 11:56:35 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <42088 bytes, hash E7urYrjAOjwy6_5H-UoUxA> to gs://ups-heat-dev-tmp/mniazstaging_ingest_validation/staging/pipeline-E7urYrjAOjwy6_5H-UoUxA.pb Dataflow SDK version: 2.4.0 May 10, 2018 11:56:38 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Printed job specification to gs://ups-heat-dev-tmp/mniazstaging_ingest_validation/templates/DataValidationPipeline May 10, 2018 11:56:40 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Template successfully created. Exception in thread "main" java.lang.NullPointerException at org.apache.beam.runners.dataflow.DataflowPipelineJob.getJobWithRetries(DataflowPipelineJob.java:501) at org.apache.beam.runners.dataflow.DataflowPipelineJob.getStateWithRetries(DataflowPipelineJob.java:477) at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:312) at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:248) at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:202) at org.apache.beam.runners.dataflow.DataflowPipelineJob.waitUntilFinish(DataflowPipelineJob.java:195) at com.example.DataValidationPipeline.main(DataValidationPipeline.java:66)

推荐答案

我也遇到了同样的问题，错误发生在p.run().waitForFinish();.然后我尝试了以下代码

I was also facing the same issue, the error was throwing at p.run().waitForFinish();. Then I have tried following code

PipelineResult result = p.run(); System.out.println(result.getState().hasReplacementJob()); result.waitUntilFinish();

这引发了以下异常

java.lang.UnsupportedOperationException: The result of template creation should not be used. at org.apache.beam.runners.dataflow.util.DataflowTemplateJob.getState (DataflowTemplateJob.java:67)

然后要解决此问题，我使用了以下代码

Then to fix the issue I used the following code

PipelineResult result = pipeline.run(); try { result.getState(); result.waitUntilFinish(); } catch (UnsupportedOperationException e) { // do nothing } catch (Exception e) { e.printStackTrace(); }