与Spark 2.4.0和EMR 5.26.0兼容JSON4S

本文介绍了与Spark 2.4.0和EMR 5.26.0兼容JSON4S的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述 Spark json4s[java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/Js]

在AWS Emr上运行Spark Scala结构化流应用程序时，在解析复杂的json时遇到上述错误.

Getting above error while parsing complex json when running spark scala structured streaming application on aws emr.

推荐答案

它看起来像是二进制兼容性错误...能否请您检查依赖树中是否存在json4s工件的不兼容版本?

It looks like a binary compatibility error... Could you please check the dependency tree for incompatible versions of json4s artifacts?

如果您将不能升级它们以使用相同版本，则可以使用这些.

If you will not able to upgrade them to use the same version then possible you can solve the problem by shading some of them with sbt-assembly plugin using rules like these.

无论如何，我建议使用更安全高效的解析器，例如 jsoniter-scala (或基于它的 dijon ).

In any case I would recommend to use more safe and efficient parser like jsoniter-scala (or dijon that is based on it).