HTrace在Hadoop 2.7.3中(HTrace in Hadoop 2.7.3)

HTrace在Hadoop 2.7.3中(HTrace in Hadoop 2.7.3)

我已经在Ubuntu 14.04上以2.6.0 x32的成本使用Zipkin与Hadoop Htrace。 现在我想使用它与Hadoop 2.7.3。，但我甚至无法启用此Hadoop版本的Htrace跟踪。 2.6.0中HTrace的设置与2.7.3不同，因为它可以在这里看到 - 2.6.0和这里 - 2.7.3 。

在2.6.0中，我会在namenode日志文件中有这一行：

INFO org.apache.hadoop.tracing.SpanReceiverHost: SpanReceiver org.htrace.impl.ZipkinSpanReceiver was loaded successfully.

我在2.7.3 Namenode日志文件中没有这样的内容。

由于没有成功使用Zipkin，我尝试使用LocalFileSpanReceiver，如在线教程中所述：

<property> <name>hadoop.htrace.sampler</name> <value>AlwaysSampler</value> </property> <property> <name>hadoop.htrace.spanreceiver.classes</name> <value>org.apache.htrace.impl.LocalFileSpanReceiver</value> </property> <property> <name>hadoop.htrace.local-file-span-receiver.path</name> <value>/var/log/hadoop/htrace.out</value> </property>

/ var / log / hadoop /存在，拥有777个权限，但没有任何...

TracingFsShell示例编译并运行时进行了以下修改：

SpanReceiverHost.get(new HdfsConfiguration(),"");

因为它可以在hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java中的hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java源代码中找到，虽然在线教程不使用该方法签名。 （来源差异 ）

这两个Hadoop版本的环境是一样的，java 1.7。 另外，hadoop是从源代码编译的，因为Ubuntu 14.04是x32位的。 Hadoop以全分布模式部署，使用lxc容器。

用于Zipkin的core-site.xml （ 这里是 Zipkin params）：

<property> <name>hadoop.htrace.spanreceiver.classes</name> <value>org.apache.htrace.impl.ZipkinSpanReceiver</value> </property> <property> <name>hadoop.htrace.zipkin.scribe.hostname</name> <value>10.0.3.100</value> </property> <property> <name>hadoop.htrace.zipkin.scribe.port</name> <value>9410</value> </property>

I've successfully used Zipkin with Hadoop Htrace in 2.6.0 x32, on Ubuntu 14.04. Now I want to use it with Hadoop 2.7.3., but I can't even enable Htrace tracing with this hadoop version. The setup for HTrace in 2.6.0 is different from 2.7.3, as it can be seen here-2.6.0 and here-2.7.3.

In 2.6.0 I'd have this line in the namenode log file :

INFO org.apache.hadoop.tracing.SpanReceiverHost: SpanReceiver org.htrace.impl.ZipkinSpanReceiver was loaded successfully.

I have nothing like that in 2.7.3 Namenode log file.

Because of not having success with Zipkin, I tried to use the LocalFileSpanReceiver as described in the online tutorial:

<property> <name>hadoop.htrace.sampler</name> <value>AlwaysSampler</value> </property> <property> <name>hadoop.htrace.spanreceiver.classes</name> <value>org.apache.htrace.impl.LocalFileSpanReceiver</value> </property> <property> <name>hadoop.htrace.local-file-span-receiver.path</name> <value>/var/log/hadoop/htrace.out</value> </property>

The /var/log/hadoop/ exists, with 777 rights on it, but nothing...

The TracingFsShell example compiles and runs with the following modification:

SpanReceiverHost.get(new HdfsConfiguration(),"");

As it can be found in the source code of hadoop in hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java although the online tutorial does not use that method signature. (Source diff)

The environment is the same for both Hadoop versions, java 1.7. Also, hadoop is compiled from source, as the Ubuntu 14.04 is x32 bit. Hadoop is deployed in fully-distributed mode, using lxc containers.

core-site.xml for Zipkin ( Zipkin params here):

<property> <name>hadoop.htrace.spanreceiver.classes</name> <value>org.apache.htrace.impl.ZipkinSpanReceiver</value> </property> <property> <name>hadoop.htrace.zipkin.scribe.hostname</name> <value>10.0.3.100</value> </property> <property> <name>hadoop.htrace.zipkin.scribe.port</name> <value>9410</value> </property>

最满意答案

感谢您试用HTrace！ 对不起，版本问题现在是如此的痛苦。

使用cloudera的Hadoop CDH5.5发行版及更高版本配置HTrace更容易。 这里有一个很好的描述： http : //blog.cloudera.com/blog/2015/12/new-in-cloudera-labs-apache-htrace-incubating/如果你想坚持一个Apache发布源代码而不是供应商发布，请尝试使用Hadoop 3.0.0-alpha1。 http://hadoop.apache.org/releases.html

在Hadoop 2.6和2.7中传输的HTrace库非常古老...我们从未将HTrace 4.x移植到这些分支。 他们是稳定的分支，所以跟踪等新功能超出了范围。 这里有一些功能，但并不多。 我建议使用新开发的HTrace 4.x库。 HTrace 4.x分支也有一个稳定的API，所以希望未来可以尽量减少破坏。

Thanks for trying out HTrace! Sorry that the version issue is such a pain right now.

It is much easier to configure HTrace with the version in cloudera's CDH5.5 distribution of Hadoop and later. There is a good description of how to do it here: http://blog.cloudera.com/blog/2015/12/new-in-cloudera-labs-apache-htrace-incubating/ If you want to stick with an Apache release of the source code rather than a vendor release, try Hadoop 3.0.0-alpha1. http://hadoop.apache.org/releases.html

The HTrace libraries shippped in Hadoop 2.6 and 2.7 are very old... we never backported HTrace 4.x to those branches. They were stability branches, so new features like tracing was out of scope. There is some functionality there, but not much. I recommend using the newer HTrace 4.x library which is actively developed. The HTrace 4.x branch also has a stable API, so hopefully breakage will be minimized in the future.