我在Windows机器上运行eclipse，并且HDFS在Unix机器上运行。
我试图连接到运行在远程机器上的HDFS实例。 。这是我已经尝试过的。
Configuration conf = new Configuration（）; conf.set（fs.defaultFS，hdfs：// remoteHostName：portNumber）; DFSClient client = null; System.out.println（try）; 尝试 { System.out.println（trying）; client = new DFSClient（conf）; System.out.println（client）; catch（IOException e）{ e.printStackTrace（）; } finally { if（client！= null） try { client.close（）; } catch（IOException e）{ e.printStackTrace（）; } }我有以下例外：
线程main中的异常java.lang.NoSuchMethodError：org.apache.hadoop.ipc.RPC。 getProxy（Ljava /郎/类; JLjava /净/的InetSocketAddress; Lorg /阿帕奇/ hadoop的/安全/ UserGroupInformation; Lorg /阿帕奇/ hadoop的/ CONF /配置; Ljavax /净/的SocketFactory; ILorg /阿帕奇/ hadoop的/ IO /重试/ RetryPolicy; Z）Lorg /阿帕奇/ hadoop的/ IPC / VersionedProtocol; at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode（DFSClient.java:135） at org.apache.hadoop.hdfs.DFSClient。< init>（DFSClient.java:280） at org.apache.hadoop.hdfs.DFSClient。< init>（DFSClient.java:245） at org.apache.hadoop.hdfs.DFSClient。< init>（DFSClient.java:235 ） at org.apache.hadoop.hdfs.DFSClient。< init>（DFSClient.java:226）顺便说一下，我从远程机器上的hdfs-site.xml获得了portNumber。
这种方法是否正确？
另外，在Python中执行此操作会更容易吗？
编辑
请注意，我的窗口上解压缩了Hadoop二进制文件，并且相应地设置了HADOOP_HOME环境变量。这可能是导致问题的原因吗？
解决方案请参阅： Hadoop 2.6.0 Browsing filesystem Java for your specific problem。
除此之外，您可能需要考虑使用REST进行远程交互。 Apache Knox可以为您提供对远程集群的访问，并防止您的代码不必知道集群内部组件，例如host：port，kerberos或不等等。这些事情可能会从您的远程客户端下改变。
i am trying to connect to an HDFS instance running on a remote machine.
I am running eclipse on a windows machine and the HDFS is running on a Unix box. Here is what i have tried
Configuration conf = new Configuration(); conf.set("fs.defaultFS", "hdfs://remoteHostName:portNumber"); DFSClient client = null; System.out.println("try"); try { System.out.println("trying"); client = new DFSClient(conf); System.out.println(client); } catch (IOException e) { e.printStackTrace(); } finally { if(client!=null) try { client.close(); } catch (IOException e) { e.printStackTrace(); } }but this gives me the following exception
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;ILorg/apache/hadoop/io/retry/RetryPolicy;Z)Lorg/apache/hadoop/ipc/VersionedProtocol; at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:135) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:280) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:235) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:226)by the way, i got the portNumber from the hdfs-site.xml on the remote machine
Is this approach correct?
Also, would it be easier to do this in Python?
EDIT
Note that i do have the Hadoop binaries unzipped on my windows and i have set the HADOOP_HOME environment variable accordingly. Could this be causing a problem?
解决方案See: Hadoop 2.6.0 Browsing filesystem Java for your specific problem.
Beyond that, you might want to consider using REST for remote interactions. Apache Knox can provide you with access to the remote cluster and shield your code from having to know cluster internals such as host:port, kerberos or not, etc. These things can change out from under your remote clients.
更多推荐
如何连接到远程HDFS
发布评论