首页 > 编程入门 文章详情

Flume：Exec源cat命令不在HDFS上写入(Flume :Exec source cat command is not writing on HDFS)

编程入门 行业动态 更新时间:2024-10-27 00:23:54
Flume：Exec源cat命令不在HDFS上写入(Flume :Exec source cat command is not writing on HDFS)

我正在尝试使用Flume-ng为exec source将数据写入Hdfs。但它总是以退出代码127结束。它也显示警告像

无法从VM获取maxDirectMemory：NoSuchMethodException：sun.misc.VM.maxDirectMemory（null）。

这是exec.conf文件

execAgent.sources=e execAgent.channels=memchannel execAgent.sinks=HDFS execAgent.sources.e.type=org.apache.flume.source.ExecSource execAgent.sources.e.channels=memchannel execAgent.sources.e.shell=/bin/bash execAgent.sources.e.command=tail -f /home/sample.txt execAgent.sinks.HDFS.type=hdfs execAgent.sinks.HDFS.channel=memchannel execAgent.sinks.HDFS.hdfs.path=hdfs://ip:address:port/user/flume/ execAgent.sinks.HDFS.hdfs.fileType=DataStream execAgent.sinks.HDFS.hdfs.writeFormat=Text execAgent.channels.memchannel.type=file execAgent.channels.memchannel.capacity=1000 execAgent.channels.memchannel.transactionCapacity=100 execAgent.sources.e.channels=memchannel execAgent.sinks.HDFS.channel=memchannel

这是我在控制台上获得的输出

15/04/17 06:24:54 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 15/04/17 06:24:54 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:exec.conf 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: execAgent 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:55 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [execAgent] 15/04/17 06:24:55 INFO node.AbstractConfigurationProvider: Creating channels 15/04/17 06:24:55 INFO channel.DefaultChannelFactory: Creating instance of channel memchannel type file 15/04/17 06:24:55 INFO node.AbstractConfigurationProvider: Created channel memchannel 15/04/17 06:24:55 INFO source.DefaultSourceFactory: Creating instance of source e, type org.apache.flume.source.ExecSource 15/04/17 06:24:55 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 15/04/17 06:24:56 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false 15/04/17 06:24:56 INFO node.AbstractConfigurationProvider: Channel memchannel connected to [e, HDFS] 15/04/17 06:24:56 INFO node.Application: Starting new configuration:{ sourceRunners:{e=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:e,state:IDLE} }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2577d2c2 counterGroup:{ name:null counters:{} } }} channels:{memchannel=FileChannel memchannel { dataDirs: [/root/.flume/file-channel/data] }} } 15/04/17 06:24:56 INFO node.Application: Starting Channel memchannel 15/04/17 06:24:56 INFO file.FileChannel: Starting FileChannel memchannel { dataDirs: [/root/.flume/file-channel/data] }... 15/04/17 06:24:56 INFO file.Log: Encryption is not enabled 15/04/17 06:24:56 INFO file.Log: Replay started 15/04/17 06:24:56 INFO file.Log: Found NextFileID 0, from [] 15/04/17 06:24:56 INFO file.EventQueueBackingStoreFile: Preallocated /root/.flume/file-channel/checkpoint/checkpoint_1429251896225 to 16232 for capacity 1000 15/04/17 06:24:56 INFO file.EventQueueBackingStoreFileV3: Starting up with /root/.flume/file-channel/checkpoint/checkpoint_1429251896225 and /root/.flume/file-channel/checkpoint/checkpoint_1429251896225.meta 15/04/17 06:24:57 INFO file.Log: Last Checkpoint Fri Apr 17 06:24:56 UTC 2015, queue depth = 0 15/04/17 06:24:57 INFO file.Log: Replaying logs with v2 replay logic 15/04/17 06:24:57 INFO file.ReplayHandler: Starting replay of [] 15/04/17 06:24:57 INFO file.ReplayHandler: read: 0, put: 0, take: 0, rollback: 0, commit: 0, skip: 0, eventCount:0 15/04/17 06:24:57 INFO file.Log: Rolling /root/.flume/file-channel/data 15/04/17 06:24:57 INFO file.Log: Roll start /root/.flume/file-channel/data 15/04/17 06:24:57 INFO tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) 15/04/17 06:24:57 INFO tools.DirectMemoryUtils: Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 18874368, Remaining = 18874368 15/04/17 06:24:57 INFO file.LogFile: Opened /root/.flume/file-channel/data/log-1 15/04/17 06:24:57 INFO file.Log: Roll end 15/04/17 06:24:57 INFO file.EventQueueBackingStoreFile: Start checkpoint for /root/.flume/file-channel/checkpoint/checkpoint_1429251896225, elements to sync = 0 15/04/17 06:24:57 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1429251897136, queueSize: 0, queueHead: 0 15/04/17 06:24:57 INFO file.Log: Updated checkpoint for file: /root/.flume/file-channel/data/log-1 position: 0 logWriteOrderID: 1429251897136 15/04/17 06:24:57 INFO file.FileChannel: Queue Size after replay: 0 [channel=memchannel] 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: memchannel, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memchannel started 15/04/17 06:24:57 INFO node.Application: Starting Sink HDFS 15/04/17 06:24:57 INFO node.Application: Starting Source e 15/04/17 06:24:57 INFO source.ExecSource: Exec source starting with command:tail -f /home/sample.txt 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: HDFS, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SOURCE, name: e, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: e started 15/04/17 06:24:57 INFO source.ExecSource: Command [tail -f /home/brillio/sample.txt] exited with 127

i'm trying to write data into Hdfs using Flume-ng for exec source.But it always ended with exit code 127.and it's also showing warning like

Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null).

This is exec.conf file

execAgent.sources=e execAgent.channels=memchannel execAgent.sinks=HDFS execAgent.sources.e.type=org.apache.flume.source.ExecSource execAgent.sources.e.channels=memchannel execAgent.sources.e.shell=/bin/bash execAgent.sources.e.command=tail -f /home/sample.txt execAgent.sinks.HDFS.type=hdfs execAgent.sinks.HDFS.channel=memchannel execAgent.sinks.HDFS.hdfs.path=hdfs://ip:address:port/user/flume/ execAgent.sinks.HDFS.hdfs.fileType=DataStream execAgent.sinks.HDFS.hdfs.writeFormat=Text execAgent.channels.memchannel.type=file execAgent.channels.memchannel.capacity=1000 execAgent.channels.memchannel.transactionCapacity=100 execAgent.sources.e.channels=memchannel execAgent.sinks.HDFS.channel=memchannel

this is the output i'm getting on console

15/04/17 06:24:54 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 15/04/17 06:24:54 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:exec.conf 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: execAgent 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:55 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [execAgent] 15/04/17 06:24:55 INFO node.AbstractConfigurationProvider: Creating channels 15/04/17 06:24:55 INFO channel.DefaultChannelFactory: Creating instance of channel memchannel type file 15/04/17 06:24:55 INFO node.AbstractConfigurationProvider: Created channel memchannel 15/04/17 06:24:55 INFO source.DefaultSourceFactory: Creating instance of source e, type org.apache.flume.source.ExecSource 15/04/17 06:24:55 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 15/04/17 06:24:56 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false 15/04/17 06:24:56 INFO node.AbstractConfigurationProvider: Channel memchannel connected to [e, HDFS] 15/04/17 06:24:56 INFO node.Application: Starting new configuration:{ sourceRunners:{e=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:e,state:IDLE} }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2577d2c2 counterGroup:{ name:null counters:{} } }} channels:{memchannel=FileChannel memchannel { dataDirs: [/root/.flume/file-channel/data] }} } 15/04/17 06:24:56 INFO node.Application: Starting Channel memchannel 15/04/17 06:24:56 INFO file.FileChannel: Starting FileChannel memchannel { dataDirs: [/root/.flume/file-channel/data] }... 15/04/17 06:24:56 INFO file.Log: Encryption is not enabled 15/04/17 06:24:56 INFO file.Log: Replay started 15/04/17 06:24:56 INFO file.Log: Found NextFileID 0, from [] 15/04/17 06:24:56 INFO file.EventQueueBackingStoreFile: Preallocated /root/.flume/file-channel/checkpoint/checkpoint_1429251896225 to 16232 for capacity 1000 15/04/17 06:24:56 INFO file.EventQueueBackingStoreFileV3: Starting up with /root/.flume/file-channel/checkpoint/checkpoint_1429251896225 and /root/.flume/file-channel/checkpoint/checkpoint_1429251896225.meta 15/04/17 06:24:57 INFO file.Log: Last Checkpoint Fri Apr 17 06:24:56 UTC 2015, queue depth = 0 15/04/17 06:24:57 INFO file.Log: Replaying logs with v2 replay logic 15/04/17 06:24:57 INFO file.ReplayHandler: Starting replay of [] 15/04/17 06:24:57 INFO file.ReplayHandler: read: 0, put: 0, take: 0, rollback: 0, commit: 0, skip: 0, eventCount:0 15/04/17 06:24:57 INFO file.Log: Rolling /root/.flume/file-channel/data 15/04/17 06:24:57 INFO file.Log: Roll start /root/.flume/file-channel/data 15/04/17 06:24:57 INFO tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) 15/04/17 06:24:57 INFO tools.DirectMemoryUtils: Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 18874368, Remaining = 18874368 15/04/17 06:24:57 INFO file.LogFile: Opened /root/.flume/file-channel/data/log-1 15/04/17 06:24:57 INFO file.Log: Roll end 15/04/17 06:24:57 INFO file.EventQueueBackingStoreFile: Start checkpoint for /root/.flume/file-channel/checkpoint/checkpoint_1429251896225, elements to sync = 0 15/04/17 06:24:57 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1429251897136, queueSize: 0, queueHead: 0 15/04/17 06:24:57 INFO file.Log: Updated checkpoint for file: /root/.flume/file-channel/data/log-1 position: 0 logWriteOrderID: 1429251897136 15/04/17 06:24:57 INFO file.FileChannel: Queue Size after replay: 0 [channel=memchannel] 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: memchannel, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memchannel started 15/04/17 06:24:57 INFO node.Application: Starting Sink HDFS 15/04/17 06:24:57 INFO node.Application: Starting Source e 15/04/17 06:24:57 INFO source.ExecSource: Exec source starting with command:tail -f /home/sample.txt 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: HDFS, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SOURCE, name: e, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: e started 15/04/17 06:24:57 INFO source.ExecSource: Command [tail -f /home/brillio/sample.txt] exited with 127

最满意答案

从源文档

1）将参数： execAgent.sources.e.type修改为exec

2）从配置中删除execAgent.sources.e.shell参数

From the source documentation

1) Modify the parameter : execAgent.sources.e.type to exec

2) Remove the execAgent.sources.e.shell parameter from your configuration

更多推荐

本文发布于:2023-04-29 11:30:00，感谢您对本站的认可！
本文链接:https://www.elefans.com/category/jswz/34/1336263.html
版权声明:本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系，我们将在24小时内删除。
本文标签：命令   cat   Exec   Flume   writing

发布评论

评论列表 （有 0 条评论）

最近发表

草根站长

>www.elefans.com

编程频道|电子爱好者 - 技术资讯及电子产品介绍！

热门文章

标签列表