我正在尝试使用Flume-ng为exec source将数据写入Hdfs。但它总是以退出代码127结束。它也显示警告像
无法从VM获取maxDirectMemory:NoSuchMethodException:sun.misc.VM.maxDirectMemory(null)。
这是exec.conf文件
execAgent.sources=e execAgent.channels=memchannel execAgent.sinks=HDFS execAgent.sources.e.type=org.apache.flume.source.ExecSource execAgent.sources.e.channels=memchannel execAgent.sources.e.shell=/bin/bash execAgent.sources.e.command=tail -f /home/sample.txt execAgent.sinks.HDFS.type=hdfs execAgent.sinks.HDFS.channel=memchannel execAgent.sinks.HDFS.hdfs.path=hdfs://ip:address:port/user/flume/ execAgent.sinks.HDFS.hdfs.fileType=DataStream execAgent.sinks.HDFS.hdfs.writeFormat=Text execAgent.channels.memchannel.type=file execAgent.channels.memchannel.capacity=1000 execAgent.channels.memchannel.transactionCapacity=100 execAgent.sources.e.channels=memchannel execAgent.sinks.HDFS.channel=memchannel这是我在控制台上获得的输出
15/04/17 06:24:54 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 15/04/17 06:24:54 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:exec.conf 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: execAgent 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:55 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [execAgent] 15/04/17 06:24:55 INFO node.AbstractConfigurationProvider: Creating channels 15/04/17 06:24:55 INFO channel.DefaultChannelFactory: Creating instance of channel memchannel type file 15/04/17 06:24:55 INFO node.AbstractConfigurationProvider: Created channel memchannel 15/04/17 06:24:55 INFO source.DefaultSourceFactory: Creating instance of source e, type org.apache.flume.source.ExecSource 15/04/17 06:24:55 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 15/04/17 06:24:56 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false 15/04/17 06:24:56 INFO node.AbstractConfigurationProvider: Channel memchannel connected to [e, HDFS] 15/04/17 06:24:56 INFO node.Application: Starting new configuration:{ sourceRunners:{e=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:e,state:IDLE} }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2577d2c2 counterGroup:{ name:null counters:{} } }} channels:{memchannel=FileChannel memchannel { dataDirs: [/root/.flume/file-channel/data] }} } 15/04/17 06:24:56 INFO node.Application: Starting Channel memchannel 15/04/17 06:24:56 INFO file.FileChannel: Starting FileChannel memchannel { dataDirs: [/root/.flume/file-channel/data] }... 15/04/17 06:24:56 INFO file.Log: Encryption is not enabled 15/04/17 06:24:56 INFO file.Log: Replay started 15/04/17 06:24:56 INFO file.Log: Found NextFileID 0, from [] 15/04/17 06:24:56 INFO file.EventQueueBackingStoreFile: Preallocated /root/.flume/file-channel/checkpoint/checkpoint_1429251896225 to 16232 for capacity 1000 15/04/17 06:24:56 INFO file.EventQueueBackingStoreFileV3: Starting up with /root/.flume/file-channel/checkpoint/checkpoint_1429251896225 and /root/.flume/file-channel/checkpoint/checkpoint_1429251896225.meta 15/04/17 06:24:57 INFO file.Log: Last Checkpoint Fri Apr 17 06:24:56 UTC 2015, queue depth = 0 15/04/17 06:24:57 INFO file.Log: Replaying logs with v2 replay logic 15/04/17 06:24:57 INFO file.ReplayHandler: Starting replay of [] 15/04/17 06:24:57 INFO file.ReplayHandler: read: 0, put: 0, take: 0, rollback: 0, commit: 0, skip: 0, eventCount:0 15/04/17 06:24:57 INFO file.Log: Rolling /root/.flume/file-channel/data 15/04/17 06:24:57 INFO file.Log: Roll start /root/.flume/file-channel/data 15/04/17 06:24:57 INFO tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) 15/04/17 06:24:57 INFO tools.DirectMemoryUtils: Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 18874368, Remaining = 18874368 15/04/17 06:24:57 INFO file.LogFile: Opened /root/.flume/file-channel/data/log-1 15/04/17 06:24:57 INFO file.Log: Roll end 15/04/17 06:24:57 INFO file.EventQueueBackingStoreFile: Start checkpoint for /root/.flume/file-channel/checkpoint/checkpoint_1429251896225, elements to sync = 0 15/04/17 06:24:57 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1429251897136, queueSize: 0, queueHead: 0 15/04/17 06:24:57 INFO file.Log: Updated checkpoint for file: /root/.flume/file-channel/data/log-1 position: 0 logWriteOrderID: 1429251897136 15/04/17 06:24:57 INFO file.FileChannel: Queue Size after replay: 0 [channel=memchannel] 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: memchannel, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memchannel started 15/04/17 06:24:57 INFO node.Application: Starting Sink HDFS 15/04/17 06:24:57 INFO node.Application: Starting Source e 15/04/17 06:24:57 INFO source.ExecSource: Exec source starting with command:tail -f /home/sample.txt 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: HDFS, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SOURCE, name: e, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: e started 15/04/17 06:24:57 INFO source.ExecSource: Command [tail -f /home/brillio/sample.txt] exited with 127i'm trying to write data into Hdfs using Flume-ng for exec source.But it always ended with exit code 127.and it's also showing warning like
Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null).
This is exec.conf file
execAgent.sources=e execAgent.channels=memchannel execAgent.sinks=HDFS execAgent.sources.e.type=org.apache.flume.source.ExecSource execAgent.sources.e.channels=memchannel execAgent.sources.e.shell=/bin/bash execAgent.sources.e.command=tail -f /home/sample.txt execAgent.sinks.HDFS.type=hdfs execAgent.sinks.HDFS.channel=memchannel execAgent.sinks.HDFS.hdfs.path=hdfs://ip:address:port/user/flume/ execAgent.sinks.HDFS.hdfs.fileType=DataStream execAgent.sinks.HDFS.hdfs.writeFormat=Text execAgent.channels.memchannel.type=file execAgent.channels.memchannel.capacity=1000 execAgent.channels.memchannel.transactionCapacity=100 execAgent.sources.e.channels=memchannel execAgent.sinks.HDFS.channel=memchannelthis is the output i'm getting on console
15/04/17 06:24:54 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 15/04/17 06:24:54 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:exec.conf 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: execAgent 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:54 INFO conf.FlumeConfiguration: Processing:HDFS 15/04/17 06:24:55 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [execAgent] 15/04/17 06:24:55 INFO node.AbstractConfigurationProvider: Creating channels 15/04/17 06:24:55 INFO channel.DefaultChannelFactory: Creating instance of channel memchannel type file 15/04/17 06:24:55 INFO node.AbstractConfigurationProvider: Created channel memchannel 15/04/17 06:24:55 INFO source.DefaultSourceFactory: Creating instance of source e, type org.apache.flume.source.ExecSource 15/04/17 06:24:55 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 15/04/17 06:24:56 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false 15/04/17 06:24:56 INFO node.AbstractConfigurationProvider: Channel memchannel connected to [e, HDFS] 15/04/17 06:24:56 INFO node.Application: Starting new configuration:{ sourceRunners:{e=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:e,state:IDLE} }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2577d2c2 counterGroup:{ name:null counters:{} } }} channels:{memchannel=FileChannel memchannel { dataDirs: [/root/.flume/file-channel/data] }} } 15/04/17 06:24:56 INFO node.Application: Starting Channel memchannel 15/04/17 06:24:56 INFO file.FileChannel: Starting FileChannel memchannel { dataDirs: [/root/.flume/file-channel/data] }... 15/04/17 06:24:56 INFO file.Log: Encryption is not enabled 15/04/17 06:24:56 INFO file.Log: Replay started 15/04/17 06:24:56 INFO file.Log: Found NextFileID 0, from [] 15/04/17 06:24:56 INFO file.EventQueueBackingStoreFile: Preallocated /root/.flume/file-channel/checkpoint/checkpoint_1429251896225 to 16232 for capacity 1000 15/04/17 06:24:56 INFO file.EventQueueBackingStoreFileV3: Starting up with /root/.flume/file-channel/checkpoint/checkpoint_1429251896225 and /root/.flume/file-channel/checkpoint/checkpoint_1429251896225.meta 15/04/17 06:24:57 INFO file.Log: Last Checkpoint Fri Apr 17 06:24:56 UTC 2015, queue depth = 0 15/04/17 06:24:57 INFO file.Log: Replaying logs with v2 replay logic 15/04/17 06:24:57 INFO file.ReplayHandler: Starting replay of [] 15/04/17 06:24:57 INFO file.ReplayHandler: read: 0, put: 0, take: 0, rollback: 0, commit: 0, skip: 0, eventCount:0 15/04/17 06:24:57 INFO file.Log: Rolling /root/.flume/file-channel/data 15/04/17 06:24:57 INFO file.Log: Roll start /root/.flume/file-channel/data 15/04/17 06:24:57 INFO tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) 15/04/17 06:24:57 INFO tools.DirectMemoryUtils: Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 18874368, Remaining = 18874368 15/04/17 06:24:57 INFO file.LogFile: Opened /root/.flume/file-channel/data/log-1 15/04/17 06:24:57 INFO file.Log: Roll end 15/04/17 06:24:57 INFO file.EventQueueBackingStoreFile: Start checkpoint for /root/.flume/file-channel/checkpoint/checkpoint_1429251896225, elements to sync = 0 15/04/17 06:24:57 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1429251897136, queueSize: 0, queueHead: 0 15/04/17 06:24:57 INFO file.Log: Updated checkpoint for file: /root/.flume/file-channel/data/log-1 position: 0 logWriteOrderID: 1429251897136 15/04/17 06:24:57 INFO file.FileChannel: Queue Size after replay: 0 [channel=memchannel] 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: memchannel, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memchannel started 15/04/17 06:24:57 INFO node.Application: Starting Sink HDFS 15/04/17 06:24:57 INFO node.Application: Starting Source e 15/04/17 06:24:57 INFO source.ExecSource: Exec source starting with command:tail -f /home/sample.txt 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: HDFS, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SOURCE, name: e, registered successfully. 15/04/17 06:24:57 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: e started 15/04/17 06:24:57 INFO source.ExecSource: Command [tail -f /home/brillio/sample.txt] exited with 127最满意答案
从源文档
1)将参数: execAgent.sources.e.type修改为exec
2)从配置中删除execAgent.sources.e.shell参数
From the source documentation
1) Modify the parameter : execAgent.sources.e.type to exec
2) Remove the execAgent.sources.e.shell parameter from your configuration
更多推荐
发布评论