Hive建表语句如下:
CREATE EXTERNAL TABLE IF NOT EXISTS
students ( id int, name string, gender string, birthday Date, clazz string, phone string, loc string)
COMMENT 'student details'
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\r\n' STORED AS TEXTFILE;
FAILED: SemanticException 5:20 LINES TERMINATED BY only supports newline '\n' right now. Error encountered near token ''\r\n''
这个的大意是仅支持 ‘\n’ 作为行分割符,(不太理解,现在hive到2点多的版本为什么还不支持自定义行分隔符,不支持为什么还设置这个LINES TERMINATED BY 配置)
但如果我们的内容必须要其他分割符,我们可以通过设置mapred来间接设置分割符。
在hive cil中输入:set textinputformat.record.linesep=\r\n;
建表语句可以中别写 LINES TERMINATED BY
CREATE EXTERNAL TABLE IF NOT EXISTS
students ( id int, name string, gender string, birthday Date, clazz string, phone string, loc string)
COMMENT 'student details'
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
完美解决,加载数据,显示正常;
原因:在 hive cil 中输入 describe extended students ;
里面有两条重要信息:
inputFormat:.apache.hadoop.mapred.TextInputFormat,
outputFormat:.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
这表示:
默认状态下,hive的输入和输出调用的类。
类TextInputFormat在hadoop-mapreduce-client-core-***.jar中。
重点看类中getRecordReader方法,该方法返回LineRecordReader对象。并且该方法中已经实现了接收自定义字符串作为换行符的代码,只要建表前在hive的CLI界面上输入set textinputformat.record.delimiter=<自定义换行字符串>;即可实现自定义多字符换行符。
public class TextInputFormat extends FileInputFormat<LongWritable, Text>implements JobConfigurable {* * * public RecordReader<LongWritable, Text> getRecordReader( InputSplit genericSplit, JobConf job,Reporter reporter) throws IOException {reporter.setStatus(genericSplit.toString());String delimiter = job.get("textinputformat.record.delimiter");byte[] recordDelimiterBytes = null;if (null != delimiter) {recordDelimiterBytes = delimiter.getBytes(Charsets.UTF_8);}return new LineRecordReader(job, (FileSplit) genericSplit, recordDelimiterBytes);}
}
更多推荐
解决办法,LINES,Hive,TERMINATED,newline
发布评论