我正在学习Hive。令人惊讶的是,我找不到如何编写简单的字数统计工作的例子。是否正确?
假设我有一个输入文件 input.tsv :
hello,world 这是一个输入文件示例我在Python中创建了一个分隔符,将每一行转换为单词:
import sys $ b.b 为sys.stdin中的行:为line.split()中的单词:打印字然后在我的Hive脚本中有以下内容:
CREATE TABLE input (line STRING); LOAD DATA LOCAL INPATH'input.tsv'OVERWRITE INTO TABLE input; - 用于存放单词的临时表... CREATE TABLE单词(单词STRING); 添加文件splitter.py; INSERT OVERWRITE TABLE单词 SELECT TRANSFORM(文本)使用'python splitter.py' AS字 FROM input; SELECT word,count(*)AS FROM FROM GROUP GROUP BY word;我不确定我是否错过了某些东西,或者确实如此复杂。 (特别是,我是否需要临时的 words 表,并且是否需要编写外部分隔符函数?)
SELECT word,COUNT (*)FROM输入LATERAL VIEW explode(split(text,''))lTable as word GROUP BY word;我使用横向视图来启用表值函数(explode)来自分割函数并为每个值输出一个新行。在实践中,我使用UDF来封装IBM的ICU4J分词器。我通常不使用转换脚本,并使用UDF来处理所有事情。你不需要临时单词表。
I'm trying to learn Hive. Surprisingly, I can't find an example of how to write a simple word count job. Is the following correct?
Let's say I have an input file input.tsv:
hello, world this is an example input fileI create a splitter in Python to turn each line into words:
import sys for line in sys.stdin: for word in line.split(): print wordAnd then I have the following in my Hive script:
CREATE TABLE input (line STRING); LOAD DATA LOCAL INPATH 'input.tsv' OVERWRITE INTO TABLE input; -- temporary table to hold words... CREATE TABLE words (word STRING); add file splitter.py; INSERT OVERWRITE TABLE words SELECT TRANSFORM(text) USING 'python splitter.py' AS word FROM input; SELECT word, count(*) AS count FROM words GROUP BY word;I'm not sure if I'm missing something, or if it really is this complicated. (In particular, do I need the temporary words table, and do I need to write the external splitter function?)
解决方案If you want a simple one see the following:
SELECT word, COUNT(*) FROM input LATERAL VIEW explode(split(text, ' ')) lTable as word GROUP BY word;I use a lateral view to enable the use of a table valued function (explode) which takes the list that comes out of split function and outputs a new row for every value. In practice I use a UDF that wraps IBM's ICU4J word breaker. I generally don't use transform scripts and use UDFs for everything. You don't need a temporary words table.
更多推荐
Hive中的字数统计程序
发布评论