Pig BigDecimal to Hive decimal(Pig BigDecimal to Hive decimal)

Pig BigDecimal to Hive decimal(Pig BigDecimal to Hive decimal)

我试图将Pig BigDecimal转换为Hive Decimal类型，但值被gettined null。 以下是示例代码：

猪脚本：

a = LOAD 'test.txt' using TextLoader() as (col1:chararray,col2:int,col3:chararray,col4:int); b = foreach a generate *,1 as rec_cnt; c = group b by col1,col3; d = foreach c generate flatten(group),(bigdecimal) SUM(rec_cnt) as grp_code; STORE d into 'user/test' Using PigStorage(','); STORE d into 'default.test' using org.apache.hive.hcatalog.pig.HCatStorer();

在上面的代码中，记录计数的总和值正好存储在存储为“user / test”的HDFS文件中。 但是对于HcatStorer，所有记录的相同字段都填充NULL。 使用DECIMAL（16,0）的此列定义创建测试表。 我正在使用Hive 1.1.0。 请建议如何解决此问题。

I am trying to convert Pig BigDecimal to Hive Decimal type but values are gettined null. Here is the sample code:

Pig Scripts:

a = LOAD 'test.txt' using TextLoader() as (col1:chararray,col2:int,col3:chararray,col4:int); b = foreach a generate *,1 as rec_cnt; c = group b by col1,col3; d = foreach c generate flatten(group),(bigdecimal) SUM(rec_cnt) as grp_code; STORE d into 'user/test' Using PigStorage(','); STORE d into 'default.test' using org.apache.hive.hcatalog.pig.HCatStorer();

In the above code, the sum value of record counts is coming properly in the HDFS file stored as 'user/test'. But with HcatStorer, the same field is populated with NULL for all the records. The test table is created with this column definition of DECIMAL(16,0). I am using Hive 1.1.0. Please suggest how to fix this issue.

最满意答案

我终于找到了null的原因。 Hcatalog在将猪的bigdecimal转换为hive中的Decimal时进行范围检查。 由于Hive定义中没有缩放（即DECIMAL（16,0）），因此在存储时范围检查期间默认为null。 当我将Hive定义更改为DECIMAL（16,2）时，它会被正确存储。 因此，这需要更改布局以确保更新比例。

I have finally figured it out the reason for null. Hcatalog does a range check when converting bigdecimal in pig to Decimal in hive. As there is no scale in the Hive definition (i.e., DECIMAL (16,0)), it is getting defaulted to null during range check while storing. When I changed the Hive definition to DECIMAL(16,2) it is getting stored properly. So this requires a change in layout for sure to update the scale.