在Apache Nifi中将两个模式合并为一个

本文介绍了在Apache Nifi中将两个模式合并为一个的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我正在尝试使用Apache nifi将两个csv文件合并为一个json. 两个csv是person.csv，其中包含有关人的信息:

I'm trying to merge two csv files into a json using Apache nifi. Two csv's are persons.csv containing information about people:

Id|Name|Surname ABC-123|John|Smith ABC-111|Allan|Wood ABC-001|Grace|Kelly

第二个csv包含这些人参加的活动的列表:

And the second csv contains list of events these people have attended:

EId|PId|Date|Desc 1|ABC-123|2017-05-01|"Groove party" 2|ABC-111|2017-06-01|"Snack No. One" 3|ABC-123|2017-06-01|"The night out"

我正在使用( git枢纽上的Nifi流):

• GetFile
• UpdateAttribute(schema.name)
• 拆分记录
• ExtractText
• UpdateAttribute(correlation.id，newschema)
• 漏斗
• 合并记录/合并内容
• PutFile

试图实现最终的json:

Trying to achieve final json:

{ "Person": { "Id": "ABC-123", "Name": "John", "Surname": "Smith", "Events": [{ "Date": "2017-05-01", "Name": "Groove party" }, { "Date": "2017-06-01", "Name": "The night out" }] } }

但是我不确定如何设置合并记录，或者在将内容合并到单个json之后如何合并多条csv行.有办法吗?

But I'm not sure how to set up Merge Record, or how to join multiple csv lines after Merge Content into a single json. Is there a way how to do it?

推荐答案

您实际上可以使用

1- ConvertRecord (CSV到JSON)-在您的情况下使用Avro架构

1- ConvertRecord(CSV to JSON) - using Avro Schema in your case

CSVReader

CSVReader

JsonRecordSetWriter

JsonRecordSetWriter

AvroSchemaRegistry

AvroSchemaRegistry

{ "name": "person", "namespace": "nifi", "type": "record", "fields": [ {"name": "Id" , "type" : "string"}, {"name": "Name" , "type" : "string"}, {"name": "Surname" , "type" : "string"} ] }

• 2-SplitJson
• 3-EvaluateJson
• 4-LookupAttribute(获取Desc SimpleCsvFileLookupService)
• 5-LookupAttribute(获取日期SimpleCsvFileLookupService)
  • 6-AttriutetoJson
  • 2 - SplitJson
  • 3 - EvaluateJson
  • 4 - LookupAttribute (Get Desc SimpleCsvFileLookupService)
  • 5 - LookupAttribute (Get Date SimpleCsvFileLookupService)
    • 6 - AttriutetoJson