连续将本地文件与从S3下载的文件组合在一起(Continuously Combining local file with files downloaded from S3)

连续将本地文件与从S3下载的文件组合在一起(Continuously Combining local file with files downloaded from S3)

我有一个NIFI流，我从S3获取文件。 一对文件通过S3获取，然后传递到MergeContent处理器。 接下来，有一个README文件需要与每对文件一起使用。

这个README文件总是相同的，我已经将它存储在本地。 我有一个从MergeContent处理器接收内容的ExecuteStreamCommand 。

我已经尝试使用ListFile / FetchFile组合将README文件传入MergeContent处理器，但未按预期工作。 我猜我最终的结果是一个MergeContent包，它包含从S3 + README文件下载的一对文件。

I have a Nifi flow where I am fetching files from S3. A pair of files are fetched through S3 and later passed into a MergeContent processor. Next, there is a README file that needs to go with each pair of files.

This README file is always same and I have stored it locally. I have a ExecuteStreamCommand that takes in content from the MergeContent processor.

I have tried passing in the README file into the MergeContent processor using the ListFile/FetchFile combination but its not working as expected. I guess the final result that I am looking for is a MergeContent package that contains a pair of files downloaded from S3 + the README file.

最满意答案

我认为在这种情况下，您会希望为README使用GetFile - 路径是静态的，您可以将Keep Source File设置为true ，以便不断检索相同的内容。

ListFile / FetchFile可能不起作用，因为一旦ListFile从目录中检索到文件名，它就会将时间戳存储在本地状态中，并且不会检索比下一次执行时更早的文件。

I think in this case you will want to use GetFile for the README -- the path is static, and you can set the Keep Source File setting to true in order to constantly retrieve the same content.

ListFile/FetchFile probably isn't working because once ListFile retrieves a filename from the directory, it stores the timestamp in its local state and won't retrieve files older than that on the next execution.