使用Akka Stream流式传输巨大的json

本文介绍了使用Akka Stream流式传输巨大的json的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我有一个用json平板产生大量http响应的问题，其中只有一部分是兴趣点. 我无法更改响应结构. 这是一个例子

I have a problem of huge http response with a json slab, where only portion is point of interest. I cannot change the response structure. here is an example

{ "searchString": "search", "redirectUrl": "", "0": { "numRecords": 123, "refinementViewModelCollector": {}, // Lots of data here "results": [ { "productCode": "123", "productShortDescription": "Desc", "brand": "Brand", "productReview": { "reviewScore": 0 }, "priceView": { "salePriceDisplayable": false, }, "productImageUrl": "url", "alternateImageUrls": [ "url1" ], "largeProductImageUrl": "url4", "videoUrl": "" }, { "productCode": "124", "productShortDescription": "Desc", "brand": "Brand", "productReview": { "reviewScore": 0 }, "priceView": { "salePriceDisplayable": false, }, "preOrder": false, "productImageUrl": "url", "alternateImageUrls": [ "url1" ], "largeProductImageUrl": "url4", "videoUrl": "" } ] //lots of data here } }

我的兴趣点是results Jason Array中的条目，但它们位于json的中间

My point of interest is entries in results Jason Array, but the are sitting in the middle of json

我这样创建了一个小型的Play WS Client:

I created a small Play WS Client like this:

val wsClient: WSClient = ??? val ret = wsClient.url("url").stream() ret.flatMap { response => response.body.via(JsonFraming.objectScanner(1024)) .map(_.utf8String) .runWith(Sink.foreach(println)) }

这将不起作用，因为它将整个json平板作为Json对象.我需要跳过一些数据，直到"results":条目出现在流中，然后开始解析条目并跳过所有其余的内容. 任何想法如何做到这一点?

this will not work because it will take whole json slab as Json object. I need to skip some data until "results": entry appear in the stream, then start parsing entries and skip all the rest. Any ideas how to do this?

推荐答案

有些解析器支持将其解析为流.举一个很好的例子，看看这个Circe例子 github /circe/circe/tree/master/examples/sf-city-lots

There are parsers that support parsing as a stream. For a good example check out this Circe example github/circe/circe/tree/master/examples/sf-city-lots