数据 API 限制?"/>
如何绕过 Amazon Aurora 1MB 数据 API 限制?
我正在努力从 Amazon Aurora Postgres DB 检索商店列表,我需要完整的列表。
我有超过 20.000 家商店,但由于 Amazon Aurora 数据 API 的 1MB 限制而失败..
我收到的消息:
{
"error": "INTERNAL_SERVER_ERROR",
"message": "select \"e0\".*, \"e1\".\"id\" as \"location_id\" from \"shop\" as \"e0\" left join \"location\" as \"e1\" on \"e0\".\"id\" = \"e1\".\"shop_id\" where \"e0\".\"name\" is not null and \"e0\".\"online\" = false and \"e0\".\"status\" = 'active' and \"e1\".\"formatted_address\" is not null and \"e1\".\"latitude\" is not null and \"e1\".\"longitude\" is not null order by \"e0\".\"uuid\" asc limit 10000 - Database returned more than the allowed response size limit"
}
I use mikro-orm and heres sample of my code:
@Get()
@Middleware(
validateQueryParams(
Joi.object({
offset: Joi.number().integer().raw().min(0),
limit: Joi.number().integer().raw().min(0).max(ShopRepository.MaxResults),
})
)
)
async list(req: Request, res: Response, next: NextFunction) {
logger.trace('ShopLocationController::list() - Enter');
const { limit, offset } = parseFixedLengthCollectionParams(req.query);
// Connect to all jurisdictional databases and retrieve their shop locations
const databases = Adapter.allJurisdictionalDataApis();
const results: ShopSetResult[] = await Promise.all(
databases.map(async database => {
await database.connect();
const shopRepository = database.em.getRepository(Shop) as ShopRepository;
const [shops, total] = await shopRepository.findHavingLocation(0, limit);
return { shops, total };
})
);
const completeShopsInJurisdictionBlocks = results.map(
(shopset: ShopSetResult) => shopset.shops
);
const completeShopSet = completeShopsInJurisdictionBlocks.flat();
// Calculate the total by adding all the totals from all the jurisdictions together
const total = results.reduce((runningTotal, shopset) => (runningTotal += shopset.total), 0);
// Sort shops and apply offsets and limits
completeShopSet.sort((a, b) => (a.uuid > b.uuid ? 1 : -1));
const shopsAfterOffsetAndLimit = completeShopSet.slice(
offset,
limit ? offset + limit : undefined
);
const list = toCollection(shopsAfterOffsetAndLimit, total, offset);
return jsonSuccess(res, list);
}
任何人都可以根据我的代码提供示例以某种方式对其进行分页但最后返回完整列表吗?
回答如下:更多推荐
如何绕过 Amazon Aurora 1MB 数据 API 限制?
发布评论