queryForList默认查询10条每次,最多可设置为10000条,意味着该api最多只能查询10000条数数据
elasticsearchTemplate.queryForList(query, ComparisonPlatformGoodsItemSearchView.class);
类似同步之类的逻辑,es特定的业务索引几百万乃至上千万的数据,显然,仅仅通过elasticsearchTemplate.queryForList()是无法满足的。
2、解决方案: 第一种:通过scroll(类似mysql游标)官方手册:https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.scroll
(注意:spring-data-elasticsearch不同版本api名称不一样,4.x.x和3.x.x存在不一样,诸如:searchScrollStart/scrollStart、searchScrollContinue/scrollContinue等)
代码示例:
public void createPurchaseActivityConfigurationGoodsSearchData(PurchaseActivityConfiguration paramBean) { BoolQueryBuilder boolQuery = QueryBuilders.boolQuery() .must(QueryBuilders.matchQuery("brandId", paramBean.getBrandId())) .must(QueryBuilders.matchQuery("comparisonCategoryId", paramBean.getCategoryId())); NativeSearchQuery query = new NativeSearchQueryBuilder().withQuery(boolQuery).build(); List<ComparisonPlatformGoodsItemSearchView> goodsItemViews = new ArrayList<>(); ScrolledPage scroll = elasticsearchTemplate.startScroll(SCROLL_TIME, query, ComparisonPlatformGoodsItemSearchView.class); while (scroll.hasContent()) { goodsItemViews.addAll(scroll.getContent()); scroll = elasticsearchTemplate.continueScroll(scroll.getScrollId(), SCROLL_TIME, ComparisonPlatformGoodsItemSearchView.class); } elasticsearchTemplate.clearScroll(scroll.getScrollId()); if (ObjectUtil.isNotNull(paramBean.getCommissionRatioThreshold())) { purchaseHighSubsidyGoodsSearchRepository.saveAll(covertPurchaseHighSubsidyGoodsSearchView(goodsItemViews)); log.info(">>>>>> createPurchaseHighSubsidyGoodsSearchData success"); } }第二种:通过Stream