线上踩坑】Spring Cloud Hystrix线程池不足"/>
【线上踩坑】Spring Cloud Hystrix线程池不足
现象:
昨天突然线上很多接口获取失败,通过 kibana发现大量异常,具体异常信息:
...into fallback. Rejected command because thread-pool queueSize is at rejection threshold.
异常代码出处:
@FeignClient(name = "api", fallbackFactory = LoadBalancingFallbackFactory.class)
public interface LoadBalancingFeignClient {@PostMapping(value = "/api/loadBalancing/server")Result currentServer();}@Slf4j
@Component
public class LoadBalancingFallbackFactory implements FallbackFactory<LoadBalancingFeignClient> {@Overridepublic LoadBalancingFeignClient create(Throwable throwable) {final String msg = throwable.getMessage();return () -> {log.error("loadBalancingFeignClient currentServer into fallback. {}", msg);return Result.error();};****}}
原因:
看到这里已经很明显了,是由于hystrix线程池不够用,直接熔断导致的。项目apollo配置:
hystrixmand.default.execution.isolation.thread.timeoutInMilliseconds = 3500
hystrix.threadpool.default.maxQueueSize = 60
hystrix.threadpool.default.queueSizeRejectionThreshold = 40
hystrix参数简析:
maxQueueSize:线程池大小,默认为-1,创建的队列是SynchronousQueue,如果设置大于0则根据其大小创建LinkedBlockingQueue。
queueSizeRejectionThreshold:动态控制线程池队列的上限,即使maxQueueSize没有达到,达到queueSizeRejectionThreshold该值后,请求也会被拒绝,默认值5
相关源码:
hystrix-core-1.5.12-sources.jar!/com/netflix/hystrix/strategy/concurrency/HystrixContextScheduler.java
private class HystrixContextSchedulerWorker extends Worker {private final Worker worker;private HystrixContextSchedulerWorker(Worker actualWorker) {this.worker = actualWorker;}@Overridepublic void unsubscribe() {worker.unsubscribe();}@Overridepublic boolean isUnsubscribed() {return worker.isUnsubscribed();}@Overridepublic Subscription schedule(Action0 action, long delayTime, TimeUnit unit) {if (threadPool != null) {if (!threadPool.isQueueSpaceAvailable()) {throw new RejectedExecutionException("Rejected command because thread-pool queueSize is at rejection threshold.");}}return worker.schedule(new HystrixContexSchedulerAction(concurrencyStrategy, action), delayTime, unit);}@Overridepublic Subscription schedule(Action0 action) {if (threadPool != null) {if (!threadPool.isQueueSpaceAvailable()) {throw new RejectedExecutionException("Rejected command because thread-pool queueSize is at rejection threshold.");}}return worker.schedule(new HystrixContexSchedulerAction(concurrencyStrategy, action));}}
解决办法:
- 适当调大Hystrix线程队列参数
- 动态水平扩容服务
- 优化下游服务,减少服务响应时间
【线上踩坑】系列主要是用于简单记录工作中实际遇到的线上问题,如需要更深入的学习和了解,可以直接联系作者。
更多推荐
【线上踩坑】Spring Cloud Hystrix线程池不足
发布评论