@astepano could you please take a look what is going on? thanks :wink:

I think this is because Java on master process eats all available resources.
Jenkins fails check jenkins-runners.
We didn't find out why Java process is so high.
We have a new VPC in new AWS region.
Currently I cannot scale K8S resources.
Need to wait until we move to new VPC + make some optimization to existing configuration.

Note the problem is that kubernetes plugin is unable to create any pods. Currently there is 0 pods attached to Jenkins master. This started on Friday (evening I think). This was not a problem before.

And just to be clear: since no pods can be created, ELN + generic + functional pipelines will all either block (and builds pile up, waiting indefinitely), or timeout after couple of hours. So the Jenkins instance is basically unusable at the moment.

I believe we found cause for this. I sent PR/merged ELN pipelines. The reason for this 40K + 60K history length + EBS as storage backend for k8s nodes.