Using existing idle runners

[UbiquitousBear]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

ARC (Actions Runner Controller)

Discussion Details

I want to reduce the latency for a new pod to be used / started up.

With my ARC GH Runner Scale Set install on k8s; I’ve set minRunners to 2. What I’m seeing is that when a new workflow job is requested, a new pod is spun up, and the oldest pod is deleted, regardless of whether it was used or not.

Ideally, a pod which wasn’t used would be spun up. minRunners should keep idle runner active, especially those which haven’t run a job. It’d be even better if once a job has been assigned and is running, a new runner is spun up to meet the minRunners once the running runner has finished and can be terminated.

[yashasam]

Review ARC documentation and settings:
Max Runners: Ensure that your maxRunners setting is configured appropriately. If it's set too low, it might be preventing ARC from maintaining enough idle pods to handle new jobs without replacing existing ones.
Runner Scale Set Configuration: Review the ARC documentation on configuring runner scale sets for any relevant settings or best practices that could influence pod recycling behavior.
Monitor pod lifecycle events:
Use kubectl describe pod to investigate the reasons behind pod deletion and termination. Look for events, termination messages, or container lifecycle details that provide clues about the specific actions and decisions made by ARC and Kubernetes.
Investigate Kubernetes scheduler behavior:
Understand how Kubernetes' scheduler selects which pods to terminate when scaling down. Factors like Pod Deletion Cost (if enabled) can influence this process.
Explore advanced ARC configurations:
The ARC documentation details advanced configuration options, including customizing the PodSpec for runner pods. You might be able to fine-tune the pod lifecycle, resource requests, and termination behavior by modifying the pod specification to better align with your desired outcome, says the GitHub Docs.
Consider using init containers:
While not directly related to your primary issue, using init containers in your runner pod template could be useful for pre-initializing or setting up a consistent environment for new runner pods, potentially reducing the overall startup time, mentions GitHub Docs.

[UbiquitousBear]

@yashasam, yes I’ve set maxRunners at 50 as that’s about right for our org, but we rarest go above 30.

It does look like I was seeing something else - a workflow job takes around 15 seconds in queing before it’s pickup up by a runner. Once the ARC / listener has picked the job up, everything is fast.