GitLab CI Job Stuck in Pending — Runner Never Picks It Up | Today I Fixed

What I Was Seeing

Pushed a commit. Pipeline created. Job sat in pending state for 10 minutes with the message "Waiting for this job to be picked up by a runner."

No error. No timeout. Just... waiting.

Why This Happens

GitLab CI jobs go pending when no available runner matches the job's requirements. There are four common causes:

Runner is offline — it crashed, was deleted, or the runner process stopped
Tag mismatch — job has a tags: block but no registered runner has that tag
Runner is busy — all runners are at concurrent limit (running max jobs)
Runner locked to a different project — runner was registered as project-specific, not shared

How I Diagnosed It

First, check what runners are available:

GitLab → Project → Settings → CI/CD → Runners

I found my runner showed as offline (grey dot, not green).

bash

# On the runner machine, check if the process is running
sudo gitlab-runner status
 
# If not running:
sudo gitlab-runner start

It was stopped. sudo gitlab-runner start fixed it immediately.

If the Runner Is Online But Jobs Still Pending

Check for tag mismatch:

yaml

# Job in .gitlab-ci.yml
deploy:
  tags:
    - production-k8s   # ← This tag must exist on a runner
  script:
    - kubectl apply -f manifests/

Go to Settings → CI/CD → Runners and verify the runner has the production-k8s tag. If not, either add the tag to the runner or remove the tags: block from the job.

Check concurrent limit:

bash

# On runner machine
cat /etc/gitlab-runner/config.toml | grep concurrent

toml

concurrent = 1   # ← Only 1 job at a time — increase this

If you have 5 jobs queuing and concurrent is 1, all but one will sit in pending:

toml

concurrent = 4   # Increase to match your runner capacity

Restart runner after changing config:

bash

sudo gitlab-runner restart

Check runner scope (locked to project):

In Settings → CI/CD → Runners, if a runner shows "This runner is available to all projects in this group/instance" it's shared. If it says "This runner is locked to the current project" it won't pick up jobs from other projects.

The Fix That Worked

bash

sudo gitlab-runner start

Runner was stopped. Starting it picked up the pending job within 30 seconds.

Prevention

Add a health check for your GitLab runner. If you're self-hosting, this systemd service file ensures the runner restarts automatically:

bash

# The gitlab-runner package sets this up automatically, but verify:
sudo systemctl is-enabled gitlab-runner
# Should return: enabled
 
# If not enabled:
sudo systemctl enable gitlab-runner

Also set up a GitLab alert or monitoring check — a runner that's been offline for more than 5 minutes should page someone.