Have you ever been debugging a pipeline job and keep having to go add bash commands, like set -x or a thousand echo statements to figure out what is going on?

Well there's an easier way! You can simply add a CI/CD variable to your project (or group) and automatically get debug output without having to make a branch/MR or changes to your scripts!

CI_DEBUG_TRACE¶

You may already know there are a number of Predefined CI/CD variables in GitLab. If not, browse through because there are some really useful ones in here, such as the one I am about to share with you!

If you take note of the CI_DEBUG_TRACE variable, this is a boolean (true/false) flag to enable debug logging in your pipeline jobs.

Now you could set this in your CI file(s)/job(s), sure. What if you want to debug a pipeline that is already in main or running from a release that may take multiple reviews even for small changes and stuff? We've all been there where the "dammit, the MR and reviews will be more of a pain than the fixing stuff!" so let's avoid that pain.

First, navigate to your project in GitLab and select Settings > CI/CD from the navigation bar on the left. CI/CD Settings

You can then select the Add variable button in this page.

Next you want to make sure to uncheck the Protect variable option (or it will nerf your ability to test in branches and stuff if you have CI rules in place preventing it).

Set the Key as CI_DEBUG_TRACE and Value to true Set value

Once this is set, the environment variable CI_DEBUG_TRACE=true will automatically be exported in your jobs.

Below you can see an example of running before I added the variable and had a terraform init command failing while building a CI/CD component/testing, and then the output showing what was wrong when I added the CI_DEBUG_TRACE variable to my project.

BEFORE No Debug

AFTER Debug

For reference, it was the use of inputs in GitLab CI/CD components as an array and trying to do $[[ inputs.init-args ]] and pull them in that resulted in leftover ' single quotes around each argument that made it an invalid terraform command.

This helped to find the actual rendered commands instead of wondering "WTF? those are too valid commands!".

Ah, the fun of pipelines...

Summary¶

That's it! A super simple way to enable verbose debugging without adding nonsense to your .gitlab-ci.yml (or other templates) and without having to make a branch/commit/merge to inspect things (and then inevitably forget to clean it up and be echoing stuff you shouldn't....)

Hope this helps someone and have fun debugging!

Frequently Asked Questions¶

Q: Will CI_DEBUG_TRACE work with GitLab SaaS runners?¶

A: Yes! CI_DEBUG_TRACE works with both GitLab.com SaaS runners and self-hosted GitLab runners.

Q: Does this affect pipeline performance?¶

A: Debug output adds minimal overhead. The main impact is longer log files, but execution time remains virtually the same.

Q: Can I use this for specific jobs only?¶

A: Yes! Set CI_DEBUG_TRACE as a job-level variable in your .gitlab-ci.yml instead of at the project level:

job_name:
 variables:
 CI_DEBUG_TRACE: "true"
 script:
 - your_commands_here

Q: What if my logs become too verbose?¶

A: You can temporarily disable by setting the variable value to false or deleting it entirely from your project settings.

Q: Does this work with Docker executors?¶

A: Absolutely! CI_DEBUG_TRACE works with all GitLab executor types: Docker, Shell, Kubernetes, and more.