Have you ever been debugging a pipeline job and keep having to go add bash commands, like set -x or a thousand echo statements to figure out what is going on?

Well there's an easier way! You can simply add a CI/CD variable to your project (or group) and automatically get debug output without having to make a branch/MR or changes to your scripts!

CI_DEBUG_TRACE¶

You may already know there are a number of Predefined CI/CD variables in GitLab. If not, browse through because there are some really useful ones in here, such as the one I am about to share with you!

If you take note of the CI_DEBUG_TRACE variable, this is a boolean (true/false) flag to enable debug logging in your pipeline jobs.

Now you could set this in your CI file(s)/job(s), sure. What if you want to debug a pipeline that is already in main or running from a release that may take multiple reviews even for small changes and stuff? We've all been there where the "dammit, the MR and reviews will be more of a pain than the fixing stuff!" so let's avoid that pain.

First, navigate to your project in GitLab and select Settings > CI/CD from the navigation bar on the left.

You can then select the Add variable button in this page.

Next you want to make sure to uncheck the Protect variable option (or it will nerf your ability to test in branches and stuff if you have CI rules in place preventing it).

Set the Key as CI_DEBUG_TRACE and Value to true

Once this is set, the environment variable CI_DEBUG_TRACE=true will automatically be exported in your jobs.

Below you can see an example of running before I added the variable and had a terraform init command failing while building a CI/CD component/testing, and then the output showing what was wrong when I added the CI_DEBUG_TRACE variable to my project.

BEFORE

AFTER

For reference, it was the use of inputs in GitLab CI/CD components as an array and trying to do $[[ inputs.init-args ]] and pull them in that resulted in leftover ' single quotes around each argument that made it an invalid terraform command.

This helped to find the actual rendered commands instead of wondering "WTF? those are too valid commands!".

Ah, the fun of pipelines...

Summary¶

That's it! A super simple way to enable verbose debugging without adding nonsense to your .gitlab-ci.yml (or other templates) and without having to make a branch/commit/merge to inspect things (and then inevitably forget to clean it up and be echoing stuff you shouldn't....)

Hope this helps someone and have fun debugging!