sysops

include:
– component: $CI_SERVER_FQDN/components/sast/[email protected]
– component: $CI_SERVER_FQDN/components/dependency-scanning/[email protected]
– component: $CI_SERVER_FQDN/components/secret-detection/[email protected]
– component: $CI_SERVER_FQDN/components/container-scanning/[email protected]

build-job:
stage: build
script: echo “Building the container image”

test-job:
stage: test
script: echo “Running unit tests”

deploy-job:
stage: deploy
script: echo “Deploying app”

Advantages and disadvantages of compliance frameworks
Ease of use
While compliance frameworks aren’t terribly complicated, they aren’t as straightforward as simple as pipeline includes. They’re meant to be written and assigned to projects by Amy and Priyanka, who now need to interact with pipeline YAML code. A framework needs to be declared in the top-level namespace and compliance pipelines need to be created and maintained, and compliance frameworks need to be attached to the right projects.
Customization
Amy and Priyanka are the authors of compliance pipelines. Like Sacha in the previous section on includes, they have full control over what they include and how they include it, giving them maximum customizability of compliance jobs such as security scanners.
Enforcement
This aspect of enforcing pipelines questions whether developers can tamper with security jobs? In an environment with a strong separation of duties, this nuance requires some extra attention. To answer this, we need to look at each pattern separately:
Wrapping pipelines
As seen before, project pipelines are included in compliance pipelines. In addition to group- or project-level CI/CD variables, every element of that project pipeline must be considered a potential threat to the compliance pipeline. Obviously, variables and jobs stick out as primary candidates. And, in fact, they can and will influence security job behavior if used maliciously.
Here is a simple example to illustrate the issue.
Compliance pipeline:
include:
– template: Jobs/SAST.gitlab-ci.yml
– template: Jobs/Secret-Detection.gitlab-ci.yml
– project: ‘$CI_PROJECT_PATH’
file: ‘$CI_CONFIG_PATH’
ref: ‘$CI_COMMIT_SHA’

Project pipeline:
variables:
SECRET_DETECTION_DISABLED: true

semgrep-sast:
rules:
– when: never

This project pipeline declares a variable SECRET_DETECTION_DISABLED (this could be done via project or croup-level CI/CD variables, too), which is evaluated in the included secret detection template. Further, the last three lines use the merging mechanism discussed previously, to not execute the job at all. Kind of redundant, we know.
Both overrides could be prevented using components, but you get the idea. Components, too, are receptive to such attacks via their inputs’ default values, which often use variables, too! Let’s take a look at how this could be taken advantage of.
Compliance pipeline:
include:
– component: $CI_SERVER_FQDN/components/sast/[email protected]
– component: $CI_SERVER_FQDN/components/secret-detection/[email protected]
– project: ‘$CI_PROJECT_PATH’
file: ‘$CI_CONFIG_PATH’
ref: ‘$CI_COMMIT_SHA’

Project pipeline:
variables:
CI_TEMPLATE_REGISTRY_HOST: “docker.io”

To understand what is happening here, look at the SAST scanner component’s Line 6:
spec:
inputs:
stage:
default: test
image_prefix:
default: “$CI_TEMPLATE_REGISTRY_HOST/security-products”

The image_prefix input uses the CI_TEMPLATE_REGISTRY_HOST to build the default value. By setting this variable to a false value in the same way we set SECRET_DETECTION_DISABLED to true before, Sacha may cause the job to load a wrong image and break SAST testing.
To prevent this override ability by the developer role, avoid templates in favor of components. This approach covers many developer-induced loopholes. To be certain of compliance, hardcode values for component inputs.
Overriding pipelines
This type is an entirely different beast. Developers get no chance of injecting actual pipeline code into the compliance pipeline. However, compliance pipelines do run with the project’s CI/CD variables. Hence, any variable specified on the group- or project-level might modify the compliance pipeline’s behavior. With SECRET_DETECTION_DISABLED set to true in the project CI/CD variables, the following compliance pipeline can be modified again:
stages: [“build”, “test”, “deploy”]

include:
– template: Jobs/SAST.gitlab-ci.yml
– template: Jobs/Secret-Detection.gitlab-ci.yml

build-job: …
test-job: …
deploy-job: …

Components can solve this particular problem, but, as before, component inputs may use CI/CD variables developers can set. Compliance pipeline authors need to identify and take care of these situations.
Policies
Compliance frameworks shortcomings have led to the next step for managing compliance: policies.
GitLab introduced policies as the way forward. Authors store a set of policies in a separate project as YAML files and apply them to projects on the group or project level. This gives Amy and Priyanka the flexibility to target individual projects with specific requirements but also to ensure compliance across the entire organization if needed. Access to the policy project can be controlled within the policy project and audited within GitLab.
Policies come in different types for different purposes. The types we are interested in right now are scan execution policies (SEP) and pipeline execution policies (PEP).
Scan execution policies
As the name suggests, SEPs require a particular scan – or set of scans – to be executed as part of the project pipeline and inject the respective scan jobs into the pipelines of associated projects. They include the respective template in the pipeline according to variables and rules set by Amy and Priyanka.
GitLab supports policy authors with a comprehensive user interface in addition to a YAML-based Git workflow. The following screenshot and code snippet illustrate a very basic example of a SEP:

name: Secret Scanner
description: ”
enabled: true
actions:
– scan: secret_detection
rules:
– type: pipeline
branches:
– “*”

For more details on SEP settings in the UI and YAML, please refer to the policy documentation.
Advantages and disadvantages of scan execution policies
Ease of use
SEPs provide a lightweight, easy-to-use mechanism that enforces security on existing and new CI/CD pipelines across the organization or on a granular level. The UI support makes them a viable tool for all relevant personas.
Customization
SEPs are restricted to predefined scanner jobs, and there is no option to extend this list with custom jobs at this point. This limitation can be restrictive for teams with unique scanning requirements that fall outside the standard options.
Enforcement
Once an SEP is applied to a project (directly or indirectly), Sacha has no way to get rid of that scan job. Though, there may be ways to – intentionally or not – manipulate the scan job’s behavior.
Jobs injected via SEPs generally are receptive to CI/CD variables and adhere to the general rules of variable precedence. For this injection, Policies incorporate logic that denies changing some predefined variables as described here and generally deny the configuration of variables that follow certain patterns such as _DISABLED or _EXCLUDED_PATHS.
Despite these security measures, inconsiderate use of policies may still open opportunities for tampering: In my test, I was able to set a project-level CI/CD variable SECURE_ANALYZERS_PREFIX to a bad value (a non-existing location) and as you can see here, the secret detection template uses that to build the location of the scanner image.
While the scan job does get included in the pipeline run, it crashes very early and, therefore, provides no scan results. Due to the allow_failure: true configuration, the pipeline will continue to run and eventually execute a deploy job.
Because SEP variables take the highest variable precedence, there is an easy fix to reduce the attack surface of the policy: Simply hardcode the correct value in your policy YAML or via the UI:
– name: Secret Scanner
actions:
– scan: secret_detection
variables:
SECURE_ANALYZERS_PREFIX: registry.gitlab.com/security-products

Pipeline execution policies
SEPs enable the injection of a set of security-related jobs into any project pipeline. In contrast, PEPs apply entire pipeline configurations to projects, offering a lot more flexibility when it comes to customizing security constraints.
There are two methods for implementing these policies, known as “actions”: inject and override. These actions function similarly to the patterns we have seen in the compliance frameworks section and provide flexible ways to enhance and enforce security standards within the development workflow.
Injecting pipelines
Injecting pipelines involves adding the jobs and other elements defined in the policy pipeline into the project pipeline. Currently, jobs should only be injected into reserved stages, namely .pipeline-policy-pre and .pipeline-policy-post to avoid unpredictable results.
GitLab handles name clashes between jobs or variables in policy and project pipelines effectively by building each pipeline in isolation before combining them. This ensures that the integration process is seamless and does not disrupt existing workflows or configurations.

The above screenshot shows an example of an injected policy pipeline. Project pipeline jobs are prefixed with prj- for easier identification.
Overriding pipelines
In the override approach, the project pipeline is completely replaced by the policy pipeline. This method is similar to compliance pipelines that do not include the project’s .gitlab-ci.yml file. Despite the override, the pipelines run using the project’s CI/CD variables, maintaining consistency with project-specific configurations. The compliance pipeline we used earlier makes a perfectly fine policy pipeline, too:
stages: [“build”, “test”, “deploy”]

build-job:
stage: build
script: echo “Building the container image”

test-job:
stage: test
script: echo “Running unit tests”

deploy-job:
stage: deploy
script: echo “Deploying app”

The image below shows a slightly more complete pipeline than the mock pipeline above:

Note: This doesn’t currently work with SEPs.
However, the existence of a Dockerfile may not always be a valid indicator, as developers might be building without Dockerfiles using Cloud Native Buildpacks, Heroku Buildpacks, Kaniko, or other tools. Managed pipelines do not encounter this challenge, as they are more controlled and centralized.

Projects with multiple container images
For projects that produce multiple container images, several container scanning jobs would be necessary for proper coverage. This raises similar questions as before: “How do we know there are multiple?” and “Is the source of that information trustworthy?”. If we wanted to rely on the existence of Dockerfiles a dynamic approach would be necessary that includes a container scanning job for each Dockerfile detected.
Get started with security scanning
In this article, you’ve learned about a variety of approaches to adding security scanning to CI/CD pipelines with a close look at ease of use, customizability, and the ability to strictly enforce scanning. You’ve seen that a pipeline author who is held responsible for project compliance needs to keep a few things in mind during the process to avoid surprises down the line. We recommend building a small testing space on your GitLab instance and then run a few tests to reproduce the main points of this article. Put yourself in the shoes of a malicious Sacha (Sachas aren’t generally malicious people, but it’s a good exercise) and think about how you could fool that annoying Amy and her security scans.
GitLab provides strong support for all sorts of requirements and all approaches are – at least in our eyes – easy to implement due the platform’s baked-in functionality. You should find ways to bulletproof your scan jobs and, if not, you should open a ticket with our support.
Happy pipelining!

Get started with security scanning today!
Sign up for a free 30-day trial of GitLab Ultimate to implement security scanning in your software development lifecycle.

Meet regulatory standards with GitLab security and compliance
How to integrate custom security scanners into GitLab
Integrate external security scanners into your DevSecOps workflow”]

Related Posts