6 Best Practices to Secure Your CI/CD Pipeline

Recently, as cyberattacks that exploit weaknesses in the Continuous Integration/Continuous Delivery (CI/CD) pipeline and developer tools have occurred, the need to strengthen the security of the developer infrastructure has emerged. In particular, the Codecov supply chain attack is a strong warning against storing confidential information in CI/CD environment variables, no matter how secure the environment.
ⓒ Getty Images Bank

By compromising the Bash uploader used by millions of developers, Codecove attackers stole login information, keys and API tokens from customer environments without being detected for two months. In addition, hundreds of limited customer networks were reportedly compromised.

There are also attacks targeting cloud-native container environments and automation tools such as Jenkins and GitHub Actions, requiring enterprises to build effective defenses against these tools. This article introduces some best practices for securing your CI/CD pipeline.

1. Do not store confidential information in CI/CD environment
The reason Codecove’s supply chain attacks have been so successful is that the attackers leaked environment variables that contained hard-coded confidential information, including passwords, tokens, and keys. Attackers also used some of these credentials to gain access to the company’s private GitHub repositories, which could lead to further data breaches from those private repositories containing data that should be kept confidential.

Codecove customers, including HashiCorp, Twilio, Rapid7 and Monday.com, did not disclose the impact of the supply chain attack, but the most significant data breaches known to date are in Japan. It happened at the e-commerce giant Mercar.

Since the Codecove attack, more than 27,000 records pertaining to Murkary’s customers’ finances, transactions, business partners, company employees, contractors and various entities were exposed to unauthorized external actors.

Each of these attacks likely originated from a Codecove breach. Some also questioned why personally identifiable information (PII), such as customer financial records, was stored in private GitHub repositories in the first place.

Similar concerns exist regarding Hashcorp’s GPG private key stored in a CI/CD environment. GPG is the secret key used to sign and verify software releases published by Hashcorp. An attacker could have exploited the leaked key to insert Hashcorp’s signature into a malicious software release before it was revoked. One developer tweeted: “Why is no one talking about why Hashcorp, the creator of the Vault, has stored their signing keys as ENVs? I am better than that,” he said.

Organizations should rethink their CI/CD tools, environment variables, and confidential information to store in private GitHub repositories. If your application needs to store credentials or tokens in these locations for operation, your best bet is to store them in an account or resource that has the least privileges required to perform that task. This is commonly referred to as the principle of least privilege. This will help contain damage even if your secrets are exposed in an unexpected attack.

2. Thorough investigation of automated pull requests and scheduled tasks
CI/CD automation tools like GitHub Actions allow developers to set up scheduled actions for code repositories, for example to automatically triage and process incoming pull requests. But what if a contributor sending a pull request to an open source project has malicious intent?

In April 2021, attackers exploited GitHub actions by sending automated pull requests to hundreds of repositories for the purpose of mining cryptocurrencies using the GitHub infrastructure. This large-scale attack occurred after a ‘glitch’ in the GitHub action was reported earlier in February.

At a minimum, these pull requests can exploit GitHub’s servers to mine cryptocurrencies or execute attackers’ malware. Merging these pull requests without a project owner’s attention introduces malware into repositories and the broader software supply chain. GitLab also reported in May that a similar cryptomining attack occurred on its platform, where attackers exploited the “free time” allotted to new accounts.

Since the properties of CI/CD automation tools such as Github Actions and GitLab are to easily automate key tasks, it is difficult to protect them. The intended functionality becomes a security flaw and is exploited by threat actors.

GitHub recently announced a new feature that prevents exploitation of the action platform by cryptomining attackers. “A pull request from the first contributor requires manual approval from a repository collaborator with write access before the action workflow is executed,” said Chris Patterson, GitHub Product Manager. When the first contributor opens a pull request, the admin will need to approve the action workflow and then a message will appear stating that the workflow can be run after that.”

3. Harden your cloud-native containers and audit them regularly
It’s best to follow standard best practices, such as properly configuring production containers and hardening them against common attack vectors. This also includes protecting the pipeline configuration.

However, there are cases where simple misconfiguration goes unnoticed. The key is to ensure that the Docker-based environment is free from vulnerabilities. That’s why it’s still helpful to regularly perform vulnerability security audits of containers and scan container images and manifest files for common security issues.

Invest in a trusted cloud-native container security solution that can automate much of this process. This is because it is virtually impossible for one person to track the vast number of security vulnerabilities reported every year.

In addition, when enterprises deploy applications by adopting the Kubernetes framework and Docker containers, a container security solution with a built-in web application firewall can detect and block suspicious network traffic at an early stage. This is effective in preventing larger breaches even if an attacker penetrates the container and gains initial access.

4. Automate code quality checks by integrating deep code scanning
Using a tool that automatically finds code quality issues, security vulnerabilities, and bugs such as memory leaks or race conditions before code commits reach production is an effective strategy for protecting your CI/CD pipeline from the start.

Although the focus is on stopping cyberattacks, even a malicious bug can have as far-reaching impact as an attack. The Fastly incident, which recently caused the outages of major sites around the world, proves this.

Solutions like the GitHub code scanner or Sonatype’s Lift integrate seamlessly into existing coding workflows and provide developers with basic safeguards at no cost. Ultimately, the organization’s goal is to keep bugs or security vulnerabilities in their applications as low as possible while helping developers do their best work. In the process, friction between the development team and the security team should also be minimized. Having the ability to notify developers in real-time of what they might have missed while coding saves everyone’s time and protects the entire CI/CD workflow from the start.

5. Quick patch for latest CI/CD tool vulnerabilities
In March 2021, attackers mined Monero cryptocurrency from vulnerable Jenkins and ElasticSearch servers with a cryptomining botnet called z0Miner. Attackers attempted to exploit remote code execution (RCE) vulnerabilities in Internet-facing servers to infect and seize infrastructure to use for malicious activities.

Similarly, as reported last year, Jenkins servers can also be exploited by attackers to quickly cause Distributed Denial of Service (DDoS). This is an attack based on the UDP Amplified Reflection DoS vulnerability classified by the code CVE-2020-2100, affecting versions of Jenkins less than v2.219 and Jenkins LTS less than 2.204.1.

As soon as these critical vulnerabilities are discovered in automation tools and pipelines, patching them as quickly as possible is still critical to ensuring the security of your CI/CD infrastructure.

6. Check integrity before applying updates
Applying the latest updates and patches is good advice, but how can you be sure that the updates you receive have not been tampered with? The ‘keep up to date’ advice, a motto of security experts for decades, has been shaken after the SolarWinds supply chain attack.

In the SolarWinds incident, attackers were able to distribute malware to more than 18,000 customers through malicious updates that infiltrated Orion IT products. In addition, there was an incident where the ‘in-place upgrade function’ of the Passwordstate password manager was violated, and malicious updates were distributed to Passwordstate users. Therefore, you should also avoid blindly applying product updates.

In the Codecove case, a simple integrity check led to the discovery of a breach that lasted two months. A customer noticed a mismatch between the checksum (hash) of the Bash uploader hosted on the server and the normal checksum registered in Codecove’s GitHub repository, and immediately contacted Codecove to fix the problem.

Therefore, a defense in depth approach is needed to ensure the integrity of updates and patches and to prevent sophisticated supply chain attacks by inspecting downloads. [email protected]


Source: ITWorld Korea by www.itworld.co.kr.

*The article has been translated based on the content of ITWorld Korea by www.itworld.co.kr. If there is any problem regarding the content, copyright, please leave a report below the article. We will try to process as quickly as possible to protect the rights of the author. Thank you very much!

*We just want readers to access information more quickly and easily with other multilingual content, instead of information only available in a certain language.

*We always respect the copyright of the content of the author and always include the original link of the source article.If the author disagrees, just leave the report below the article, the article will be edited or deleted at the request of the author. Thanks very much! Best regards!