Large enterprises scramble after supply-chain attack spills their secrets
tj-actions/changed-files corrupted to run credential-stealing memory scraper.
Open source software used by more than 23,000 organizations, some of them in large enterprises, was compromised with credential-stealing code after attackers gained unauthorized access to a maintainer account, in the latest open source supply-chain attack to roil the Internet.
The corrupted package, tj-actions/changed-files, is part of tj-actions, a collection of files that’s used by more than 23,000 organizations. Tj-actions is one of many GitHub Actions, a form of platform for streamlining software available on the open source developer platform. Actions are a core means of implementing what’s known as CI/CD, short for Continuous Integration and Continuous Deployment (or Continuous Delivery).
Scraping server memory at scale
On Friday or earlier, the source code for all versions of tj-actions/changed-files received unauthorized updates that changed the “tags” developers use to reference specific code versions. The tags pointed to a publicly available file that copies the internal memory of severs running it, searches for credentials, and writes them to a log. In the aftermath, many publicly accessible repositories running tj-actions ended up displaying their most sensitive credentials in logs anyone could view.
“The scary part of actions is that they can often modify the source code of the repository that is using them and access any secret variables associated with a workflow,” HD Moore, founder and CEO of runZero and an expert in open source security, said in an interview. “The most paranoid use of actions is to audit all of the source code, then pin the specific commit hash instead of the tag into the … the workflow, but this is a hassle.”
An overview of the malicious functioning of tj-actions/changed-files.
As the supply-chain attack demonstrates, many GitHub users weren’t following these best practices. Repositories using tj-actions that trusted tags rather than hashes of vetted versions ended up running the memory scraper/logger. The attack poses a possible threat to any such repository, because credentials should never appear in human-readable form. The risk is most acute for repositories that are publicly viewable, since the credentials are then viewable by anyone.
A tj-actions maintainer said Saturday that the attacker somehow compromised a credential a @tj-actions-bot uses to obtain privileged access to the compromised repository. The maintainer said it remained unclear how the credential was compromised. The password used by the bot has since been changed, and for added security, the account is now protected by a passkey, a form of credential that, as specified by the FIDO Alliance, requires two-factor authentication by default.
GitHub officials said in a statement that they have no evidence the company or its platform has been compromised.
“Out of an abundance of caution, we suspended user accounts and removed the content in accordance with GitHub’s Acceptable Use Policies,” the officials wrote. “We reinstated the account and restored the content after confirming that all malicious changes have been reverted and the source of compromise has been secured.” They went on to remind users they should “always review GitHub Actions or any other package that they are using in their code before they update to new versions.”
The supply-chain attack was first spotted by security firm StepSecurity, which said it came to notice through an “anomaly detection when an unexpected endpoint appeared in the network traffic.” The incident appeared to start around 9 am Saturday Pacific time.
In a separate writeup, researchers at security firm Wiz said preliminary analysis of the attack has already established that dozens of tj-actions users have faced real harm in the supply-chain attack. The researchers wrote:
While conducting threat hunting related to this malicious activity, in several instances Wiz Threat Research has observed the deployment of a script designed to dump secrets as part of the malicious payload’s execution. Additionally, Wiz Threat Research has so far identified dozens of repositories affected by the malicious GitHub action, including repos operated by large enterprise organizations. In these repositories, the malicious payload successfully executed and caused secrets to leak in workflow logs. Some of the leaked secrets we’ve identified so far include valid AWS access keys, GitHub Personal Access Tokens (PATs), npm [tokens], private RSA Keys and more.
The tj-actions incident is the latest example of a supply-chain attack on a widely used open source package. Last year, a lone developer working for Microsoft discovered the presence of a backdoor that had been intentionally planted in xz Utils, an open source data-compression utility used by millions of organizations, many of them Fortune 500 companies. In a stroke of luck, the backdoor, which gave the attackers the ability to log in to any server with privileged access, was discovered just weeks before it was scheduled to go into production versions of Linux. Other recent supply-chain attacks have been covered here and here.
Anyone responsible for a system that uses tj-actions should carefully inspect their systems to check for signs of compromise. The supply-chain attack should also serve as impetus for admins to review any GitHub Actions they use to ensure they use cryptographic hashes, instead of tags, that point to code that has been vetted previously. The above-linked posts from StepSecurity and Wiz provide useful guidance, as does this one from Semgrep.
Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. Follow him at here on Mastodon and here on Bluesky. Contact him on Signal at DanArs.82.
