Open source software security challenges persist

This year's Equifax breach was a reminder that open source software and components pose a giant risk to enterprise security despite their many benefits, especially when not properly maintained.

In April, researchers at Flashpoint Intelligence said criminals were using brute-force password attacks against the popular open source Magento ecommerce platform, leveraging the compromised access to scrape credit card records and install malware focused on cryptocurrency mining.

The researchers discovered at least 1,000 compromised Magento admin panels and said interest in the platform on the deep web and dark web has continued unabated since 2016. Moreover, there is also a noted interest in Powerfront CMS and OpenCart.

Open source code has grown in popularity over the years and is used by companies of all sizes, in all industry verticals.

Aside from the widely-known open source operating systems on the market, enterprise users also leverage open source productivity software, tools for administrators and developers, and various code libraries used to build their own software. Even commercial software is typically built on a foundation of open source code.

"I'm seeing more widespread adoption of open source software in the enterprise," says Andrew Howard, CTO at Kudelski Security. "As enterprises are moving to agile methodologies, open source becomes more valuable, and there are more tools available to them. If you look at new software developers entering the market today, they are trained to be very comfortable with open source technologies."

Open source security advantages

Developers rely heavily on open source software and companies are especially comfortable with major open source projects that have large groups maintaining them, says Howard. Plus, there's the "many eyes" approach to security. "That is the major advantage of using open source software -- other than it being lower cost," he says. "You theoretically have more eyes looking at it." (Though he adds that this doesn't apply to smaller projects or code libraries. "Some software has no community whatsoever," he says.)

Another security advantage of open source code is that if there's a problem, a company can open it up and fix it immediately. "If the code is licensed under proprietary agreements, they generally have to wait for vendors to respond," says Mel Llaguno, open source solution manager at Synopsys, Inc.

Why open source software poses a security threat

Synopsys manages Coverity Scan, a free service that scans open source code for defects. "Overall, the quality of open source software has been improving," Llaguno says. "We have about 750 million lines of open source code that participate in our scan projects, and identified 1.1 million defects -- and 650,000 defects have already been addressed." He adds that many projects, especially smaller ones, do not scan their code for potential security weaknesses.

Black Duck Software, Inc., for example, tracks more than 10 billion lines of open source code in more than 550,000 projects. Even that isn't a complete picture. The Linux Foundation reports that 31 billion lines of code have been committed to open source repositories.

Who's using all that open source code? Everybody. According to the latest Black Duck report, open source components are now present in 96 percent of commercial applications. The average application had 147 different open source components -- and 67 percent of the applications used components with known vulnerabilities.

The US government sponsors the Common Vulnerability Enumeration list and the National Vulnerability Database. In 2017, more than 8,000 new vulnerabilities were added to the CVE list, a record high.

So why are companies using open source software? "In the average application, over a third of the code base is open source," says Mike Pittenger, Black Duck security strategist at Synopsys, Inc. "To replace that third of the code base, you're going to have to increase either your development team or development time by 50 percent -- and I don't think those are viable options in today's world."

Take, for instance, London-based Skyscanner Ltd., which makes a travel search engine. The company used to run on proprietary, closed platforms like .NET. Over the past few years, it's migrated to a wider range of languages and technologies, including open source, says Skyscanner's security engineer Alex Harriss. "This has meant that our engineers are now able to pull dependencies from multiple sources and deploy their code in minutes," he says.

This has also created security challenges, Harriss says. "I think there's a misguided reliance on the fact that, being open source, these libraries are being reviewed for security bugs by the community," he says. "In reality, it seems this isn't always the case."

There are well-documented vulnerabilities in many of the most common libraries, he says. Since engineers would just pull code from these libraries and deploy it, this created a visibility problem. "We were simply unaware of the number of dependencies in our products," he says.

More due diligence on open source software security needed

Skyscanner wasn't alone. According to the latest Veracode report, only 28 percent of organizations do any kind of regular analysis to find out what components are built into their applications. As the use of open source code grows, this risk surface expands.

New vulnerabilities are constantly being found in open source code and many projects have no mechanisms in place for finding and fixing problems. According to a recent Snyk survey of open source maintainers, 44 percent have never had a security audit, and only 17 percent says that they had a high level of security know-how.

There's also no standard way of documenting security on open source projects. In the top 400,000 public repositories on GitHub, only 2.4 percent had security documentation in place.

Then, if the problem is fixed, there's often no way to find and notify all of the users of the old code. "The open source community has no idea of who is using their components," says Black Duck's Pittenger.

According to the Snyk survey, 88 percent of open source code maintainers add security-related announcements to the release notes, and 34 percent say that they deprecate the older, insecure version. Twenty-five percent say that they make no effort at all to notify users of vulnerabilities and only 10 percent file a CVE.

Many aren't aware of how the CVE process works, or don't have the time to go through it, says Guy Podjarny, CEO and co-founder at Snyk Ltd. "In Javascript, for example, only 13 percent of the vulnerabilities we track have a CVE number," he says. "The rest are just logged as bugs."

Snyk has a security research team that looks for signs of security problems in open source libraries by looking for clues in places such as the release notes and the GitHub and Apache issue tracking systems. The results are published in its vulnerability database, and, when possible, Snyk also submits them to the CVE list.

However, getting a CVE can be a complicated process, and requires a committee to agree on the CVE details, as well as agreement from the project owner. "The way the method currently works doesn't scale," says Podjarny.

Finally, if a vulnerability is found and patched, and the patch is broadly publicized, enterprises that use that code might not be aware that they have it or may have problems finding all instances of it. This year's giant Equifax breach, for example, involved a vulnerability in the Apache Struts open source software. The patch came out a couple of months before the breach occurred, and Equifax was aware of the patch, but was still not able to make the fixes in time.

Another problem is that some companies are running older versions of the code, and are unable to move to the latest version because of compatibility issues, compliance, or other reasons. According to Snyk, only 16 percent of vulnerability fixes are backported to other versions.

Find and fix

In an ideal world, applications would all update themselves the instant a security patch became available, without any intervention required. In practice, however, this isn't always possible.

Instead, enterprises need to have a way to find all instances of open source code in their environments, to update this list continuously, to steer developers away from old, insecure libraries, and finally to go out and deploy patches whenever new vulnerabilities are discovered.

"Removing vulnerable libraries from your products is only possible when you know where they are, and the earlier you do it in the lifecycle, the cheaper and easier it is to do," says Skyscanner's Harriss.

Many companies turn to vendors like Snyk, Black Duck, and Veracode for help. That's what Skyscanner did, as well. "Snyk allowed us to see what packages were being used in which projects, the vulnerabilities they contained, and how they were introduced into our code," says Harriss. In addition, Snyk would immediately flag vulnerabilities to developers as they were writing code, so that the problems could be addressed before the code went into production, he says.

Integrating open source vulnerability scans into the development process is especially important for large enterprises, since it can be difficult to track down all the code that is in use. "Most companies we deal with do not know the full inventory of applications they have, which is a little scary," says Chris Eng, VP of research at Veracode.

When Veracode does a vulnerability scan, the companies upload their binary code, and Veracode checks it against the National Vulnerability Database. To help companies discover what applications they have running that they might not know about, Veracode can also scan the company's perimeter. "It's not going to find other internal apps that are not exposed, but those are going to be lower risk anyway because of the exposure," Eng says.

There are also network tools that companies can use to find out what's running internally, he added, but there can be blind spots if the network is segmented. Companies can also install agents on endpoint devices to track what's running. "And you might still have some blind spots if you don't have the agent installed everywhere," Eng says. "There's no one single way to get all this, which is why it's such a difficult problem."

It was certainly a difficult problem for Equifax. In October, former CEO Richard Smith told Congress that the company's information security department ran scans looking for the Apache Struts vulnerability that eventually led to the breach. The scans "did not identify any versions of Apache Struts that were subject to this vulnerability, and the vulnerability remained in an Equifax web application much longer than it should have," he said.

"You have to make sure you know every server in your environment to scan it with this tool," says Black Duck's Pittenger. Even when the scans are complete and accurate, they create a great deal of management overhead for organizations. Without security checks built into the development process, these scans have to be running continuously, and applications individually checked and rechecked for vulnerabilities.

Not only do developers introduce new vulnerable libraries to their applications as they update them, but new vulnerabilities are discovered in old libraries that were previously thought secure. "Software doesn't age like wine," Eng says. "It ages like milk."

Developers rarely go back and review the libraries they used in old projects, Eng says. "I download whatever version is current today, incorporate it into my app, and just never think about it again," he says.

In a sign of how important this topic is right now, Black Duck was purchased by Synopsys, Inc. this month for over $500 million. "That's a big market validation, I would say," says John Dickson, principal at Denim Group, Ltd., who says he was surprised by the size of the deal. "I thought it was a hot space, but I didn't think it was that hot. That acquisition turned a lot of heads."