How lean development improved software security at Fannie Mae''

Continual improvement methodologies strengthen security, shorten dev cycles and help the c-suite see infosec’s value.

Want to reduce your release cycle from 203 to 100 days — and make it more secure? Fannie Mae did. Vice-president of development services Michael Garcia credits lean, a quality improvement philosophy focused on maximizing customer value while minimizing waste. It’s a tactic typically associated with manufacturers. Fannie Mae doesn’t make widgets, and to benefit, you don’t have to either. The lean mentality can apply to anything

Lean thinking shifts managerial focus across technologies and departments to optimize every value stream in your business providing products or services to customers. More simply, it’s figuring out how to do better with what you already have. Every business department — especially information security — should want to do better. “At the end of the day,” says Garcia (who left Fannie Mae shortly after our interview for another financial services firm), tech teams “are here to serve an organization and we need to do that in a way that's safe.” Security intrinsically delivers customer value and, regardless of how good of a job you do already, can always be improved.

Choosing a continuous improvement model

Whether optimization comes as the result of lean or from a competing system like Kaizen or Six Sigma, you must use some type of continuous improvement model if you want shorter development cycles that don’t compromise security, Garcia says. He promises results: Lean not only cut Fannie Mae’s cycle in half, but the company’s now developing more code. Garcia adds, “Quality has gone up by 50 percent over the same period of time.” Fannie Mae started using lean in 2013, but if it hadn’t, he continues, “It would have cost us hundreds of millions more dollars to deliver what we're delivering today.”

This all sounds great for Garcia, especially as when started his job, the approach already had buy-in from Fannie Mae’s CEO. Adoption was a top-down decision, he says, and mandatory for all departments. When you don’t have a CEO driving change, how can you implement the right improvement methodology for your team?

“Embrace agile, embrace debt-loss, embrace operational excellence.” Those are the starting points Garcia suggests. Most quality models break optimization down into steps like this. In the book Lean Thinking (Simon & Schuster, 2003), authors Jim Womack and Dan Jones recommend you focus changes across three areas: purpose, process and people.

For some companies, this might mean revamping organizational structure: How readily can development and security teams communicate? Do they simply Slack each other or does cross-departmental collaboration require an all-channels-approved, pre-scheduled, sit-down meeting? Simplifying communication processes saves people time so they can get back to the true purpose of their work.

The lean mentality: Developing secure code from the start

Looking for ways to cut waste across the three p’s is a great start. For those who want help beyond that, Lean Enterprise Institute and the American Society for Quality offer training. However, there are no hard-and-fast guidelines for thinking lean — no rules or roadmaps you can follow to perfection. That’s one reason some security officers might not like it: Lean isn’t a process. It’s a mentality.

The upside to little rules is that teams become empowered to find their own ways to be more agile. In a lean culture, self-efficiency and self-improvement are inherent. “Ironically, the interesting thing is that when you actually do get [the mentality] right, then you can deliver things — integrate it in a faster way,” Garcia says.

New code is designed more safely from its foundations, he explains, “because you're delivering smaller increments and you're testing them fast. You're testing them all the time and you also have a commitment to that.” Smaller pushes are easier to fix, and engineers can more readily learn from those iterated fixes. Integrated communication helps security catch vulnerabilities before a commit.

Six Sigma: A more structured approach

For those who prefer clearer parameters, though, there’s Six Sigma, a trademarked process that breaks optimization down into five steps: define, measure, analyze, improve and control. If you think that’s a lot to remember, there’s this handy acronym: DMAIC. It even comes with a map.

DMAGIC/Wikimedia

Under Six Sigma, if you wanted to make a code base more secure while it’s still under development, you’d have to follow these steps in order:

1. Define the problem: Development is writing unsecure code.
2. Measure — or quantify — their performance: Is just one line easy to hack or is the entire push bad?
3. Analyze to determine the problem’s root cause: Does development not know what they’re writing is vulnerable, or do they not care?
4. Improve addresses or eliminates the cause.
5. Control looks for ways to keep it from happening again.

Why Fannie Mae chose lean over Six Sigma

For Fannie Mae, Six Sigma requires too many steps. That 100-day development cycle was just for larger projects. Garcia says, “For many things, [lean is] much, much faster. I am basically responsible for delivering the agile DevOps transformation from the technology side, and we're partnered heavily with the business side on lean transformation.” Because Fannie Mae supports the mortgage industry, he adds, “We had to look at rationing down our heavy governance that we had and integrating that into our technology so it's automated and faster and more reliable.” Lean was right for them and he says, “The results show.”

In addition to improving communication among teams, Garcia says lean helped Fannie Mae reduce the number of checkpoints in its system development life cycle (SDLC), adding “more secure, automated controls.” He also says they developed “a risk acceptance process that takes into account the way in which people make decisions,” explaining that “people will identify things all the time and people will make risk decisions.”

If a full-blown system sounds ambitious, there’s also Bayesian decision theory, which pits probability against cost to measure the tradeoff of any potential decision. It, at least, can help you choose the most optimized option: “If you look at the Bayesian decision making,” Garcia says, “it's like, ‘Yes, you're right. Here’s a gap in that process, but we've been alright for 75 years and we've yet to have an issue…. There has to be a risk acceptance process that takes into account the way in which people make decisions."

Who knows? If management starts to see radical improvement coming from your team, maybe it will help them and other departments take security more seriously. Garcia says, “What we have found in our transformation of teams and the way we work has been fundamentally getting that trust-base to enter the process when our teams work together. So, you have the security folks, you have the infrastructure folks, you have the lawyers, you have the business people — you have them all look at these problems in a way that really focuses on one thing and that's the clients.” If you can cut waste and improve the security that customers receive, the business value will be undeniable.