By Bob Lord, Yahoo Paranoid in Chief (@boblord)
Recent headlines might lead you to believe that when a company runs a red team exercise that the red team should fail. After all, the company has invested in security teams, products and processes. So the outcome should be a win for the blue team and a failure for the red team. (For those of you who are lost already, a red team is an independent group within a company’s security organization that challenges the effectiveness of its security defenses. The red team performs analysis of systems and process gaps. Then it attacks you, hopefully before a real adversary does.) Let’s set the record straight on this critical aspect of modern security programs.
The red team always wins. Always.
It can be humiliating. And the timing is rarely convenient. Friday late night or on Christmas morning? Fair game.
The red team adopts the tools and techniques of actual adversaries. They use their understanding of attacks on other organizations that have been made public. They mimic the work of adversaries that the blue team has caught. They do not fight fair, nor will your adversaries.
Most companies prepare their defenses around best practices and compliance. Those alone will not get you very far. Even the organizations that use threat models and attack chains (i.e. the common events in an attack) need to practice. Practice. Measure. Learn. Repeat.
Most companies think they have a security plan. One of the great philosophers of our time, Mike Tyson, once remarked “Everybody has a plan until they get punched in the mouth.” Will your muscle memory kick in after getting hit? Or will you be stunned? Companies that engage in continuous red/blue battles are far more likely to detect and survive real attacks.
Having a security program without a red team is like practicing martial arts in the mirror rather than with a worthy sparring partner.
A red team exercise should not be an annual activity. It should represent a continuous clear and present danger. An employee, for example, may (incorrectly) doubt that they are the target of state-sponsored actors. They might think “Why should I close these minor gaps? It’s not like anyone would use these vulnerabilities against us!” They can, however, be sure that their red team is actively targeting them. Continuous red team exercises, over time, will give the blue team a fighting chance.
After the red team attack, what do you do? Do you “fix the glitch”? Or do you take time in the post-mortem to find the root cause and to fix it? More mature organizations will revisit the gaps over time. They provide input into the next planning cycle. Lessons learned from red team exercises contribute to a stronger defense and a better chance of stopping the real adversaries.
The real scandal is not that a red team won (the red team always wins!), but that many companies do not have red teams. Reporters: want a great story? Ask every CISO you talk to if they have a full-time, dedicated red team. Prepare yourself to hear some spin.
Unacceptable answers:
- We are not the target of sophisticated adversaries.
- We already know we have a lot of work to do so adding a red team report isn’t going to help.
- We work in a highly regulated industry so it’s not necessary.
- We have not had a breach in years.
- Our attack surface is small.
- Our IT team is great and we do a good job of user training.
Yahoo has its own internal red team known as Offensive Engineering (yes, that can be read two ways!). Their job is to take a contrarian view of Yahoo systems. They don’t care what the code was designed to do. They care about what it actually does. And yes, this red team always wins. Always. It’s what we pay them to do.
Let’s stop talking about red team wins as if they are a bad thing and let’s start talking about the red vs blue feedback loop: Practice. Measure. Learn. Repeat.
Doug DePerry, Senior Security Engineer, Paranoids
In our inaugural post to The Paranoid, we discussed the human element behind online attacks–the human adversary. We sought to give some perspectives as to who is behind online threats in order to better understand how to defend against them. Yahoo’s bug bounty program applies that insight in our ongoing efforts to provide a safe environment for our users. By thinking about the economics of security, we’ve found that we can tilt the advantage in our favor by partnering with industry-leading security researchers.
We often get questions from both security researchers, and people just interested in learning about how programs like these work. We thought we’d use this opportunity to take a quick look under the hood.
First, some background. Bug bounty programs essentially crowd-source security. They allow companies to improve coverage so they are able to add additional eyes where they need them. Bug bounty researchers also bring depth of expertise and different skill sets that can uncover hard to find bugs.
For the past two years, Yahoo has developed one of the largest and most successful bug bounty programs in the industry. We’ve paid out over $1.7 million dollars in bounties, resolved more than 2,000 security bugs and maintain a “hackership” of more than 2,000 researchers, some of whom make careers out of it.
Security researchers often ask us how we decide the payout associated with a given bug report. At first it might seem logical that we pay based on the type or classification of a security bug. Some bug types tend to be bad, so you might think that they would be paid the same. However, in the vast majority of cases, that’s not the complete story. So if the bug type alone is not what we use to determine the payout, what is? The missing input to the calculation is the impact of the vulnerability. We take into account what data might have been exposed, the sensitivity of that data, the role that data plays, network location and the permissions of the server involved. Those factors are of great importance.
Given the importance of the impact of a bug, the Yahoo bug bounty program does not reward researchers solely based on bug type. The type of bug a security researcher finds is mostly irrelevant. It’s what the bug allows them to do and where that are most important. What can an attacker actually do with this specific bug to potentially affect the security of Yahoo or our users? Furthermore, Yahoo’s application landscape is not necessarily uniform; certain properties or applications are more equal than others.
Here’s an example to show how these factors work in practice. SQL injection bugs are often a devastating bug class because they can provide full access to a database. Odds are, if a company has a presence on the web, they are storing sensitive information in databases. But just because an attacker can access the database does not mean it’s game over. The real reason that the SQL injection bug class can be so devastating is the data stored in the database may be accessed or changed by unauthorized parties. The typical impact of a SQL injection bug is high because the data exposed is typically sensitive, except when it’s not. What if the database doesn’t contain any sensitive data?
Part of the process in determining impact can seem opaque to the researcher, and we understand that. That obscurity is an unfortunate but necessary fact of life in a bug bounty program. As an external party, it is just not possible to have all the information. The sort of testing available to participants in a public bug bounty program is inherently “black box”–no documentation, no source code, what you see is what you get.
So we encourage bug reporters to include in their reports what they believe the impact of the vulnerability to be (example report here). Submitting a report that contains a thorough and detailed explanation of a legitimate security issue is much more highly valued and rewarded.
We also work closely with the developers to ensure the bug is fixed in a timely manner, and to obtain their expert opinion on impact if necessary. If the developers that created the application tell us that no sensitive data is stored in a particular database, we take that into consideration when awarding your bug. More detailed guidelines for our bug bounty program are available at hackerone.com/yahoo.
To paraphrase a little-known quote, “bug bounty programs don’t reward you for being clever.” Users and researchers should know that we place far more weight on how impactful bugs are to our platforms.
By Bob Lord, Yahoo CISO (Paranoid in Chief)
If the countless data breaches we read about in the news have confirmed anything, it’s that online security is somewhat of a moving target. We’ve witnessed compromised security at one point or another across every industry and government. From health records and email to financial information, intellectual property and critical infrastructure, it would seem nothing is secure these days.
Yet, despite being armed with this fundamental understanding of online security, it’s often treated as a static challenge–as if there is one solution for one vulnerability. In an inherently insecure world with ever changing threats, our conventional wisdom must evolve just as online threats do.
The obvious next question is how, and that’s a good question to ask with a plethora of answers. But in order to understand how we adapt to emerging threats, it’s first and foremost critical to understand the dynamics behind the threats themselves. Why are the threats changing and what allows them to continue to be successful?
In fact, the next best question to ask is who is behind today’s online threats. The most important aspect of online security that we can internalize is that we are up against dedicated, human adversaries who organize their activities into campaigns.
They are dedicated, which means they have a job to do, or a calling. They’re going to keep coming back until they achieve their goals. Maybe they work for a criminal syndicate, or for a foreign military. Or maybe they are on a mission from God.
They are also human, which means they can be creative and resourceful. They are like water in a cracked vase. It will find a way to seep out. They spend time learning your internal processes and reading your internal documentation before acting.
And finally, they work in campaigns. The data they seek from a system may not be valuable by itself. It may be that the data is valuable because it provides information about human rights activists in their own country. Or because they want to know what their political opponents are doing. They are likely targeting other services of peers and competitors. The data they collect is only valuable to the extent the campaign objectives are known.
Our activities as defenders, whether the casual user to the chief information security officer, need to line up against these characteristics of our adversaries. Are we considering how a phone call from an unfamiliar number but a familiar voice might be part of a social engineering scheme? Are we employing security tactics that eliminate an attack instead of letting it shift to a new vector?
Until we start thinking about online adversaries this way, we’ll continue to find ourselves playing whack-a-mole without ever turning the tide.
This is the first edition of our new Yahoo Tumblr series–The Paranoid–where we will delve into the security space and share how we’re working to protect our users, as well as useful tips for users to consider as they go about their everyday lives online. Like all good security researchers, we will look at security issues from the viewpoint of an adversary. Our goals with this series are to break conventional wisdom, ask tough questions about how we approach online security, and ultimately allow our users to hold us to a higher standard. Most importantly, we want to start a conversation to ultimately improve the safety and security of our users and our network.
By Doug DePerry, Senior Paranoid
We
put our users’ security first at Yahoo, and today we’re proud to
highlight one way in which we’re protecting our users against evolving
online threats through our bug bounty program. Partnering with HackerOne,
Yahoo’s bug bounty program has grown dramatically since our launch
about two years ago. Our bug bounty program boasts more than 2,000
security researchers and we’ve awarded $1.6 million in the last two
years. Our security team, known as the Paranoids, work night and day to
secure our users, but, with an online property as large as Yahoo, having
as many eyes as possible focused on the security of our users
crowd-sources what would otherwise be an impossible task for the
resources of a few.
Learn more about our growing bug bounty program here.
