This is a personal blog. My other stuff: book | home page | Twitter | G+ | CNC robotics | electronics

March 12, 2011

Pwn2own considered (somewhat) harmful

I think that hacking challenges and bug bounty programs can be extremely valuable. This is true when they involve transparent, sustained efforts to evaluate the security of a particular platform. For example, I believe that there is a substantial value in Mozilla bug bounties, or in the Chrome reward program. These programs greatly improve the security of the browsers in question, occasionally advance our understanding of web security, and provide tons of statistical data about vendor response processes and attitudes toward security flaws. That last part is arguably the most important metric when dealing with code so complex that for better or worse, it is unlikely to ever be perfectly safe.

I also think that Pwn2own, an annual browser hacking contest run by TippingPoint, does not deliver the same value. The formula of the contest boils down to this: once a year, a single, secretly developed exploit is exchanged for a substantial amount of money. No information about the flaw or its back story is revealed in the process, and given that this trade is negligible in comparison to the annual volume of browser vulnerabilities, there is absolutely no intrinsic value in observing it.

That, alone, is not a compelling criticism; at best, it's a reason not to watch. But then, there are some negative consequences, too: it is in the interest of the conference and contest organizers, and the participating researchers, to get publicity for their findings - and journalists, who do not necessarily have a holistic view of the day-to-day browser security research, embrace such high-profile developments with disproportionate enthusiasm. The resulting ecstatic press coverage ultimately undermines any attempt to have a meaningful and reasonable discussion about the state of browser security.

Take this quote, which likely will be repeated in every Safari-related story for the next twelve months:

"A team was able to exploit Safari to exploit a MacBook Air in five seconds. Yes, five seconds - less time than it takes most people just to type 'Safari got hacked in less than five seconds'."

That's remarkable, but also completely wrong. It takes days or weeks to find and exploit a vulnerability, and Pwn2own is no exception: the actual exploits are prepared months or weeks in advance, and simply executed on the day the contest takes place. I do not think there is a single person in the information security industry who would say that the discovery of a normal browser vulnerability is a notable event: several hundred such flaws are discovered and resolved every year in every browser, as evidenced by release notes maintained by the vendors with varying degrees of accuracy. Neither the fact that somebody discovered a vulnerability before Pwn2own, nor that this person needed needed five seconds to execute that pre-made code, is a useful measure of anything.

Similarly, the survival of Firefox and Chrome intuitively makes me happy, because I know that these browsers give a lot of thought to security - but I do not think that Pwn2own is a meaningful testament to this. Perhaps these two vendors merely patched up the vulnerability somebody wanted to use, and there was not enough time to find a new one. Or perhaps nobody attending the event (which brings together only a tiny fraction of the infosec community) had the expertise and the inclination to target this particular browser.

Yes, there are vendors who lag behind the rest when it comes to vulnerability response and proactive security work; and there are some hard problems we still have to solve to make the web a safer environment. But the headlines inspired by Pwn2own (and probably encouraged by the organizers) are very unfair, and unnecessarily alienate the parties who should be paying attention to their security posture. Investigating real data, and asking some hard-hitting questions, can make more of a difference... and if done right, it can be more fun.

24 comments:

  1. I think Pwn2Own is helpful for the vast majority of clueless lusers who imagine that Apple stuff is invulnerable, or that all browsers are the same. Sure, it may be non-representative of real vulns and encourage unhealthy customs among competitors, but that's how all world-class sports work.

    The benefit of Pwn2Own is that it takes something 99% of people cannot comprehend and makes it a sports event with a big prize, which they CAN understand. It's Public Relations, and hacking NEEDS better public relations. Hackers as sports heroes are a lot better than hackers as scary shadowy deviants. I want to see @0xcharlie's photo on a box of Wheaties!

    ReplyDelete
  2. It's convenient for us when the essentially random outcome happens to match our personal beliefs. But there will be a year where Firefox goes down first, or Chrome - and the more you legitimize the contest as some sort of a litmus test of browser security, the more damage this outcome will cause.

    Plus, untrue statements ("it takes 5 seconds to pwn you") will really not compel any particular vendor to step up the act; if anything, they make their security teams and PR depts needlessly (but rightly) defensive.

    ReplyDelete
  3. They also tend to miss the fact that people might not be targeting Firefox or Chrome at all.

    ReplyDelete
  4. @Blizzard - Wouldn't it follow that the browser with the most glaring (or easily) exploitable hole would be targeted? I'm sure that FF/Chrome get a fair share of review.

    I think that Pwn2Own would legitimize themselves as more than a stunt if they gave some focus to the development of the exploit / were more open about the sharing of that information. I might be wrong, but I do believe that the exploit details are passed on to the vendor for fixing as is.

    ReplyDelete
  5. Chrome was targeted, by at least two people.

    Both failed to show up, perhaps because their exploits were plugged due to a patch the day before the contest.

    ReplyDelete
  6. The last-minute patches of Apple and Google products does raise the possibility that vendors are looking at Pwn2Own as a deadline for fixes. In a way this is good because it's always good to have some motivation, but it does take some of the objectivity out of the process, as Michal says.

    ReplyDelete
  7. I don't think the security and web dev communities are the audience for the pwn2own media message. Everyday users of Internet are. The journalists who wrote about the five second fall of Safari could not exchange that article with a technical review of the flaw or the history of finding the flaw and writing a reliable exploit.

    I think it's more about the second agenda. The CSW organizers want the big players' attention. They want Google, Mozilla, Microsoft, RIM, and Apple to look their way and even send some people there. Hey, it increases the impact of the conference.

    How to go about doing it? Create a media happening of course. And not like Phrack or (IN)SECURE. No, they want a regular media happening. So they fool journalists to think it's a speed competition and voilĂ  – the big players are listening, tweeting, commenting ... and writing blog posts!

    Given that, I think TippingPoint could do the right thing for the communities and require for instance a technical write-up of the exploit to publish once a patch is available.

    ReplyDelete
  8. I want to make it clear that I'm not actually complaining about the press: my concern is that contests like this one specifically encourage a particular flavor of (detrimental) PR coverage. The journalists are not at fault for following it.

    Security studies from reputable sources make the headlines quite often, too. Many of them are pretty bad, because they focus on superficial metrics (e.g., number of bugs), and are meant to promote products and services. But there is plenty of yet-to-be-analyzed data that with some luck, could make a difference.

    ReplyDelete
  9. Hi Michal,

    I usually agree with your posts, but this ones is different and I greatly disagree with it, for the following:

    Your first talk about money, and you actually consider the payment at Pwn2Own substantial:

    ..."The formula of the contest boils down to this: once a year, a single, secretly developed exploit is exchanged for a substantial amount of money."...

    And afterwards you talk about the work that it takes do the job:

    ..."It takes days or weeks to find and exploit a vulnerability, and Pwn2own is no exception: the actual exploits are prepared months or weeks in advance,"...

    You are saying that you consider that a top researcher (or a group of researchers) that work for a month, at least, in order to hunt for more than one vulnerability (which has been the case in this year's Pwn2Own and also in last year) and reliably exploits it, bypassing all the exploitation mitigations and in some cases a sandbox, it's not worth $15k ?
    A top researcher working for a company worth more than that, and of course his/her work is kept private for whoever they work for.

    I do agree what you say about the press, but that is also part of the show ZDI and CanSecWest are looking for, whether you and other people like it or not.

    The information about the exploit is provided to the affected vendors. So if microsoft IE was pwn, they receive all the informations about the vulnerabilities and the exploit, so that they know how their exploitation mitigation were bypassed. Microsoft then can fix the vulnerability and think about other strategies and how to improve their mitigations. It's a win-win for Microsoft and/or any other company who is taken down in the competition. On the top of that if the researched decides not to write about the technical details of his/her work, nobody except the affected vendors win security wise.

    The noise the press does is because the software in question was successfully compromised, which differs from just finding a vulnerability. By fuzzing anybody can find vulnerabilities without mayor knowledge, but exploiting them reliably is a whole different thing.

    Now if you consider what Mozilla and Google pay for vulnerabilities reported to them, I consider that a ripoff. Getting paid less than a couple of thousand dollars for AN EXPLOITABLE vulnerability after the code has been audited by guys like Marc Dowd? These vendors should be kidding, a day of work without results by only one top consultant costs more than what these vendors are paying.

    You can't compare a vulnerability with a vuln+reliable-exploit.

    If you tell 15k per vulnerability, I would agree with you. But we are talking about a piece of work that goes way beyond finding a vulnerability.

    Google was pwn with a crappy exploit that didn't even target actual browsers. The exploit technology that is being sold by only 15k could pwn any company in the world using the most up-to-date software and operating systems, with all the mitigations in place, NOW.

    Those are my 2 cents.

    As I said before, I'm a big fan and I mostly agree with your posts, but this particular port was an exception, and I wanted you to know why.

    Regards,
    Sergio

    ReplyDelete
  10. I don't quite understand where you are going with this. I am not arguing that $15k is / is not a fair compensation for X days of work. I had a couple of posts about how people should not feel *entitled* to compensation for unsolicited security work (http://lcamtuf.blogspot.com/2010/07/hi-im-security-researcher-and-heres.html), but when it's solicited, I don't have a problem with any amount of money changing hands.

    I am saying that the format of the contest (and the associated publicity) is harmful for the quality of the discourse about the challenges in browser security. The "development of an exploit" part is a red herring: it's nice, but for most part, it's not very necessary, other than as a publicity stunt.

    ReplyDelete
  11. In this case it is not an exactly an 'unsolicited security work', since ZDI is the one offering a price for it. Although this year Google put a price for the vulnerabilities in Chrome, so in this case it was even solicited by the vendor too.

    With ZDI, iDefense, Pwn2Own, etc, the vendors get a free audits all the time.

    Anyway, what captured my attention in your original post was the "substantial amount of money" statement which I didn't agree with. My comment was all about that and explaining why it was *not* a substantial amount of money.

    The format of the contest is about the exploit. With a vulnerability alone you don't get anything at the Pwn2Own.
    The contest is not about vulnerabilities it's about exploitation. If you don't pop it, it doesn't matter who cool you bug is, you don't win anything. So in this case *yes*, it is necessary.
    The publicity is part of the payment, and competitors are very well aware of that, since such exploits worth much more than the monetary prize in the exploit market.
    The exploit that dvorak and his team release there is worth *a lot* of money.

    Now I understand that you were talking about source code/vulnerability hunting. Pwn2Own instead is about exploitation and not about finding vulnerabilities.

    ReplyDelete
  12. Yes, as noted, I do not have a problem with the "unsolicited" part in this case.

    My criticism is really unrelated to money in any way :-)

    The exploit is necessary as per the rules of the contest, but my point is that it is not really necessary for any other reasons other than generating more PR buzz. Which is still perhaps fine when the generated and publicized data is at least indicative of anything: but when you have several hundred browser vulnerabilities a year, picking and exploiting one at random tells us fairly little about the comparative security of every browser.

    ReplyDelete
  13. That's a different story. Measuring Browsers security via what browsers were exploited and what browsers survive the contest is a very different story.

    Chrome and IE are the strongest browsers when it comes to exploitation mitigations and probably Chrome with its sandbox the strongest. If you ask me it was unfair that Mozzila wasn't taken down.
    But once again, since it is an exploitation contest you can look at it from two points of view.

    1- Press: What browser was taken down and what browser 'survived'.
    2- Exploit writers to his/her pears: Get that!, I reliably overcome all the exploitation mitigations and escape the sandbox everyone was talking about. And I get everyone to see how cool I am. (read: I can break in to any box on earth if I want to).

    Competitors are exploit writers, and the press are...well we all know how they handle stuff and tweak news to gain audience and readers.

    ReplyDelete
  14. Again, my criticism is that the contest is not even just conductive, but probably explicitly encourages, bad reporting (as a PR vehicle for the organizers, researchers, etc); which is somewhat irresponsible given that the exploitation data is, indeed, very imperfect and unfair (I used that exact word in my original post).

    You are defending things I generally agree with, so I am not sure that we really disagree in any way?

    ReplyDelete
  15. I believe we do agree now. :)
    But I do support Pwn2Own, that's all.

    ReplyDelete
  16. Michal, myself and the Tipping Point folks go to great lengths to discourage this "how many second to exploit" meme that those unfamiliar to security seem to latch on as a sensationalistic point, and if you look at it that kind of sensationalism has been pretty much dampened, most of the reporters writing about pwn2own have been slowly educated about the development process for vulnerability research, and mostly stopped succumbing to those ezy sensationalism headlines. While we can''t take full credit for educating them I'm positive pn2own has helped convey a more realisitc picture of the real situation and led to reporters writing more responsible stories about vulnerabilities.

    These days you mostly see this meme on new blogger sites writing about pwn2own without much information or based on secondary information, and both myself an Aaron (and the TP PR folks) try to respond a guide any such misinformation back to a more accurate explanation. Being in a public limelight like this inevitably inaccuracies some time spread, but we go to great lengths in almost every interview or discussion with reporters to try to stop them from jumping to the wrong conclusions, And on the whole I think the reportage has been getting much better.

    That N seconds to Exploit thing first came up in Shane Macaulays mac exploit a few years back and has been pretty much quieted down these days. I will get around to documentng some of the cool things that have ben spawned by pwn2own going on and some of the vendor responses that improve security that have arisen as a result of this contest. Inaccurate reporting is an unfortunate inevitability when you are dealing with complicated matters as these, but after watching and sheperding this contest for many years, I'm convinced it spurs improvements in security, and plan to continue to invest time into improving it and and continuingnit, because I'm sure it's a good thing.

    ReplyDelete
  17. Also, I'd like to point out that this contest is well established, and myself and the other organizers aren't just grasping for any reportage we can get, this thing seems to get enough attention as it is without needing to fan the flames of any sensationalist angle a reporter may grab on to. We spend most of our time with reporters trying to guide them away from the easy but sensationalistic FUD angles to try to guide them to the more complicated but realistic (and far less deterministic) conclusions you can and can't draw.
    You can use the results of the sometimes chance related availability of exploits for a target platform a somewhat litmus test of overall security, but it's very hard to draw definitive conclusions.

    A few reporters (and I'd like to specifically single out the registers Dan Goodin, dig's Bob McMillan, and ZD's threat post folks) have been taking the time to dig deeper into the situation behind researcher exploit development and vendor patching and software development to write stories and communicate conclusions about what security situations are.being illustrated inpwn2own, and overall communication of the contest has been greatly improving in recent years. And we'll keep working on trying to make the reportage more accurate.

    ReplyDelete
  18. Dragos: from my perspective, the contest seems structured specifically to be simultaneously very high profile, and essentially useless for drawing any conclusions about the security of any of the browsers: only one winner per browser, random draws, measured time-to-exploitation, etc. And I think that's an inherently dangerous mix, because it promotes a skewed and extremely imperfect picture of browser security (and enjoys it).

    Would you agree that Pwn2own is not good as a litmus test of browser security? If yes, I hope you can at least see my point: it gets so much exposure, and not just accidentally so, that it overshadows many other, more valuable data points.

    You can make the contest low-key, technical fun, but realistically, that is probably not what you guys want. You can also tweak it to provide more useful data (e.g. by extending the duration, rewarding all successful entries, etc), but these changes would turn it into something else entirely, and blur the line between a conference-tied contest and just a regular vulnerability reward / purchase program. This leads me to believe that the formula itself is flawed, and is difficult to salvage.

    ReplyDelete
  19. I don't disagree with what you've said, but it's largely irrelevant. Pwn2Own is a contest run by and paid for by a security vendor. Their job is to make money. It's simply a marketing gimmick meant to attract press attention at a discounted price, no?

    ReplyDelete
  20. Well, that's a cynical view. I don't think that the people who run it are bent on getting rich at any expense, but I do think that the incentives it creates are a bit undesirable.

    ReplyDelete
  21. Michal,

    The purpose of Pwn2Own is not about which browser is more secure than it's peers. Your point about how this doesn't actually help draw any high-level conclusions about browser security is true, but again you're missing the point. We all know there are hundreds of vulnerabilities affecting each of these browsers on a yearly basis. At the ZDI, we know this perhaps better than anyone. The point of Pwn2Own has always been to entice those who are able to actually exploit these vulnerabilities to come to Vancouver to show off their techniques. More importantly, it's to show those not in our industry that weaponized exploits do exist. We see them on a daily basis at the ZDI and I'm sure you see them in your line of work as well, but how many others even within our industry have actually seen a fully working exploit bypassing a mitigation like Protected Mode?

    Now you may say that the audience of Pwn2Own doesn't actually see the exploit, and this is true for numerous reasons--mainly that it is sensitive information until the bug is patched. However, you also state that "no information about the flaw or its back story is revealed in the process", which I disagree with. Firstly, we provide the vendor with direct access to the researcher to query them on any detail they'd like. We also provide them (obviously) with the actual vulnerability proof of concept. Secondly, several of the past Pwn2Own winners have blogged about their bug publicly once it was patched (http://vreugdenhilresearch.nl/ms11-002-pwn2own-heap-overflow/) thus providing the industry with some insight into how they actually won the contest.

    Now, about your grievances with the media coverage... I'm with you on that. I have actively discouraged every reporter I meet with about such misleading headlines as "x was hacked in y seconds". However, when you have contestants being quoted like so:

    "It took a couple of seconds—they clicked on the link and I took control of the machine," Miller said after the successful hack, according to ZDNet

    it makes it quite difficult to combat such sensationalist stories. The media coverage around Pwn2Own is so substantial nowadays that it is quite literally extremely difficult to control. If all accounts were from primary or even secondary sources it may be a bit easier to ensure the story is accurate. However it is quite often the case that stories are being written based on coverage citing other coverage citing other coverage. There's only so much we can do; last year Pwn2Own generated over 400 unique international articles.

    All in all, though, I believe the coverage is good for the industry. Not necessarily due to the merits of its accuracy, but because the news is reaching far beyond our otherwise limited audience. Those who do not understand the actual implications of the contest for browser security simply see the contest as affecting them solely because, "Hey, I use a web browser!". The reason I believe this to be beneficial is because these are the consumers, these are the ones the vendors *have* to pay attention to (as opposed to security researchers). The problem becomes one they can't afford to ignore if enough consumers (or business decision makers) are discussing it. It may be a stretch for me to assume that this will lead to a greater focus on exploit mitigations, but it seems logical that vendors would like a favorable turnout at the contest and that can't be achieved simply by patching bug after bug.

    ReplyDelete
  22. As noted in my earlier comments, I do not think that the organizers are cynical or evil, or that the contest should be condemned. That said, I do believe that its formula is purposefully structured to be very conductive to a certain type of mainstream reporting. And this reporting is very damaging to the discourse about browser security.

    I think this goes beyond the "oh well, tech journalists are clueless" mantra, too: it's simply difficult to imagine that this format would be reported as anything else than a high-profile, mainstream, very unfair vendor bashing fest.

    That's, again, not a criticism of the parties involved. It's just a criticism of the outcome. I think it's in the participants' and organizers' best interest to keep it this way, and I do not contest their right to do so. I just think this is (somewhat) harmful in the long haul.

    As to the amount of information exchanged: my point is that the value in observing the transaction, and keeping it so high profile, is minimal. There may be some benefit to the vendor (although most of them already know that their browsers can be exploited successfully this day and age). There may be some benefit to public disclosure independent of the contest (such as the cases you cite), but these are not a part of the contest itself.

    ReplyDelete
  23. Michal,

    Well, perhaps I can modify the rules for next year to require contestants to submit a blog post of sorts describing their bug and we can publish it on the DVLabs blog once the bug is patched; I'm always open to suggestions.

    As for the results/reporting being damaging to the discourse on browser security, that may be true in the eyes of the general public, but those "in the know" should understand that the contest isn't about judging the security posture of the applications side by side. There is no easy way to do so, anyway. You can compare metrics like number of bugs disclosed, patch turnaround times, mitigations in place, and so forth to begin to accomplish that. Again, that is not the point of Pwn2Own.

    I plan on re-architecting the competition next year to hopefully curtail the "certain type of mainstream reporting" that you mention, but it's quite difficult. Anyway, the contest has grown quickly over the last 5 years and I can comfortably state that when Dragos and I first designed the competition it was not "purposefully structured" to generate what you refer to as detrimental media coverage, it's just turned out that way due to the problems inherent in communicating the outcome of a a technical contest to the masses.

    My reasoning behind these posts is simply to get on the record that we do not "specifically encourage" such coverage as you declared in the original post and in subsequent comments. I believe your blog has a large following and it is our responsibility as the organizer to point out incorrect assumptions when they are stated as probable fact.

    Cheers

    ReplyDelete
  24. I did not mean to attribute this to an evil plan to cause harm, and I owe you and Dragos apologies if it appears this way.

    I have something slightly different and less sinister in mind: I do think that the contest is being drummed up by the organizers and the participants (for a variety of good short-term reasons) - and that this contributed significantly to its current prominence.

    When you get there, and browser X is randomly owned in the first YY seconds, it's difficult to expect different reporting. So, I'm not implying that you meant it to be this way; but that the sum of other, benign interests led to this outcome.

    ReplyDelete