At Shopify, our bounty program complements our security strategy and allows us to leverage a community of thousands of researchers who help secure our platform and create a better Shopify user experience. We first launched the program in 2013 and moved to the HackerOne platform in 2015 to increase hacker awareness. Since then, we've continued to see increasing value in the reports submitted, and 2017 was no exception.
Our Highest Payout to Date
In 2017, we saw our highest payout to date awarded, $20,000 to uzsunny who reported that by creating two partner accounts sharing the same business email, it was possible to be granted unauthorized "collaborator" access to any store without any merchant interaction. We tracked down the bug to incorrect logic in a piece of code that was meant to automatically convert an existing normal user account into a collaborator account. This bug was fixed within hours of having received the report.
Other awesome reports included cache-money for discovering a race condition in our partner auto conversions, bored-engineer for discovering a XSS vulnerability in our SVG parser and zombiehelp for reporting an XSS that could be triggered on the storefront of any Shopify store.
H1-415
In February 2017, we participated in HackerOne's H1-415 hacking event in San Francisco, bringing world class bug bounty hunters together to test Shopify. This resulted in 15 bugs being resolved from 5 hackers, earning them a total of $42,000 with the average payout being $2,800. This was a huge success for us.
The event kicked off with a cocktail party the night before the actual hacking where we got to chat with many hackers, better understand what's important to them from a bounty program and how they approach their targets. The following day, we had the opportunity to watch the hackers in action and gain better insights into how they test platforms, what functionality stands out to them and how they identify corporate assets. In some cases, we got to sit with hackers to answer questions, learn from each other and build relationships. We even hired one of the attendees from the event as a full time Application Security Engineer.
Participating in H1-415 definitely amplified the attention on our program after the event and is something we're considering doing again in the future.
Stats
2017 was a very successful year for our bounty program. Not including the stats from H1-415, we saw our average bounty jump to almost $1,100 from $659 in 2016. Our total amount paid to hackers was also up approximately $7,000 compared to the previous year, to $67,550 with half of all resolved reports having received a bounty.
We continued to focus on fast response and triage times recognizing the importance of these two metrics to hackers. While our initial response time slipped by 2 hours, our average first response was still only 4 hours and triage time was 4 days. The difference between response and triage for us is due to our effort to ensure all reports are validated with the appropriate development team to avoid the disappointment of closing a triaged report as informative or not applicable. Both our time to bounty and resolution were one month.
While we still received a fair number of invalid reports, this was down about seven percent compared to 2016, with 63.1% of all submissions being closed as not applicable. This was accompanied by a one and a half percent increase in the number of resolved reports, 10.5%.
We also disclosed 25 bugs on our program. While we request disclosure on all resolved reports, we respect the reporter's wishes to limit the disclosure or keep the report private. We believe it is extremely important that we build a resource library to enable ethical hackers to grow in our program. We strongly encourage other companies to do the same.
While we had a very successful 2017, we know there are still areas to improve upon. The total number of resolved reports was down from 141, with 100 hackers thanked, in 2016 to 121, with 71 hackers thanked, despite having added new properties to our scope and shipping tonnes of new features last year. We also continue to have some low severity reports remain in a triaged state well beyond our target of 1 month resolution, which means hackers are waiting to be paid.
Recognizing the increasing competition for attention on HackerOne, we began re-evaluating our own program in fall 2017 and attended the HackerOne Security@ conference in October to gain better insight into what other programs have found effective. One of our own Application Security Engineers also participated on the Understanding Hacker Motives panel to share insights on what's important to hackers. Based on our review and what we heard at Security@, we're excited to announce a number of changes to our program, including focusing on paying hackers sooner, providing better program clarity, and making functionality available sooner.
Quicker Payments
We know being paid quickly is a huge motivator so beginning today, we will be paying our program minimum of $500 when reports are triaged! We've also added a payout table so hackers know what to focus on and what to expect from their submissions. Our max typical payout is also explicitly set at $25,000, making it among the most competitive on the HackerOne platform.
Better Program Clarity
We know a hacker's time is valuable and we want everyone to get the best return on their investment. To help hackers, we've updated our scope to clarify what is already known and ineligible to submit to our program. We also created bugbounty@shopify.com to test how helpful it can be to allow hackers to contact us about our program for those situations when hackers have questions.
Feature Access
Shopify ships new code on a daily basis and we are working to give hackers access to new functionality before its full release. To be eligible, hackers must use a whitehat partner account to create shops for testing or add +hackerone to their email address (e.g. john.doe+hackerone@shopify.com). After doing so, their stores will be enrolled in Shopify beta tests before new functionality is available globally.
Our bounty program is an important part of our overall security strategy. We’d like to thank all bug reporters for their participation to date and for helping us improve the security of our platform. We hope these changes will encourage more hackers to work with us.
Happy Hacking!