Production engineers from Shopify will be crossing the pond to speak at SREcon Europe from July 11 to 13, 2016 in Dublin, Ireland. From flash sale engineering to fuzz testing to multi-tenant architecture across multiple datacenters, we got you covered!
- Flash Sale Engineering - Emil Stolarsky
From stores with ads in the Super Bowl to selling Kanye’s latest album, Shopify has built a name for itself handling some of the world’s largest flash sales. These high profile events generate write-heavy traffic that can be four times our platform’s baseline throughput and don’t lend themselves to off-the-shelf solutions.
This talk is the story of how we engineered our platform to survive large bursts of traffic. Since it’s not financially sound for Shopify to have the required capacity always running, we built queueing and page caching layers into our Nginx load balancers with Lua. To guarantee these solutions worked, we tested them with a purpose-built load testing service.
Although flash sales are unique to commerce platforms, the lessons we learn from them are applicable to any services that experience bursts of traffic.
Monday, July 11, 1:40 pm to 3:00 pm — Track 1: Capacity Planning
- Extreme OS Kernel Testing - Kirk Russell
Fuzz testing has been used to evaluate the robustness of operating system distributions for over twenty years. Eventually, a fuzz test suite will suffer from reduced effectiveness. The first obstacle is the pesticide paradox: as you fix the easy defects, it gets difficult to find the remaining obscure defects. Also, the test execution time and the debug/fix cycle tends to be manual work that can take hours or even days of effort. During the presentation, a structured framework for creating new fuzz tests will be introduced, along with a competitive analysis approach used to minimize defect reproduction complexity.
Wednesday, July 13, 11:00 am to 12:20 pm — Track 1: Wildcard
- Scaling Shopify’s multi-tenant architecture across multiple datacenters - Florian Weingarten
Multi-tenant architectures are a very convenient and economical way to share resources like web servers, job workers, and datastores among several customers on your platform. Even the smallest Shopify store on a $9/month plan can easily survive getting hammered with a 1M RPM flash sale by leveraging the resources of the entire platform. However, architectures like this can also have several drawbacks. They are potentially harder to scale and things like resource starvation or back-end outages are harder to isolate.
In this talk, I’m going to walk you through the history of how Shopify grew from being a small standard single-database single-datacenter Rails application to the multi-database multi-datacenter setup that we run today. We will talk about the advantages in terms of resiliency, scalability, and disaster recovery that this architecture gives us, how we got there, and where we want to go in the future.
Wednesday, July 13, 11:00 am to 12:20 pm — Track 2: Network
Image credit: Giuseppe Milo
Over the past few years the mobile development community has seen a dramatic shift towards the use of continuous integration (CI) systems similar to changes present in other communities — particularly web developers. This shift has been a particularly powerful moment for mobile developers, as they’re able to focus on their apps and code rather than spending their time on provisioning, code signing, deployment, and running tests.
I’m a software developer at Shopify currently working on our Developer Acceleration’s Mobile team. My job is to design, create, and manage an automated system to provide an accelerated development experience for our developers.
Based on our experiences at Shopify, we will be talking about “hosted” vs “BYOH” systems, how to provision Mac OS X and Ubuntu machines for iOS and Android, and the caveats we ran into throughout this series. By the end, you should be ready to go build your very own CI setup.
New posts will drop every few weeks and you can keep track of them here. Consider this a table of contents of setting up mobile CI environments without user interaction (aka fully automated).
Android Setup (Coming Soon!)
- Provisioning Ubuntu for Android
- Android versions and Android Emulators
Shared Setup (Coming Soon!)
- VMWare and ESXi — Hosting your VMs
- Packer and VMWare
- Testing your infrastructure with Serverspec
Last updated: June 16, 2016
At Shopify, our mission is to make commerce better for everyone. When we say better, we’re talking about caring deeply about making quality products. To us, a quality web product means a few things: certainly beautiful design, engaging copy, and a fantastic user experience, but just as important are inclusivity and the principles of universal design.
“Everyone” is a pretty big group. It includes our merchants, their customers, our developer partners, our employees, and the greater tech community at large, where we love to lead by example. “Everyone” also includes:
- 9.1% of adults with vision trouble, and 8% of men and 0.5% of women who are colorblind
- 16.8% of adults with hearing trouble
- 15.1% of adults with physical functioning difficulty
- 4.4% of adults with cognitive disabilities [PDF]
We take our mission to heart, so it’s important that Shopify products are useable and useful to all our users. This is something we’ve been thinking about and working on for a few years, but it’s an ongoing, difficult challenge. Luckily, we love tackling challenging problems and we’re constantly chipping away at this one. We’ve learned a lot from the community and think it’s important to contribute back, so — in celebration of Global Accessibility Awareness Day — we’re thrilled to announce a series of posts on accessibility.
Over the next few weeks, we’ll release posts on the Shopify's UX publication on Medium about some of the real-world problems we’ve faced trying to make different parts of Shopify accessible and then how we’re trying to overcome them. Like many other teams, we’re not starting from scratch: the Shopify platform already exists, and with these posts we want to dig into our approach for making an existing platform accessible. You’ll hear from a bunch of talented people on our UX team — front end developers, designers, content specialists, and user experience researchers — who work closely with engineering and product teams to solve these problems.
For this round, we’re releasing six articles, with a new one dropping every two weeks. We’ll cover:
- Contrast and accessible text
- Accessible pricing in Shopify themes
- Semantic HTML at scale
- Mobile-friendly, accessible tables
- Running an accessibility user testing session
- Creating an accessible culture
We hope you’ll find these useful and that they’ll help you make your product (and industry!) better for everyone. Feel free to drop us comments and questions in the posts. We want this to be a conversation.
PS If you’re passionate about quality and are looking for an awesome place to work, check out our Careers page.
Updated June 9, 2016
RailsConf is tomorrow! For the first time, the conference will be in Kansas City, known for jazz and barbeque and home to the Royals. If you're heading down, here are the details for the five presentations we'll be giving:
- How We Deploy Shopify - Kat Drobnjakovic
Shopify is one of the largest Rails apps in the world and yet remains to be massively scalable and reliable. The platform is able to manage large spikes in traffic that accompany events such as new product releases, holiday shopping seasons and flash sales, and has been benchmarked to process over 25,000 requests per second, all while powering more than 243,000 businesses. Even at such a large scale, all our developers still get to push to master and deploy Shopify in 3 minutes. Let's break down everything that can happen when deploying Shopify or any really big Rails app.
Wednesday, May 4, 11:40 am to 12:20 pm, Room 3501 G
- Foreign API Simulation with Sinatra - Konstantin Tennhard
- How Sprockets Works - Rafael Mendonça França
Wednesday, May 4, 3:40 pm to 4:20 pm, Room 3501 H
- Testing Rails at Scale - Emil Stolarsky
- Rails 5 Features You Haven't Heard About - Sean Griffin
We've all heard about Action Cable, Turbolinks 5, and
Rails::API. But Rails 5 was almost a thousand commits! They included dozens of minor features, many of which will be huge quality of life improvements even if you aren't using WebSockets or Turbolinks.
This will be a deep look at several of the "minor" features of Rails 5. You won't just learn about the features, but you'll learn about why they were added, the reasoning behind them, and the difficulties of adding them from someone directly involved in many of them.
Come say hey, and we're looking forward to chatting all things Rails! And if you're a dev interested in joining Shopify. Kayla Boyer will be at RailsConf: reach out to her through Twitter. To check out open roles or to learn more, head over to our Careers page.
rbtraceto quickly track down these issues. In this post, we’ll explain how to use gdb to retrieve a Ruby call stack, inspect environment variables, and debug a really odd warning message in production.
Hi there! We’re Kat and Omosola and we’re software developers at Shopify. We both started working at Shopify back in May, and we felt both excited and a little nervous before we got here. You never know exactly what to expect when you start at a new company and no matter what your previous experience is, there are always a lot of new skills you need to learn. Thankfully, Shopify has an awesome onboarding experience for its new developers, which is what we want to talk about today. The developer onboarding process at Shopify is constantly adapting to the needs...
After a year of internal use, we’re excited to open-source our deployment tool, Shipit. With dozens of teams pushing code multiple times a day to a variety of different targets, fast and easy deploys are key to developer productivity (and happiness) at Shopify. Along with key improvements to our infrastructure, Shipit plays a central role in making this happen. Motivation Coordinating frequent deployments in a large development team poses a number of challenges. You need to ensure that no one else is currently deploying, that the revision you are about to deploy has been successfully tested on CI, and even...
This is a continuation of our series describing our evolution of Shopify toward a Docker-powered, containerized data centre. Read the last post in the series here.
One of the challenges along the road to containerization has been establishing a way to move application secrets like API keys, database passwords, and so on into the application in a secure way. This post explains our solution, and how you can use it with your own projects.
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Debugging is always challenging, and as programmers we can easily spend a good chunk of every day just trying to figure out what is going on with our code. Where exactly has a method been overwritten or defined in the first place? What does the inheritance chain look like for this object? Which methods are available to call from this context?
This article will take you through some under-utilized convenience methods in Ruby which will make answering these questions a little easier.