Shopify makes extensive use of ingress-nginx, an open source Kubernetes Ingress controller built upon NGINX. Nearly every request Shopify serves is handled at one point by ingress-nginx, and we are active contributors to the project. Since we make use of ingress-nginx and its many features (like annotations and configmap) so heavily, the quality of its debugging experience is very important to us. While debugging ingress-nginx was always possible using a complex litany of kubectl commands, the experience was frequently frustrating. To help solve this issue, I recently contributed a kubectl plugin to the project. It provides a number of features that make ingress-nginx much easier to upgrade and debug, saving us time and increasing our confidence while working with it.
Easier Upgrades
ingress-nginx is a fast moving project. New releases happen every few weeks, and usually contain one or two breaking changes. When running a very large cluster, it can be difficult to know whether or not any configuration changes need to be made to remain compatible with the new version. Our usual process for upgrading ingress-nginx before the plugin existed was to read the CHANGELOG for the new version, export every single ingress in a cluster as YAML, then manually grep through those YAMLs to find anything that would be broken by the new version. With the plugin, it's as simple as running:
to get a nice, formatted list of everything you might need to change.
Improved Ingress Listing
When running the vanilla kubectl get ingresses, a fairly minimal amount of information is returned:
Often you don’t just care about the hosts and addresses of a whole ingress, you care about the individual paths inside the ingress, as well as the services they point to. Using the plugin, you can get a more detailed view of the contents of those ingresses without inspecting each one:
Using the plugin, it is much easier to answer questions like “what service is this path hitting?” or “does this site have TLS configured?”.
Better Debugging
There are many common debugging strategies for ingress-nginx that can become tedious to carry out manually. Usually, you are required to find and select an ingress pod to inspect, and you are required to filter the output of whatever commands you run in order to find the information you’re looking for. The plugin provides convenient wrappers for many kubectl commands that make it quicker and easier to perform these tasks, selecting a single ingress pod automatically to run the command in. As an example:
can be replaced by a single command:
Likewise, inspecting the internal state of the controller is much easier. In the case of reading the controller’s generated nginx.conf for a particular host:
can be replaced by:
ingress-nginx stores some of its configuration state dynamically, making use of the openresty lua-nginx-module to add additional request handling logic. The plugin can be used to inspect this state as well. As an example, if you are using the session affinity annotations to add a session cookie, but the cookie doesn’t seem to be applied to requests, you can first use the plugin to check if the controller is registering the annotation correctly:
This shows that the annotation is correctly reflected in the controller’s dynamic configuration.
My Experience
Since the addition of the plugin, I have found that the time it takes me to upgrade or debug ingress-nginx has decreased substantially. When I first arrived as an intern here at Shopify in January 2019, I was tasked with upgrading our ingress-nginx deployments to version 0.22.0. I spent a long time grepping through ingress manifest dumps looking for breaking changes. I spent time trying to come up with the kubectl invocation that would allow me to inspect nginx.conf inside the controller. I didn’t even know that the dynamic configuration information existed, let alone the arcane incantations that would allow me to read it. There existed no way at all to read certificate information. It took me days to fully roll out the new version.
Near the end of my internship, I upgraded our deployments to ingress-nginx version 0.24.1. Finding breaking changes required only a few invocations of the lint subcommand. Debugging the controller configuration was similarly quick. I had the confidence to ship the new version much more quickly, and did so in a fraction of the time it had taken me a few months ago. Much of this can be attributed to the fact that there was now a single tool that allowed me to easily perform every debugging function I had previously been doing, plus a few more that I hadn’t even known existed. In addition, having all of these previously unintuitive and little-documented debugging tricks collected together in one easily usable tool will make it far easier to get started with debugging ingress-nginx for those who are unfamiliar with the project, as I was.
It’s also true that much of this time difference is due to my growth in both confidence and competence at Shopify. I’ve learned a great deal about Kubernetes, especially the nuts and bolts of how requests to Shopify services get from client to server and back. I’ve had the privilege of being paid to work on an open-source project. I’ve learned the skills, both technical and interpersonal, to function as part of a team in a large organization. Changes that I’ve made, code that I’ve written, have helped to process literally billions of web requests during my time here. This has been an extremely productive four months.
The plugin was released as part of ingress-nginx version 0.24.0, but should be compatible with version 0.23.0 as well. You can find the full plugin documentation and install instructions in the ingress-nginx docs. To get involved with the ingress-nginx project, or to ask questions, drop into the #ingress-nginx channel on the Kubernetes Slack.
If solving problems like these interests you, consider interning at Shopify. The intern application window is now open and closes on Wednesday, May 15th at 9:00 AM EST. Applications will not be accepted after this date.