Recently, the SRE team at Doximity launched a new tool enabling our web teams to run development-related workloads in the cloud. It was important to the SRE team that this tool did not require developers to substantially alter their existing workflows, as many had already put time and effort into setting up development tools on their local machines such as their choice of terminal, editor, and browser.
What we had in mind was a standardized remote development environment which gave us increased visibility, allowing uniformity across development and production-grade environments. We refer to this tool internally as "Dox Compose Cloud".
But what about Github Codespaces?
You may find this project to be somewhat reminiscent of Github Codespaces. Here's where they diverge:
- Both Dox Compose Cloud and Github Codespaces launch VMs to run development workloads.
- Both live in global data centers spread across multiple regions.
- Using dev containers (config-as-code for Codespaces environments), Github allows users to configure the VMs they want, while Dox Compose Cloud launches standard instances that can be used out of the box to develop any Doximity application.
- Codespaces is a browser-based code environment that allows users to use VS Code in their browser. It also gives users the ability to SSH onto their instance, if they want to commit to using a shell-based environment instead of a graphical editor. Dox Compose Cloud enables users to continue developing on their own local machine using their editor of choice.
So what is the main difference? Dox Compose Cloud users have the ability to use any development tools they prefer, as opposed to a specific set of tools. The underlying technology that enables us to do this is Mutagen. Mutagen securely forwards file system changes and select ports to a dedicated EC2 server through an SSH tunnel. This allows the user to interface with their local machine using commands like docker container ls without any concern for what the Mutagen daemon is doing in the background. In our case, it synchronizes all the processes that are part of a user’s development workflow: commands from the docker compose CLI, requests to local host port 80, and of course, all code changes (files in specific folder that is designated for development).
Problems with Local Containers
So what motivated the SRE team at Doximity to create this tool?
The primary motivation was faster issue resolution. Historically, resolving issues with our development environments has been slow and cumbersome. Before Dox Compose Cloud, it would have been common for the SRE team to receive several requests per week for support with a development environment-related issue. Oftentimes, the SRE team was unable to reproduce the issue and therefore unable to resolve.
The main problem with this sort of troubleshooting was that the SRE team lacked visibility. We couldn't SSH onto developers’ machines. We weren't collecting any metrics on the state of our development environments, as we do with our production environments. We lacked instrumentation to gather insight and the general ability to maintain development environments.
As a team, we began to realize just how much of an issue this was. A stable development environment enables developers to be effective. Before Dox Compose Cloud, the only way for us to achieve this "production-like" level of support was to go back and forth over Slack, or to hold a video call, which was far from ideal.
The second motivator for us was the performance degradation we had been seeing specifically with Macbook Pros. Many of our developers are in fact developing on Mac OS. While we allow developers to choose the OS that they prefer, we do strongly recommend that they use Docker in development, as much of our development tooling relies on it. However, since Docker can't run natively on the Mac OS, Mac users at Doximity have been dependent on using "Docker for Mac" which, in our experience, has been a huge contributor to overall lack of responsiveness on Macbook Pros. With Dox Compose Cloud, we aimed to remove the dependency on Docker for Mac. It is ultimately better to run Docker on Linux, or any OS with a kernel that is suited for container architecture. Docker for Mac can only mimic this by using a VM, which creates a lot of overhead, and a lot of stress on the machine.
Running Containers in the Cloud
Running our Docker containers on EC2 allows us to rely on the benefits of Docker on Linux while allowing developers to continue to use their preferred development machine. With EC2, we have the ability to select an AMI for the instances we provision, which gives us the freedom to choose the OS we want to use for (in this case) our development environments. Furthermore, using custom AMIs enables us to get our developers started up faster by prepackaging these AMIs with the source code, software, and dependencies our developers need in development.
By running Linux on our EC2 nodes, we are taking advantage of the performance improvements that come along with running Docker natively, with the added advantage of running our Docker containers in isolation (without all the noise that is everything else running on our local machines). EC2 grants us added visibility. We're able to collect metrics on the state of these development instances and can resolve issues (particularly those at the system level) before they ever become a problem for our developers.
As part of a one-time setup, we enable developers to provision their own VM using some internal tooling. Each VM uses a custom AMI with dependencies and source code pre-installed on to it. Use of this AMI across development instances ensures uniformity for all environments and reduces the impact that upgrades may have. These VMs are designed to be easily replaced, as often as needed. In the event a developer's environment becomes unstable, we can easily start fresh with a new environment that is also up to date with the latest AMI.
Once the one-time setup is complete, Doximity developers are effectively running all workloads relating to their existing development workflows in the cloud, on their own dedicated instance. As an added benefit, this remote instance is one with which the SRE team at Doximity has all the tools and insights needed to ensure its reliability.
We're happy with the cloud-based environment and how it has enabled developers to be more effective. We plan to continue to make improvements and to continue to share what we learn.
BTW, we're ⚡ hiring Infra, SRE, Web, Mobile, and Data engineers at Doximity (see roles) -- find out more about our technical stack.
Be sure to follow @doximity_tech if you'd like to be notified about new blog posts.
