As a Staff Software Engineer at Slack, I've spent the last six years witnessing and driving the evolution of our developer tools and infrastructure.
I joined Slack in 2018 to the newly formed test infra team being one of the 2 engineers. During my time in the test infra team I worked on scaling CI demands as Slack was rapidly growing. In 2021, we formed a new team called dev infrastructure with 2 engineers where I worked on the remote development project.
One of the most impactful initiatives I've led is our transition to remote development environments. This shift has dramatically improved developer productivity and collaboration while solving some thorny challenges around consistency and onboarding.
TL;DR
Slack's transition to remote development environments significantly boosted developer productivity and collaboration.
Reduced onboarding time, eliminated OS issues, and enabled working on parallel tasks.
Prioritize user feedback, flexibility, performance, and ease to use.
In this article, I'll share our journey, the key benefits we've realized, and some lessons learned that could help other organizations considering a similar move.
The Challenge: Local Development Friction
Like many fast-growing companies, Slack faced increasing friction with local development as our codebase and team expanded.
At the heart of our challenges lay Slack's web app repository - a massive monorepo containing the core of Slack's codebase. Written in Hack, this repository was becoming increasingly difficult to work with locally.
Some key pain points we encountered:
Long setup times for new developers to get a working local environment
Inconsistencies between developer machines leading to "works on my machine" issues
Challenges keeping local environments in sync with production, especially around dependencies
Performance issues on laptops when running resource-intensive builds and tests
Difficulty collaborating on code in real-time across distributed teams
With each OS update, our developers found themselves grappling with compatibility issues, spending precious time making HackLang work on their particular operating system or macOS version.
These friction points were slowing down our velocity and causing developers to be frustrated. We knew we needed a solution to alleviate these pain points and allow our developers to focus on what they do best—writing code.
Moving to Remote Development
After extensive research and experimentation, we decided to invest in building out remote development environments for our engineering org. The core idea is simple but powerful - instead of each developer maintaining their own local setup, we provision cloud-based environments that are pre-configured with all the necessary tools, dependencies, and code.
Here's how we approached it:
Leveraging existing tools: We built on top of Slack's internal CLI tool, which engineers were already familiar with. This allowed us to add remote development capabilities without forcing a completely new workflow.
Automated environment provisioning: We set up a pool of EC2 instances pre-loaded with our codebase, dependencies, and tooling.
Ephemeral workspaces: Environments are temporary and can be easily spun up or torn down. This encourages experimentation and prevents configuration drift.
Production parity: The remote environments closely match our production infrastructure, reducing surprises when code is deployed.
The results were dramatic. New developer onboarding time dropped significantly, OS compatibility issues were largely eliminated, and engineers could easily work on multiple tasks in parallel by spinning up separate environments.
This solution dramatically reduced the friction developers faced with local environments. No longer did they have to wrestle with setup and compatibility issues. Instead, they could dive straight into coding.
Evolving the Developer Experience
As we rolled out remote development, we learned valuable lessons about creating a great developer experience.
User feedback drives improvement
We initially focused on CLI-based interactions, but user feedback showed a demand for a more visual interface. This led us to develop a web dashboard for managing environments.
Flexibility is key
While we standardized the core environment, we made sure to support customization through dotfiles and user-specific configurations. This allowed developers to tailor their setup while maintaining consistency.
Performance matters
We continuously optimized environment startup times and resource allocation to ensure a snappy experience. Our current goal is to keep the P90 for spinning up a new environment under 90 seconds.
IDE integration is critical
We focused heavily on VS Code integration since it's the primary IDE used at Slack. Seamless remote development within familiar tools was essential for adoption.
Challenges and Future Considerations
As with any complex system, we face ongoing challenges:
Environment Familiarity: We're constantly working on ways to make each new environment feel familiar to developers, reducing the cognitive load of context switching.
Security: Balancing the need for developer freedom with company-wide security concerns is an ongoing challenge, especially when dealing with credentials on remote machines.
Tool Stability: We've encountered issues with VS Code's Remote SSH extension, which has occasionally disrupted work. We're exploring alternatives like browser-based IDEs to provide more stability and flexibility.
The Impact: Metrics and Feedback
The impact of our remote development solution has been significant and measurable. In our quarterly developer surveys, we've consistently received positive feedback about the remote environments. Within a year of implementation, we achieved an adoption rate of 80-85%, which has remained steady since then.
While there's always room for improvement, and we continue to address issues as they arise, the overall reception has been overwhelmingly positive. Our developers appreciate the streamlined workflow, faster onboarding, and increased productivity that our remote development environments provide.
Remote development environments transformed our engineering workflow. It's not just about speed—it's about removing friction and letting developers focus on what they do best: writing great code.
Future Directions: Exploring Nix and Kubernetes
As we continue to evolve our remote development solution, we're exploring new technologies to enhance our offering:
Nix: We're investigating Nix as a solution for managing dependencies on our host machines. This would allow us to mirror our code's container environments more closely, reducing version discrepancies.
Kubernetes Pods: For services that run on Kubernetes, we're working on a solution using EKS to spin up DevSpaces. This approach aims to maintain consistency between development and deployment environments.
These explorations are driven by our commitment to providing the best possible development experience while managing costs and maintaining consistency across our infrastructure.
Lessons Learned and Best Practices
While our transition to remote development has been overwhelmingly positive, it hasn't been without challenges. Here are some key lessons we've learned:
Invest in a great developer experience: The success of remote development hinges on providing developers a seamless, intuitive experience. We've invested heavily in tooling and automation to make spinning up and working in remote environments as frictionless as possible.
Plan for offline scenarios: While cloud-based development is great, developers still need to be productive when internet connectivity is limited. We've implemented solutions for syncing code locally and working offline when needed.
Security is paramount: Remote environments introduce new security considerations. We've worked closely with our security team to implement robust access controls, encryption, and monitoring.
Optimize for performance: Network latency can be a concern with remote development. We've focused on optimizing our setup to minimize lag and ensure a responsive experience.
Provide flexibility: While remote environments are our primary workflow, we still support local development for edge cases or personal preference. Giving developers options has been key to adoption.
Looking Ahead
We're constantly iterating on our solution, exploring new technologies, and seeking ways to make our developers' lives easier.
The journey of revolutionizing developer workflows at Slack has been challenging, rewarding, and enlightening. It's a testament to what's possible when we prioritize developer experience and aren't afraid to rethink established practices.
For teams considering a similar transition, my advice is to start small, focus relentlessly on developer experience, and be prepared to iterate based on feedback.
The future of development at Slack is remote, efficient, and exciting—and we're just getting started.
