The Challenges of Maintaining a Large Monorepo
Stripe’s engineering team faced a significant challenge in managing their extensive codebase, which was housed in a single large monorepo. As the company grew from a few hundred engineers to over a thousand, the complexity of their development environment increased exponentially. The developer productivity team, responsible for maintaining the tools and infrastructure that supported Stripe’s engineers, had to navigate a web of technical and organizational obstacles to ensure a seamless development experience.
Scaling the Monorepo
One of the primary challenges Stripe encountered was the sheer scale of their monorepo. With over 15 million lines of code spread across 150,000 files, the codebase had become unwieldy, testing the limits of the git hosting solution they had in place. As the monorepo continued to grow, the team had to grapple with the implications of this size, including the impact on build times, code navigation, and overall developer productivity.
Maintaining Consistent Environments
Keeping developer environments up-to-date and consistent was another major hurdle. When dependencies and configurations were managed at the individual laptop level, it was a constant struggle to ensure that all engineers were working with the same versions of tools and libraries. This led to frequent issues where changes would break in unexpected ways due to differences in local environments.
Collaborating and Debugging
Collaborating on code and debugging issues was also complicated by the distributed nature of the development environment. With engineers located around the globe, the team had to find ways to facilitate efficient collaboration and provide centralized support for troubleshooting environment-related problems.
Balancing Flexibility and Consistency
Stripe’s developer productivity team had to strike a delicate balance between providing a standardized, reliable development environment and maintaining the flexibility that engineers desired. While a centralized approach offered advantages in terms of consistency and maintainability, it also had the potential to stifle individual preferences and creativity.
Stripe’s Approach to Developer Environments
To address these challenges, Stripe’s developer productivity team implemented a comprehensive solution that leveraged a combination of cloud-based infrastructure and custom tooling.
Centralized Development Environments
The core of Stripe’s approach was the use of centralized, cloud-based development environments, known as “devboxes.” These were ephemeral virtual machines provisioned on-demand, providing engineers with a consistent and reliable environment for their work.
By moving the execution of code to the cloud, the team was able to address several key issues:
-
Dependency and Configuration Management: With the devboxes, the responsibility for managing dependencies and configurations shifted from individual laptops to the centralized infrastructure. This allowed the team to ensure that all engineers were working with the same versions of tools and libraries, reducing the risk of environment-related issues.
-
Collaboration and Debugging: The shared nature of the devboxes enabled engineers to easily collaborate on code and debug issues. By SSH-ing into a devbox, engineers could access the same environment, facilitating real-time troubleshooting and code review.
-
Compute Resources: The cloud-based devboxes provided ample computing resources, allowing engineers to work with large codebases and complex services without being constrained by the limitations of their local machines.
Seamless Code Synchronization
To bridge the gap between the local development environment and the cloud-based devboxes, Stripe implemented a sophisticated code synchronization process. Engineers continued to use their preferred local editors and IDEs, with the code residing on their laptops. However, a custom “sync” tool, powered by the Watchman file-monitoring system, automatically and transparently synchronized changes to the devboxes.
This approach allowed engineers to maintain their preferred development workflows while benefiting from the consistency and reliability of the centralized environments.
Integrated Development Tooling
To further enhance the developer experience, Stripe’s developer productivity team built a suite of custom tools that integrated seamlessly with the devbox infrastructure. This included a command-line interface (CLI) tool called “pay,” which provided a unified way for engineers to interact with their devboxes, run tests, and execute other development workflows.
The team also integrated the Sorbet type checker, Stripe’s Ruby type system, into the Visual Studio Code (VSCode) editor. By running the Sorbet language server on the devbox, engineers could benefit from advanced code intelligence features like autocomplete and go-to-definition, without the performance overhead of running the server locally.
The Tradeoffs and Lessons Learned
Stripe’s approach to developer environments, while highly effective, was not without its tradeoffs and challenges. The developer productivity team had to carefully navigate various technical and organizational considerations to ensure the overall success of the system.
Balancing Flexibility and Centralization
One of the key challenges was finding the right balance between providing a centralized, reliable development environment and maintaining the flexibility that engineers desired. While the devbox model offered numerous benefits, it also introduced some constraints, such as the need to manage code synchronization and the limitations of running development tools on remote infrastructure.
The team had to work closely with engineers to understand their needs and preferences, and to find ways to integrate the centralized tools and workflows seamlessly into their daily development practices.
Maintaining Reliability and Scalability
Ensuring the reliability and scalability of the developer environment infrastructure was another significant challenge. As Stripe’s engineering team grew, the developer productivity team had to continuously invest in the stability and performance of the devbox system, as well as the various supporting tools and services.
This included addressing issues like network latency for international engineers, optimizing the code synchronization process, and ensuring that the devbox provisioning and management systems could handle the increasing demand.
Balancing Centralized and Decentralized Approaches
While the monorepo approach offered numerous benefits, it also presented some unique challenges. The developer productivity team had to navigate the tension between the need for centralized control and the desire for decentralized decision-making and autonomy.
For example, the team had to find ways to support the development of services and components outside the main monorepo, while still ensuring a consistent and reliable development experience across the entire organization.
The Ongoing Evolution of the Developer Environment
Stripe’s developer environment was not a static solution, but rather an ongoing work in progress. As the company’s needs and the engineering landscape evolved, the developer productivity team had to continuously adapt and refine their approach.
This included incorporating new technologies, such as improvements to the Sorbet type checker and the adoption of Bazel for build and dependency management. It also required the team to stay attuned to the changing needs and preferences of Stripe’s engineers, and to be responsive to their feedback and suggestions.
Conclusion
Stripe’s approach to developer environments, as described in this article, highlights the significant challenges and tradeoffs involved in maintaining a large-scale monorepo development environment. The developer productivity team’s efforts to create a centralized, reliable, and flexible development experience offer valuable insights for organizations of all sizes, particularly those grappling with the complexities of managing extensive codebases and supporting the evolving needs of their engineering teams.
While Stripe’s specific solutions may not be a perfect fit for every organization, the principles and lessons learned from their experience can serve as a valuable guide for IT professionals and engineering leaders looking to optimize their own development environments and improve overall productivity and collaboration.
