Introducing External Secrets: Enterprise Secrets Management
Open Core Ventures (OCV) is proud to announce the launch of External Secrets, an enterprise secrets lifecycle manager built around the open-source project External Secrets Operator (ESO). A CNCF Sandbox project, ESO is a Kubernetes operator that integrates external secret systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, etc. It has quickly become a de facto secrets operator for Kubernetes environments.
External Secrets co-founders Lucas Severo Alves and Gustavo Carvalho, along with advisor Moritz Johner, have played pivotal roles in shaping the ESO project’s inception and growth. “The vision was to grow the project to be the de facto solution in the Kubernetes secret management space and donate it to the CNCF,” said Lucas. The project quickly gained popularity thanks to a centralized effort to build best practices into the technology. “In the beginning, there were a lot of projects that did something similar and we decided to centralize the effort in a single project,” said Lucas. “We were careful to listen to all the users and maintainers of other projects to follow best practices and make something that would work for everyone.”
After the project was successfully accepted by the CNCF in 2022, Lucas and Gustavo started to think about building a company. Partnering with OCV to build an open-core company was the jumpstart they were looking for. “There’s significant momentum behind the External Secrets Operator project evidenced by the project’s monthly activity,” said Betty Ma, COO at OCV. “It takes dedicated, ambitious open-source maintainers like Lucas and Gustavo to gain meaningful traction in open source. We’re thrilled to work with them to build a commercial enterprise offering.”
With funding from OCV, Lucas and Gustavo plan to build a holistic enterprise secrets management solution that will automate the most cumbersome parts of secret management like access management, workflow automation, and compliance. “No one really tackles the full lifecycle of a secret,” said Gustavo. “Instead of storing secrets, we focus on distribution—reading and writing between systems—so users can easily manage their secrets across all their systems.”
Taming secret sprawl
The co-founders worked together at a consultancy when the idea to create ESO arose after noticing a trend—big companies struggled with access maintenance and compliance. They didn’t know where their secrets were coming from, where they were going, how to access new secrets, or how to give access to new employees. “We had a lot of clients with the same problems and noticed we were doing the same work over again,” said Lucas. “We started looking for solutions we could reuse and creating an open-source project was one way to do that.”
A former platform engineer, Gustavo has felt the same pain as the clients they were serving and felt the urgency to solve the problem, “I’ve suffered the pain of trying to stay compliant when there is no good way to rotate secrets across 900 systems. When I started working at the consultancy I realized how big the problem was and thought, ‘We really need to solve this.’”
The security of an entire IT ecosystem depends on how well secrets like passwords, keys, APIs, and tokens are managed, and today’s IT environments are more complicated than ever. Hybrid multi-cloud architecture is the new normal, and with each service requiring its own access protocols, secrets are distributed across an ever-increasing surface area. Rapid decentralization has created secret sprawl making it harder to manage access across all systems—data breaches increased by 20% from 2022-23 compared to previous years.
Most platforms and cloud providers have native secrets managers built-in but they don’t integrate with other providers. If a credential is changed in one secret store of one system, it doesn’t automatically sync to other, external systems in use. “The majority of large organizations use more than one secret store and need a standardized way to control access,” said Sid Sijbrandij, General Partner at OCV. “Integrating multiple external secret store providers into a single interface could greatly simplify enterprise secrets management.”
Enterprise secret management
External Secrets is designed to synchronize secrets from external providers directly into Kubernetes clusters. For platform engineers and IT professionals, managing sensitive information such as database credentials, API keys, and other secrets is a constant challenge. External Secrets addresses this pain point by providing a seamless, secure, and future-proof method to handle these critical components. Its ability to integrate with multiple secret providers ensures that users are not locked into a single vendor. This flexibility allows organizations to maintain their current infrastructure while adopting new technologies without disruption.
Unlike a secret store, External Secrets doesn’t store any sensitive information. Using bi-directional syncing it retrieves and pushes information from system to system. “We support interfacing with custom client webhooks which a native secret store can’t do because they are locked into the provider,” said Gustavo. “Our templating feature allows platform engineers to manage access configuration from one file and then apply it across multiple systems.” Without a single solution to manage secrets across systems, an engineer manually changes each system—a time-consuming process.
While the technology is currently Kubernetes-specific, the plan is to extend functionality to other systems. “Sensitive data management is a common problem,” said Lucas. “One of our goals is to use the existing patterns and apply them to other environments.”
Taking ESO from a Kubernetes operator to an enterprise-ready secrets management solution includes adding front-end capabilities like dashboards and automated secret store discovery and access management. “Ultimately, ESO is a cog in the engine,” said Lucas. “We built ESO and now need to build the engine to deliver a holistic enterprise secrets management solution.”