Overcoming the Secrets Management Challenge with Hashicorp Vault and the Hashi-Tools Library

Hashicorp Vault is a “secrets” management system, one of the various Hashicorp open-source tools used by the Digital Turbine DevOps team.

This post explains how we use the Vault at Digital Turbine, as well as our new, open-sourced “Hashi-Tools” library.

The security challenge

About a year ago, we deployed Vault to our production environment.

We embraced Vault to eliminate certain bad practices that are quite common in the industry; storing credentials statically in a configuration file or even hard-coded within the project’s repository.

Aside from the obvious security issue, this practice usually leads to a situation where multiple services share the same credentials. As a result, these credentials require access to many resources. This breaks the least-privileges principle and makes the revoking process of these keys – if they are compromised – very challenging.

For this reason (and others..), we wanted to change the way in which we work with secrets.

We figured that it would be best to have:

Secrets for applications and systems in one secure, centralized place.
Dynamic and programmatic application access; services can obtain temporary, unique credentials (called a lease) on demand, on runtime, per service.
A solution for the “first introduction” problem, where a service must hold a secret to be able to access other systems, either directly or through a third party.
Easy key revocation.
Secure auditing.

Vault seemed like a natural fit. It technically provides all of the above – supporting dynamic secrets for all popular systems and tools (cloud providers, databases, ssh, etc.) and providing a central, secure and configurable place to store secrets. In addition, it offers a variety of authentication methods which perform authentication and assigning policies to a user and services and finally, it has automatic revocation and auditing.

If you are not familiar with Vault, take a few minutes to review the documentation here.

The plan

In Vault, we use policies to restrict access, enforcing the “need to know” principle and implementing a “Role-Based Access Control” by specifying access privileges.

For each backend (AWS/Mysql, etc.), we define roles with a different set of permissions granted.

We want that each service can obtain credentials on demand, for each system, generated by the Vault server with the appropriate permissions. When a service is replaced, rebooted or dies – its credentials should be revoked.

We also like to use Vault as a “secret vault” to store constant sensitive information there, as we do with LastPass.

The flow should then be something like this:

The implementation challenge

We constructed a Vault cluster. The remaining implementation of the above workflow is on the developer’s side. I feel safe in claiming that for most developers, dealing with secrets is a no-brainer. Credentials with more than enough permissions either find their way to a configuration file inside the node, or they just all have access to them and they can put them wherever they like in the Git repo. With Vault, to be able to communicate with the Vault server, the developer implements an authentication mechanism in each project.

This is how it is done:

Store a pre-defined Vault token
Most of our services are dynamic in the sense that they scale up and down, so creating a mechanism where we generate and place a Vault token in the service config file is not ideal, and anyway, we prefer not to keep any secret physically inside a node.
Use a Vault authentication method
Vault has pluggable authentication methods, making it easy to authenticate with Vault, using whatever form works best for you. For example, you can authenticate using your personal GitHub access token. However, as in this example, there is a need to store a secret physically, and the implementation is not always trivial. In both cases, the temporary token must be periodically renewed by the application.

After logging-in, to obtain credentials from the Vault server, an HTTP call with the appropriate properties is required; and then parse the response and periodically renew the lease, along with the regulars, such as handle failures and monitoring.

The ‘hashi-tools’ library

To ease the migration of the secrets management to Vault, I wrote an SDK library. Most of our services are written in Scala, and so is the client.

The library contains two clients:

Secrets for Vault
Discovery for Consul

The library is used by all of our backend applications written in the past year.

So then let’s update the workflow:

Main features

The main features for Secrets are:

Scala-based
Friendly DSL (also with Java applications)
Implementation of the AWS AMI method
Renews its Vault token automatically, if required
Creates leases for Secret Engines and renews them automatically
It exposes operational metrics
It can be easily configured with the ‘application.conf’ file or with a Typesafe object
Async functionality

The full documentation can be found in the project’s repository.