Vault Agent with AWS
You must pass an authentication token with any request from an authenticated endpoint in Vault. This includes all API requests, as well as via the Vault CLI and other libraries.
Vault supports several different authentication methods to enable delivery of the initial token. If you can securely get the first secret from an originator to a consumer, then all secrets later exchanged between them can be authenticated. Getting that first secret to the consumer is the secure introduction challenge.
![Secure Introduction](/_next/image?url=https%3A%2F%2Fmktg-content-api.vercel.app%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault%252Fdiagram-secret_consumer-and-originator.png%26width%3D675%26height%3D130&w=1920&q=75)
To that end, Vault provides integration with native authentication capabilities in various environments. For example: IAM in AWS and Google Cloud, Managed Service Identities in Azure, and Service Accounts in Kubernetes are all supported.
Challenge
Although Vault provides the auth method, the client is still responsible for managing the lifecycle of tokens created from the auth method. The challenge then becomes how to enable authentication to Vault, and manage token lifecycle in a standard way without writing custom logic.
Solution
Vault Agent provides a number of different helper features, specifically addressing the following challenges:
- Automatic authentication
- Secure delivery/storage of tokens
- Lifecycle management of these tokens (renewal & re-authentication)
NOTE: The Vault Agent Auto-Auth functionality addresses the challenges related to obtaining and managing authentication tokens (secret zero).
Note
The Secure Introduction of Vault Clients introduced three basic approaches: Platform Integration, Trusted Orchestrator, and Vault Agent. This tutorial demonstrates how Vault Agent works.
Prerequisites
To complete this section of the tutorial, you need the following:
- Terraform v1.0.0 or later installed.
- AWS account and associated credentials that allow for the creation of resources.
tree
utility for visualizing directory contents (you can usels -1R
instead).
Provision the cloud resources
Clone the demo assets from the HashiCorp Education repository GitHub repository to perform the steps described in this tutorial.
$ git clone https://github.com/hashicorp-education/learn-vault-agent-demo.git
This repository has supporting content for all the Vault learn tutorials. You can find the content for this tutorial within a sub-directory.
Be sure to set your working directory to location of the
learn-vault-agent-demo/terraform-aws
folder.$ cd learn-vault-agent-demo/terraform-aws
The working directory should contain the provided Terraform files:
$ tree . ├── aws.tf ├── iam.tf ├── kms.tf ├── network.tf ├── outputs.tf ├── security-groups.tf ├── templates │ ├── userdata-vault-client.tftpl │ └── userdata-vault-server.tftpl ├── terraform.tfvars.example ├── variables.tf ├── vault-client.tf ├── vault-server.tf └── versions.tf 2 directories, 13 files
NOTE: The example Terraform configuration in this repository is for demonstration purposes, and not suitable for production use. For production deployment, refer the example Terraform in the Vault Guides repository /operations/provision-vault.
Set an
AWS_ACCESS_KEY_ID
environment variable to hold your AWS access key ID.$ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"
Set an
AWS_SECRET_ACCESS_KEY
environment variable to hold your AWS secret access key.$ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"
Tip
The above example uses IAM user authentication. You can use any authentication method described in the AWS provider documentation.
Copy the
terraform.tfvars.example
file and rename it asterraform.tfvars
. Within this file, edit thekey_name
parameter to be the name of your EC2 key pair.$ cp terraform.tfvars.example terraform.tfvars
Example:
terraform.tfvars
# SSH key name to access EC2 instances (should already exist) key_name = "vault-test"
You should also change the values of the AWS region and availability zone or instance type as necessary.
Tip
If you don't have an EC2 key pair, follow the AWS documentation to create one.
Initialize the Terraform workspace, and download the necessary provider resources.
$ terraform init Initializing provider plugins... ...snip... Terraform has been successfully initialized!
Run
terraform apply
and review the planned actions. Your terminal output should show the plan, resources Terraform will provision.$ terraform apply ...snip... Plan: 20 to add, 0 to change, 0 to destroy. Changes to Outputs: + endpoints = (known after apply) Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes
Enter
yes
to confirm and resume.When the
apply
command completes, the Terraform output will display the public IP address to SSH into your Vault server and client instances.Example output:
Apply complete! Resources: 20 added, 0 changed, 0 destroyed. Outputs: endpoints = Vault Server IP (public): 54.219.129.15 Vault Server IP (private): 10.0.101.50 For example: ssh -i vault-test.pem ubuntu@54.219.129.15 Vault Client IP (public): 54.183.212.51 Vault Client IP (private): 10.0.101.209 For example: ssh -i vault-test.pem ubuntu@54.183.212.51
Configure AWS IAM auth method
Note
Perform this step on the Vault server instance.
In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.
SSH into the Vault server instance.
At the prompt, enter "yes" to continue.
$ ssh -i <path_to_key> ubuntu@<public_ip_of_server> ... Are you sure you want to continue connecting (yes/no)? yes
Tip
If you received
Permissions 0664 for '<key_name>.pem' are too open
error, be sure to set the file permission appropriately.$ chmod 600 <key_name>.pem
To verify your Vault installation, run
vault status
command and notice that Initialized isfalse
.$ vault status Key Value --- ----- Recovery Seal Type awskms Initialized false Sealed true Total Recovery Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version 1.12.2 Build Date 2022-11-23T12:53:46Z Storage Type raft HA Enabled true
Run the
vault operator init
command to initialize the Vault server:$ vault operator init ...snip... Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq Success! Vault is initialized ...snip...
Tip
The Vault server uses auto-unseal with AWS Key Management Service (KMS). When you initialize the Vault, the server is automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.
Copy the Initial Root Token value.
Log into Vault. Enter the generated initial root token when prompted.
$ vault login Token (will be hidden):
Examine and then execute the
/home/ubuntu/aws_auth.sh
script.$ cat aws_auth.sh
The script enables key/value v1 secrets engine at
secret
and writes some test data atsecret/myapp/config
(at line 1 and 2).At line 4 through 6, it creates a
myapp
policy, and line 8 enables theaws
auth method.It creates a role named
dev-role-iam
which is valid for 24 hours with capabilities set by themyapp
policy. (Note: The${account_id}
should be your AWS account ID.) Thebound_iam_principal_arn
is the Amazon Resource Name (ARN) of the IAM role created by theiam.tf
file (at line 11).aws_auth.sh
vault secrets enable -path="secret" kv vault kv put secret/myapp/config ttl='30s' username='appuser' password='suP3rsec(et!' echo "path \"secret/myapp/*\" { capabilities = [\"read\", \"list\"] }" | vault policy write myapp - vault auth enable aws vault write -force auth/aws/config/client vault write auth/aws/role/dev-role-iam auth_type=iam bound_iam_principal_arn="arn:aws:iam::${account_id}:role/vault-agent-demo-vault-client-role" policies=myapp ttl=24h
Execute the script.
$ ./aws_auth.sh Success! Enabled the kv secrets engine at: secret/ Success! Data written to: secret/myapp/config Success! Uploaded policy: myapp Success! Enabled aws auth method at: aws/ Success! Data written to: auth/aws/config/client Success! Data written to: auth/aws/role/dev-role-iam
Check the
myapp
policy.$ vault policy read myapp path "secret/myapp/*" { capabilities = ["read", "list"] }
Check the secrets written in
secret/myapp/config
.$ vault kv get secret/myapp/config ====== Data ====== Key Value --- ----- password suP3rsec(et! ttl 30s username appuser
Run Vault Agent with auto-auth
Note
Perform this step on the Vault client instance.
Now, you are going to see how Vault Agent Auto-Auth method works, and write out a token to an arbitrary location on disk. Vault Agent is a client daemon and its Auto-Auth feature allows for easy authentication to Vault.
![Vault Agent](/_next/image?url=https%3A%2F%2Fmktg-content-api.vercel.app%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault%252Fdiagram-vault_agent-workflow.png%26width%3D850%26height%3D256&w=1920&q=75)
Open a new terminal and SSH into the Vault Client instance.
At the prompt, enter "yes" to continue.
$ ssh -i <path_to_key> ubuntu@<public_ip_of_client> ... Are you sure you want to continue connecting (yes/no)? yes
In the client instance, explore the Vault Agent configuration file (
/home/ubuntu/vault-agent.hcl
).$ cat vault-agent.hcl
Starting at line 4, the
auto_auth
block has two configuration entries:method
andsink
. In this example, Auto-Auth uses theaws
auth method enabled at theauth/aws
path on the Vault server. The Vault Agent will use thedev-role-iam
role to authenticate.The
sink
block specifies the location on disk where to write tokens. You can define the Vault Agent Auto-Authsink
more than once if you want Vault Agent to place the token into additional locations. In this example, thesink
path is/home/ubuntu/vault-token-via-agent
.The
exit_after_auth
parameter istrue
. This means that the agent will exit with code 0 after a single successful authentication. The default isfalse
and the agent continues to run and automatically renew the client token for you.vault-agent.hcl
exit_after_auth = true pid_file = "./pidfile" auto_auth { method "aws" { mount_path = "auth/aws" config = { type = "iam" role = "dev-role-iam" } } sink "file" { config = { path = "/home/ubuntu/vault-token-via-agent" } } } vault { address = "http://10.0.101.53:8200" }
The
vault
block points to the Vault serveraddress
. This should match to the private IP address of your Vault server host.Tip
For the full details of Vault Agent configuration parameters, refer to the Vault Agent documentation.
Execute the following command to get help:
$ vault agent -h
Now, you are ready to run the Vault Agent. Execute the following command:
$ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug ==> Vault agent started! Log data will stream in below: ==> Vault agent configuration: Cgo: disabled Log Level: debug Version: Vault v1.12.2, built 2022-11-23T12:53:46Z Version Sha: 415e1fe3118eebd5df6cb60d13defdc01aa17b03 2023-01-25T16:55:56.473Z [INFO] sink.file: creating file sink 2023-01-25T16:55:56.473Z [INFO] sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent mode=-rw-r----- 2023-01-25T16:55:56.483Z [DEBUG] would have sent systemd notification (systemd not present): notification=READY=1 2023-01-25T16:55:56.483Z [INFO] template.server: starting template server 2023-01-25T16:55:56.483Z [INFO] template.server: no templates found 2023-01-25T16:55:56.484Z [INFO] auth.handler: starting auth handler 2023-01-25T16:55:56.484Z [INFO] auth.handler: authenticating 2023-01-25T16:55:56.484Z [INFO] sink.server: starting sink server 2023-01-25T16:55:57.562Z [INFO] auth.handler: authentication successful, sending token to sinks 2023-01-25T16:55:57.562Z [INFO] auth.handler: starting renewal process 2023-01-25T16:55:57.562Z [INFO] sink.file: token written: path=/home/ubuntu/vault-token-via-agent 2023-01-25T16:55:57.562Z [INFO] sink.server: sink server stopped 2023-01-25T16:55:57.563Z [INFO] sinks finished, exiting 2023-01-25T16:55:57.563Z [INFO] template.server: template server stopped 2023-01-25T16:55:57.563Z [INFO] auth.handler: shutdown triggered, stopping lifetime watcher 2023-01-25T16:55:57.563Z [INFO] auth.handler: auth handler stopped 2023-01-25T16:55:57.563Z [DEBUG] would have sent systemd notification (systemd not present): notification=STOPPING=1
Because the
vault-agent.hcl
configuration file contained the lineexit_after_auth = true
, Vault Agent authenticated and retrieved a token once, wrote it to the defined sink, and exited. Vault Agent can also run in daemon mode where it will continuously renew the retrieved token, and try to re-authenticate if that token becomes invalid. Vault Agent Caching tutorial will show running Vault Agent as a daemon.Vault Agent writes the token to
/home/ubuntu/vault-token-via-agent
.$ more vault-token-via-agent hvs.CAESIH5hFzxXxBABzyeq78mHlbQZsN8ETqaEQX24XbEZHpdyGh4KHGh2cy5oNjZrMnpyWG02T01xZURkQTVoTDRwbkk
Export a VAULT_ADDR environment variable to address the Vault server directly.
$ export VAULT_ADDR=$(grep address vault-agent.hcl | awk '{print $3}' | tr -d '"')
Try an API call using the token that Vault Agent retrieved to test:
$ curl \ --silent \ --header "X-Vault-Token: $(cat /home/ubuntu/vault-token-via-agent)" \ $VAULT_ADDR/v1/secret/myapp/config | jq -r ".data"
Output:
{ "password": "suP3rsec(et!", "ttl": "30s", "username": "appuser" }
Response wrap the token
Note
Perform this step on the Vault client instance.
It may not be ideal to write the token as a plaintext depending on the firewall around the client instance. Vault Agent supports response-wrapping of the token to offer an additional layer of protection for the token. You can wrap tokens by either the auth method or by the sink configuration, with each approach solving for different challenges as described in Response-Wrapping Tokens.
In the client instance, explore the
/home/ubuntu/vault-agent-wrapped.hcl
file which supports response-wrapping of the token.$ cat /home/ubuntu/vault-agent-wrapped.hcl exit_after_auth = true pid_file = "./pidfile" auto_auth { method "aws" { mount_path = "auth/aws" config = { type = "iam" role = "dev-role-iam" } } sink "file" { wrap_ttl = "5m" config = { path = "/home/ubuntu/vault-token-via-agent" } } } vault { address = "http://10.0.101.53:8200" }
Notice the
wrap_ttl
parameter in thesink
block. This configuration uses the sink method to response-wrap the retrieved tokens.From the client, run the Vault Agent again and inspect the output:
$ vault agent -config=/home/ubuntu/vault-agent-wrapped.hcl -log-level=debug ==> Vault agent started! Log data will stream in below: ==> Vault agent configuration: Cgo: disabled Log Level: debug Version: Vault v1.12.2, built 2022-11-23T12:53:46Z Version Sha: 415e1fe3118eebd5df6cb60d13defdc01aa17b03 2023-01-25T17:03:13.056Z [INFO] sink.file: creating file sink 2023-01-25T17:03:13.056Z [INFO] sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent mode=-rw-r----- 2023-01-25T17:03:13.065Z [DEBUG] would have sent systemd notification (systemd not present): notification=READY=1 2023-01-25T17:03:13.065Z [INFO] template.server: starting template server 2023-01-25T17:03:13.065Z [INFO] template.server: no templates found 2023-01-25T17:03:13.065Z [INFO] auth.handler: starting auth handler 2023-01-25T17:03:13.065Z [INFO] auth.handler: authenticating 2023-01-25T17:03:13.066Z [INFO] sink.server: starting sink server 2023-01-25T17:03:14.121Z [INFO] auth.handler: authentication successful, sending token to sinks 2023-01-25T17:03:14.121Z [INFO] auth.handler: starting renewal process 2023-01-25T17:03:14.128Z [INFO] auth.handler: renewed auth token 2023-01-25T17:03:14.144Z [INFO] sink.file: token written: path=/home/ubuntu/vault-token-via-agent 2023-01-25T17:03:14.144Z [INFO] sink.server: sink server stopped 2023-01-25T17:03:14.144Z [INFO] sinks finished, exiting 2023-01-25T17:03:14.144Z [INFO] template.server: template server stopped 2023-01-25T17:03:14.144Z [INFO] auth.handler: shutdown triggered, stopping lifetime watcher 2023-01-25T17:03:14.144Z [INFO] auth.handler: auth handler stopped 2023-01-25T17:03:14.145Z [DEBUG] would have sent systemd notification (systemd not present): notification=STOPPING=1
Instead of a token value, you now have a JSON object containing a wrapping token as well as some additional metadata. To get to the true token, you need to first perform an
unwrap
operation.Unwrap the response-wrapped token and save it to a
VAULT_TOKEN
environment variable that other applications can use:$ export VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /home/ubuntu/vault-token-via-agent))
View the unwrapped token value.
$ echo $VAULT_TOKEN hvs.CAESIM0bghylBC3fTJ2RWV-9ILquhTEgUaf1pZOdzIGyP1hnGh4KHGh2cy5rME90YzBHUkxHeVNpZXNhdXhEN3BtdUo
Test to make sure that the token has the read permission on
secret/myapp/config
.$ curl \ --silent \ --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/secret/myapp/config \ | jq -r ".data"
Notice that the value saved to the
VAULT_TOKEN
is not the same as thetoken
value in the/home/ubuntu/vault-token-via-agent
file. The value inVAULT_TOKEN
is the unwrapped token retrieved by Vault Agent.If you try to unwrap that same value again or wait longer than 5 minutes (
wrap_ttl
), it will throw an error:$ export VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /home/ubuntu/vault-token-via-agent)) Error unwrapping: Error making API request. URL: PUT http://active.vault-va-demo.service.dc1.consul:8200/v1/sys/wrapping/unwrap Code: 400. Errors: * wrapping token is not valid or does not exist
You can unwrap a response-wrapped token just once. Additional attempts to unwrap an already-unwrapped token will result in triggering an error.
Note
In this tutorial, you learned about the basic mechanics of Vault Agent. Read the Additional Discussion section about the next step.
Clean up
Return to the first terminal where you created the cluster and use Terraform to destroy the cluster.
Destroy the AWS resources provisioned by Terraform.
$ terraform destroy -auto-approve
Delete the state file.
$ rm *tfstate*
Additional discussion
In the above examples, you manually ran Vault Agent to show how it works. How you actually integrate Vault Agent into your application deployment workflow will vary based on several factors. Some questions to ask to help decide appropriate usage:
- What is the lifecycle of my application? Is it more ephemeral or long-lived?
- What are the lifecycles of my authentication tokens? Are they long-lived and requiring repetitive renewal to show liveliness of a service or do you want to enforce periodic re-authentications?
- Do I have a group of applications running on my host which each need their own token? Can I use a native authentication capability (e.g. AWS IAM, K8s, Azure MSI, Google Cloud IAM, etc.)?
The answers to these questions will help you decide if Vault Agent should run
as a daemon or as a prerequisite of a service configuration. Take for example
the following Systemd
service definition for running Nomad:
[Unit]
Description=Nomad Agent
Requires=consul-online.target
After=consul-online.target
[Service]
KillMode=process
KillSignal=SIGINT
Environment=VAULT_ADDR=http://active.vault.service.consul:8200
Environment=VAULT_SKIP_VERIFY=true
ExecStartPre=/usr/local/bin/vault agent -config /etc/vault-agent.d/vault-agent.hcl
ExecStart=/usr/bin/nomad-vault.sh
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Notice the ExecStartPre
directive that runs Vault Agent before the desired
service starts. The service startup script expects a Vault token value set
as shown in the /usr/bin/nomad-vault.sh
startup script:
#!/usr/bin/env bash
if [ -f /mnt/ramdisk/token ]; then
exec env VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /mnt/ramdisk/token)) \
/usr/local/bin/nomad agent \
-config=/etc/nomad.d \
-vault-tls-skip-verify=true
else
echo "Nomad service failed due to missing Vault token"
exit 1
fi
Applications which expect Vault tokens typically look for a VAULT_TOKEN
environment variable. Here, you're using Vault Agent to get a token and write it out to a
RAM disk and as part of the Nomad startup script. You read the response-wrapped
token from the RAM disk, and save it to your VAULT_TOKEN
environment variable before starting Nomad.
Vault Agent in Vault version 1.3.0 or higher also supports secret templates for use in configuration files, and other similar use cases. You can learn more about this feature in Vault Agent Templates.
Help and reference
- Blog post: Why Use the Vault Agent for Secrets Management?
- Video: Streamline Secrets Management with Vault Agent and Vault 0.11
- Secure Introduction of Vault Clients
- Vault Agent Auto-Auth documentation
- AWS Auth Method documentation
Other Vault Agent tutorials:
- Vault Agent with Kubernetes
- Vault Agent Templates
- Vault Agent Caching
- Vault Agent Windows Service
- Using HashiCorp Vault Agent with .NET Core
To learn more about the response wrapping feature, refer the following: