Managing Kubernetes Secrets Using Sops and Age

References

Introduction

Many kubernetes deployments require some kind of key, password or other sensitive data. These are known collectively as 'secrets'. Managing the files containing secrets in a source control system like git can be difficult as you likely want to avoid storing clear text passwords and the like in a public repository. The normal solution is to exclude the secrets files from source control through the use of a .gitignore file. This solves the unintentional release of the secrets files in the repository, but they still need to be managed somehow.

The obvious solution is to encrypt the secrets files, but this then adds another level of complexity when it comes time to deploy the secrets file to a kubernetes cluster.

A Warning About Secrets in a Kubernetes Cluster

It is probably prudent to bring up an important point about kubernetes secrets before going any further. By default, secrets stored in a kubernetes cluster (they are actually stored within the etcd database) are not encrypted, and are stored in the clear in their base64 format. This can be seen by retrieving any existing secret from the system;

$ cat secret.yaml
---
#############################################
# - creds for website
# - generate value using;
#      echo -n '<text>' | base64
#############################################

apiVersion: v1
kind: Secret
metadata:
  name: some-pass
data:
  SOME_PASSWD: bm90IHJlYWxseSBhIHBhc3N3b3Jk 

# EOF

$ kubectl apply -f secret.yaml
secret/some-pass created

$ kubectl get secrets some-pass -o yaml
apiVersion: v1
data:
  SOME_PASSWD: bm90IHJlYWxseSBhIHBhc3N3b3Jk
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"SOME_PASSWD":"bm90IHJlYWxseSBhIHBhc3N3b3Jk"},"kind":"Secret","metadata":{"annotations":{},"name":"some-pass","namespace":"default"}}
  creationTimestamp: "2021-12-22T21:25:14Z"
  name: some-pass
  namespace: default
  resourceVersion: "19798883"
  uid: 5694ba95-39df-466f-a03c-93aaeafb9681
type: Opaque

As can be seen, the secret is stored essentially in the clear.

This can be partially remedied through careful access control configuration using RBAC, but a good fix is available using the information at Encrypting Secret Data at Rest. One thing to note about this though, which is highlighted in the article, is that while this will encrypt the secrets values in the system, the encryption key used in the EncryptionConfig object is still potentially visible for anyone that has access to this object, depending on the key management system selected. Proper access control configuration must be applied to the EncryptionConfig object.

All that being said, this post is about managing the secrets files themselves, and being able to store them securely in a source management system.

The 'Enterprise' Solution

The normal 'go-to' solution chosen to manage secrets in an enterprise environment is HashiCorp Vault. This is a well regarded enterprise level solution that can manage and protect all manners of sensitive data. It is also open source so it could be a deployed as a free self-managed solution, but it is a bit big (IMHO) for HomeLab use.

Another enterprise level system that is similar to HashiCorp Vault is CyberArk Conjur. Conjur is also open source.

A third alternative with an interesting twist on managing the secrets is Bitnami Sealed-Secrets. This solution is tightly integrated into kubernetes, and deploys a controller on the cluster that manages decryption of the secrets prior to deployment.

A Solid HomeLab Solution: Mozilla Sops + Age

The system I've chosen for my HomeLab is Mozilla sops. Sops is a utility that can encrypt specified key data values within a number of file formats, including yaml, json and others. You can encrypt just the required key data fields in a yaml secrets manifest file, and then that file can be safely stored in source control. When it comes time to deploy that manifest to kubernetes, sops can decrypt to standard output which can then be piped to kubectl.

Sops hands the encryption off to an external application, of which a number of options are available. Sops can use cloud providers such as Azure or GCP for encryption, but can also utilize local encryption utilities as well. PGP is one option, but the one I've chosen is age.

Environment Set-up and Configuration

Installation of both sops and age is relatively simple, as there are binaries available within the releases section of each repository.

Once both applications are installed the next step is creation of the age key file. I've chosen to store my key file in my $HOME/bin directory. If multiple users will be managing kubernetes secrets manifests, or if an automation system such as ansible will be used, the key file can be placed in a file system location appropriate to the requirement. Alternatively, multiple recipients could be specified during encryption (each with their own age key file), which would allow multiple users to each decrypt the file using their own key.

Please note that by default the age key file is not password protected, although it can be password protected, if desired (see the following example). If you choose to leave your key file without a password be sure to protect the destination directory and file appropriately.

$ age-keygen -o $HOME/bin/age-key.txt
Public key: age1x7aazmg26qf5vm7hnvxjqy77yvv5lc7jez7untjfnwrg8pa6aqysxlaa42

- to password protect your age key file, use the following;

$ age-keygen | age -p > $HOME/bin/age-key.age

After key creation, create two bash environment variables for sops. These aren't required, but do simplify sops usage by eliminating the need to specify the age recipient and age key file location as command line arguments to sops. The following should be added to the .bashrc for the user that manages kubernetes secrets manifest files;

export SOPS_AGE_RECIPIENTS="age1x7aazmg26qf5vm7hnvxjqy77yvv5lc7jez7untjfnwrg8pa6aqysxlaa42"
export SOPS_AGE_KEY_FILE="${HOME}/bin/age-key.txt"

Encrypting a Secrets Manifest File

Once sops and age are setup and configured, it is relatively simple to encrypt a secrets manifest file.

$ cat secret.yml 
---
#############################################
# - creds for website
# - generate value using;
#      echo -n '<text>' | base64
#############################################

apiVersion: v1
kind: Secret
metadata:
  name: some-pass
data:
  SOME_PASSWD: bm90IHJlYWxseSBhIHBhc3N3b3Jk 

# EOF

$ sops --encrypt --encrypted-regex '^data' secret.yml >secret.enc.yml

$ cat secret.enc.yml
#############################################
# - creds for website
# - generate value using;
#      echo -n '<text>' | base64
#############################################
apiVersion: v1
kind: Secret
metadata:
    name: some-pass
data:
    SOME_PASSWD: ENC[AES256_GCM,data://OIS0cajpG3mI6c832Hauy+R/voNPw4M1q3/Q==,iv:jGi0FIwI/ZqPFmb8Re68VC/m/QzB3WtlAQG88OCzlO4=,tag:gMajKzNRrcwCkFLhoMo4TA==,type:str]
# EOF
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1x7aazmg26qf5vm7hnvxjqy77yvv5lc7jez7untjfnwrg8pa6aqysxlaa42
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZRFFvS3FMU1g5Q3ZTUjRm
            WVBJcEQ4TkhQb0dBbEZ6Rm9VallEdmtZZXo4Ck13aEw4Q043SGdtWXlzVWhCNk9u
            c2ZiK1VvNHpUV3lxaGxzR3craHB5aW8KLS0tIGxsa044dGtjeEZSeTBYQ2lzS2E4
            bFlxL0dNSjlESmtlcFdFc0FYTzBwS0EKvksaFFkx1PEw9ULPVWNOtqcRobV9VdFm
            ZpydHNaF9EQrhtTR+dvJZp8BZMQEaJwZQN8F3gQ71z955Ryd7TYYUQ==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2021-12-21T18:04:54Z"
    mac: ENC[AES256_GCM,data:QRyBSBD2JdAaXc1Xm9rut+c8aiWBtG8MZt2H7WpH+vyw3UUFtNOwbtwm4n+TCadDwY8Exg+8+k3M6hRIF6+wBpWIKtXd53TRbDs09aZhxY4v6q8ak5yoIcOgF3KSKGlL+tHYBLYSoPbqNGGgCNJlEvWou1UH2MRmyEMEBy6NGNE=,iv:FScvzwzAczwq5vWsVtvbnjoIcyUK0g1MrdiYrnR8nTg=,tag:pzmeeR7G+I5Nds06FxvG4w==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.7.1

The --encrypted-regex '^data' option directs sops to only encrypt data values under the 'data:' key. If you don't specify what key data values to encrypt sops will encrypt the data values for all keys it finds in the target file.

Once you have the secrets manifest file encrypted you can remove the unencrypted file.

Decryption and Deployment of an Encrypted Secrets Manifest File

Using the encrypted secrets manifest file is relatively easy, with only a slightly more complicated command line usage;

$ sops --decrypt secret.enc.yml | kubectl apply -f -
secret/some-pass created

Docker ENV Files

While this post has been primarily targeted at using sops + age for kubernetes secrets manifest files, it can also be used for docker env files. This then allows storage of the env files in source control as well.

$ cat docker-app.env
USER=some-user
PASS=some-password

$ sops -e docker-app.env >docker-app.enc.env

$ cat docker-app.enc.env 
USER=ENC[AES256_GCM,data:XsA4Vmcqgs9s,iv:lFnUUSogZ6ijiMgQsjCxJxpTzN/PoK4c+DJTH71ah/w=,tag:tXRoCH7mdUmWvvpSP4/A6A==,type:str]
PASS=ENC[AES256_GCM,data:YkK3gFxOz9GMeKGP0g==,iv:s4XxwBoRNUfK+PMbwE7QsJhEw+bD5NWSz5Sm73FiBoA=,tag:XWS6Cr+AIOFSA9rc5QV/jw==,type:str]
sops_age__list_0__map_enc=-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIdWNTSUxpMmVnWjhCbHFi\nZC9BYllWZHVCVWdibUFTMGZsZ0UyamI0dFZ3CmpTNHBPK09WSldPbyswSDlQVFNx\ncklxRUJTcG01dHJPckVWL2pUdHJWSnMKLS0tICtVRWVvV2s5STFJbDlLeXNobm5z\naHhnU1BlYTdwM1REdjBaekYwcDljZjgKfeE0kL2ScHXzDBL0j1tWPRte/FpeikQ0\nhmhDi7mWPII12RMp34MryN72RmFi79ET5VphYEYPSXwr5IyE+0g4Gg==\n-----END AGE ENCRYPTED FILE-----\n
sops_lastmodified=2021-12-22T22:04:36Z
sops_unencrypted_suffix=_unencrypted
sops_version=3.7.1
sops_age__list_0__map_recipient=age1x7aazmg26qf5vm7hnvxjqy77yvv5lc7jez7untjfnwrg8pa6aqysxlaa42
sops_mac=ENC[AES256_GCM,data:Slr4iwrZJ2iHymCWWnq4jJ1iWfkRWu3iyEZTsyeZuvJ1vg9CLG+JijIA8prNp2E3Ts7P/k278QPe0pVZ8rc/oRisFyF1nRl2GoWrm2RxLxQ/wFihYDnSYkSXAHGM43Ml7gFr2FgmLskCggkaI+P6oudmnn+WVRqrpBe1VJZfzgA=,iv:8dSlp8BgFZrPXA312mnaehuWIesvvgfIo5tMuqmrOp8=,tag:vEhIHfLnIQcBurOMWCtb3w==,type:str]

The unencrypted env file can then be deleted.

To use the encrypted env file, simply decrypt it in place once the repository has been deployed to the destination docker host. Alternatively it can be decrypted on the management host and then copied to the destination docker host. The source env filename (ie, docker-app.env) should be added to a .gitignore file, so that the decrypted env file won't inadvertently be added back into the source control.

Created: 2021-12-22 19:47
Last update: 2021-12-27 20:21