This article provides a hands-on walkthrough demonstrating how to use the Conjur Secretless Broker in a Kubernetes cluster using the Secretless Broker Sidecar. We’ll begin by going through the steps to set up the Conjur Secretless Broker in a Kubernetes cluster. Note that an administrator generally performs these steps. Then, we’ll demonstrate how to add the Secretless Broker Sidecar to an application to retrieve secrets. This article is a sequel to the previous article, Setting Up the Conjur Kubernetes Authenticator, and uses the same environment.
Prerequisites
Before we can use the Conjur Secretless Broker, there are a few things that need to be in place:
- We need to have Conjur Open Source deployed to our Kubernetes cluster and configured to use Kubernetes Authenticator.
- We need to have an app ready that uses one of the following supported databases. We use the Pet Store Demo App for our walkthrough:
- MySQL
- PostgreSQL
- MSSQL
- The database is in our Kubernetes environment.
- We are running a supported version of Kubernetes which is version 1.7 or higher.
Configuration
Let’s define a few configurations so that their use is clear when we refer to them in the following sections:
Required information | Desc | How we refer to it | The value we use for it |
---|---|---|---|
App Name | The Kubernetes name of the application. | ${APP_NAME} | testapp-secure |
App Namespace | The Kubernetes namespace where the application pods are deployed. | ${APP_NAMESPACE} | testapp |
App Service Account Name | The Kubernetes service account is assigned to the application pods. | ${APP_SERVICE_ACCOUNT_NAME} | testapp-secure-sa |
Kubernetes Authenticator ID | The ID of the Kubernetes authenticator is configured in your Conjur Open Source instance. | ${AUTHENTICATOR_ID} | dev |
Policy branch where Kubernetes application identities are defined | The fully qualified Conjur Open Source ID of a policy branch where new Kubernetes application host identities should be added. | ${APP_POLICY_BRANCH} | app/testapp/secret |
Policy branch with database credentials | The fully qualified ID of the policy branch in Conjur Open Source that contains your database credentials. | ${APP_SECRETS_POLICY_BRANCH} | app/testapp/secret |
Layer and group with access to database credentials | The fully qualified Conjur Open Source ID of a layer or group whose members have access to the database secrets in The examples in this topic refer to a layer. if you have a group, replace all references too | ${APP_SECRETS_READER_LAYER} | app/testapp/layer |
Conjur Open Source account | The account name is configured for your Conjur Open Source deployment. | ${CONJUR_ACCOUNT} | default |
Conjur Open Source Kubernetes service account | The Kubernetes service account is associated with your cluster’s Conjur Open Source instance. | ${CONJUR_SERVICE_ACCOUNT_NAME} | conjur-cluster |
Conjur Open Source Kubernetes namespace | The Kubernetes namespace in your cluster where the Conjur Open Source instance is deployed. Secretless retrieves the database credentials from this Conjur Open Source instance. | ${CONJUR_NAMESPACE} | conjur-server |
X.509 CA Certificate Chain (optional) | The PEM encoded X.509 certificate chain for the Conjur Open Source instance deployed to your cluster. | conjur.pem | conjur-default.pem |
Conjur URL | Fully Qualified Domain name to access our Conjur instance. | ${CONJUR_APPLIANCE_URL} | https://conjur.demo.com |
Deploying our Application with Conjur
The Secretless Broker Sidecar acts as a mediator between our applications and targeted services, such as a database. It prevents direct access to the target service, which mitigates the security risk of secrets being stored directly in the application (which would render our application secretless). Instead, a request is sent through the Secretless Broker to the targeted service, but before we can use the Secretless Broker we must enroll our app in Conjur. This ensures that Conjur can identify it and add our secrets to the Conjur Vault. Enrolling an app refers to the process of adding new stuff to Conjur.
Adding Secrets to Conjur
To get our secrets into Conjur, we define a policy for our app. This policy defines the secrets required by our app. Since we want to store our database connection details, we require four variables to store the host
, port
, username
, and password
. The policy for our app is below:
cat secretless/testapp-policy.yml
- !policy
id: app/testapp
owner: !group kube_admin
annotations:
description: This policy contains the credentials to access the DB in conjur.
body:
- &variables
- !variable secret/host
- !variable secret/port
- !variable secret/username
- !variable secret/password
- !layer layer
- !permit
resource: *variables
privileges: [ read, execute ]
roles: !layer layer
Be sure to modify it for your specific needs.
To add the above policy to Conjur, execute this command:
conjur policy load root /root/secretless/testapp-policy.yml
To verify that we have added the variables, we run conjur list
.
All that we have to do now is to add value, which we do with the following commands:
conjur variable values add app/testapp/secret/password "5b3e5f75cb3cdc725fe40318"
conjur variable values add app/testapp/secret/username "test_app"
conjur variable values add app/testapp/secret/host "testapp-db.testapp.svc.cluster.local"
conjur variable values add app/testapp/secret/port "5432"
In case you have missed the previous article, and you’re wondering where these values come from, we defined them in our db.yml file. This is our PostgreSQL DB configured in our Kubernetes cluster.
apiVersion: v1
kind: Namespace
metadata:
name: testapp
---
kind: Service
apiVersion: v1
metadata:
name: testapp-db
namespace: testapp
spec:
selector:
app: testapp-db
ports:
- port: 5432
targetPort: 5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: testapp-db
labels:
app: testapp-db
namespace: testapp
spec:
replicas: 1
selector:
matchLabels:
app: testapp-db
template:
metadata:
labels:
app: testapp-db
spec:
containers:
- name: testapp-db
image: postgres:9.6
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5432
env:
- name: POSTGRES_PASSWORD
value: 5b3e5f75cb3cdc725fe40318
- name: POSTGRES_DB
value: test_app
- name: POSTGRES_USER
value: test_app
Adding the Application to the Conjur Policy
All the values that we will use have been placed in a file, secretless/env.sh. We can refer back to it during the setup.
APP_NAME=testapp-secure
APP_NAMESPACE=testapp
APP_SERVICE_ACCOUNT_NAME=testapp-secure-sa
AUTHENTICATOR_ID="dev"
APP_SECRETS_POLICY_BRANCH="app/testapp/secret"
APP_SECRETS_READER_LAYER="app/testapp/layer"
CONJUR_ACCOUNT="default"
CONJUR_APPLIANCE_URL="https://conjur-cluster-conjur-oss.conjur-server.svc.cluster.local"
CONJUR_ADMIN_AUTHN_LOGIN="admin"
CONJUR_ADMIN_API_KEY="MySecretP@ss1"
OSS_CONJUR_SERVICE_ACCOUNT_NAME="conjur-cluster"
OSS_CONJUR_NAMESPACE="conjur-server"
APP_AUTHENTICATION_CONTAINER_NAME="secretless"
We can now generate the secretless/app-policy.yml file that adds our application to Conjur.
- !policy
id: app/testapp
owner: !group kube_admin
annotations:
description: This policy contains the credentials to access the DB in conjur.
body:
- &variables
- !variable secret/host
- !variable secret/port
- !variable secret/username
- !variable secret/password
- !layer layer
- !permit
resource: *variables
privileges: [ read, execute ]
roles: !layer layer
To review the generated policy, we can use cat secretless/app-policy.yml
.
Once we’re happy with the app policy, we can load it on Conjur using this command:
conjur policy load root /root/secretless/app-policy.yml
We loaded our app on Conjur. But at this stage, it doesn’t have permission yet to access the pods. To resolve that, we must create a role with relevant access, as well as a role binding the application service account to the role.
We generate the role and save it to the conjur-authenticator-role.yml file using this command:
. ./secretless/env.sh
cat << EOL > secretless/conjur-authenticator-role.yml
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: conjur-authenticator
namespace: ${APP_NAMESPACE}
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods", "serviceaccounts"]
verbs: ["get", "list"]
- apiGroups: ["extensions"]
resources: [ "deployments", "replicasets"]
verbs: ["get", "list"]
- apiGroups: ["apps"] # needed on OpenShift 3.7+
resources: [ "deployments", "statefulsets", "replicasets"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create", "get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: conjur-authenticator-role-binding
namespace: ${APP_NAMESPACE}
subjects:
- kind: ServiceAccount
name: ${OSS_CONJUR_SERVICE_ACCOUNT_NAME}
namespace: ${OSS_CONJUR_NAMESPACE}
roleRef:
kind: Role
name: conjur-authenticator
apiGroup: rbac.authorization.k8s.io
EOL
As before, you can review it with cat secretless/conjur-authenticator-role.yml.
kubectl create -f secretless/conjur-authenticator-role.yml
. We must also store the certificate in the config map to ensure our application passes the certification validation using the following code:
. ./secretless/env.sh
kubectl \
--namespace "${APP_NAMESPACE}" \
create configmap \
conjur-cert \
--from-file=ssl-certificate="conjur-default.pem"
secretless
configuration for our app and store it in the ConfigMap
manifest. The script to generate the config is as follows:
. ./secretless/env.sh
cat << EOL > ./secretless/secretless.yml
version: "2"
services:
postgres-db:
connector: pg
listenOn: tcp://0.0.0.0:5432
credentials:
host:
from: conjur
get: ${APP_SECRETS_POLICY_BRANCH}/host
port:
from: conjur
get: ${APP_SECRETS_POLICY_BRANCH}/port
username:
from: conjur
get: ${APP_SECRETS_POLICY_BRANCH}/username
password:
from: conjur
get: ${APP_SECRETS_POLICY_BRANCH}/password
sslmode: disable
EOL
Review that the script generated the configuration correctly with cat secretless/secretless.yml file.
Once we have validated the configuration file, we can store it in the ConfigManifest
with this command:
. ./secretless/env.sh
kubectl \
--namespace "${APP_NAMESPACE}" \
create configmap \
secretless-config \
--from-file=./secretless/secretless.yml
The terminal responds with “configmap/secretless-config created.
”
The Security Admin must perform all these steps for each application deployed. It might seem like a lot of steps, but you can automate them by using a script and changing the env.sh file to the appropriate values.
The Application Developers Part
The application developer then carries out the steps that follow. There are significantly less than those of the Security Admin. This is because the idea is to keep the developer’s responsibility and access to secrets to a minimum, leaving them to focus on the actual development of the application.
The application developer also has an environment script that contains all the relevant configuration variables, but it doesn’t contain any secrets.
The contents of the script look something like this:
secretless/developer-env.sh
APP_NAME=testapp-secure
APP_NAMESPACE=testapp
APP_SERVICE_ACCOUNT_NAME=testapp-secure-sa
CONJUR_ACCOUNT="default"
CONJUR_APPLIANCE_URL="https://conjur-cluster-conjur-oss.conjur-server.svc.cluster.local"
AUTHENTICATOR_ID="dev"
All the developer must do is generate the script that deploys the app with the Secretless broker. We can do that with the following command:
. ./secretless/developer-env.sh
cat << EOL > secretless/testapp-secure.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: testapp-secure-sa
namespace: testapp
---
apiVersion: v1
kind: Service
metadata:
name: testapp-secure
namespace: testapp
spec:
type: NodePort
ports:
- port: 8080
targetPort: 8080
selector:
app: testapp-secure
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: "${APP_NAME}"
name: "${APP_NAME}"
namespace: "${APP_NAMESPACE}"
spec:
replicas: 1
selector:
matchLabels:
app: "${APP_NAME}"
template:
metadata:
labels:
app: "${APP_NAME}"
spec:
serviceAccountName: "${APP_SERVICE_ACCOUNT_NAME}"
hostAliases:
- ip: "10.105.68.126"
hostnames:
- "conjur.demo.com"
containers:
- image: cyberark/demo-app
imagePullPolicy: IfNotPresent
name: testapp-secure
ports:
- containerPort: 8080
env:
- name: DB_URL
value: postgresql://localhost:5432/postgres
- name: DB_USERNAME
value: dummy
- name: DB_PASSWORD
value: dummy
- name: DB_PLATFORM
value: postgres
- name: secretless
image: cyberark/secretless-broker:latest
imagePullPolicy: Always
args: ["-f", "/etc/secretless/secretless.yml"]
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CONJUR_APPLIANCE_URL
value: "${CONJUR_APPLIANCE_URL}"
- name: CONJUR_AUTHN_URL
value: "${CONJUR_APPLIANCE_URL}/authn-k8s/${AUTHENTICATOR_ID}"
- name: CONJUR_ACCOUNT
value: "${CONJUR_ACCOUNT}"
- name: CONJUR_AUTHN_LOGIN
value: "host/conjur/authn-k8s/${AUTHENTICATOR_ID}/apps/service-account-based-app"
- name: CONJUR_SSL_CERTIFICATE
valueFrom:
configMapKeyRef:
name: conjur-cert
key: ssl-certificate
readinessProbe:
httpGet:
path: /ready
port: 5335
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 60
volumeMounts:
- mountPath: /etc/secretless
name: config
readOnly: true
volumes:
- name: config
configMap:
name: secretless-config
defaultMode: 420
EOL
Verify the file with cat secretless/testapp-secure.yml.
We can now deploy our application with kubectl apply -f secretless/testapp-secure.yml
.
And with that step, we have completed the setup and successfully deployed our application, which is configured to use the Conjur Secretless Broker with Kubernetes.
Testing our Application with the Secretless Broker
Next, we ensure that our application is working as we intended. We can get our application endpoint from the service with the following command:
export URL=$(kubectl describe service testapp-secure --namespace=testapp |grep Endpoints | awk '{print $2}' )
We can add a random value to the demo app that we have been using by running the following command:
curl -d "{\"name\": \"$(shuf -n 1 /usr/share/dict/american-english)\"}" -H "Content-Type: application/json" $URL/pet
The above command takes a random word from the dictionary installed in the Linux system and adds it as a pet to the demo application.
We can then view the results of this addition with the command curl -s $URL/pets | jq
.
Our output looks like the line below, depending on how many times we ran the command to add a pet.
Conclusion
Do we still have exposed secrets? Let’s confirm by checking the two files to which developers have access with the command: cat secretless/testapp-secure.yml
.
There are no exposed secrets here. Let’s check the environment script.
cat secretless/developer-env.sh
There’s nothing here either. We’re completely secure and have mitigated all risks of exposing secrets to anyone that shouldn’t have access to them.
For more information about Conjur Secretless Broker, read Secretless Broker: Open Source Secret Management or have a look at the full documentation.
If you’re interested in developing expert technical content that performs, let’s have a conversation today.