Batten down the Etcd hatches

When exploring new technologies we used to tend to take on a laissez-faire attitude. Our time and attention was typically involved with getting complex systems up and running. However, even in these early stages, we need to keep an eye toward securing the solution. Our DevOps focus evolves into a new, more holistic DevSecOps approach. With DevSecOps we layer security into our application development end-to-end.

This means not just building an MVP, but building a secure, reliable MVP which can withstand the myriad of external threats. In the case of our earlier etcd implementation, it’s time to harden our solution by adding TLS between our clients and our distributed cluster.

When last we left our etcd implementation, we created a (3) node distributed cluster and created sample Key/Value pairs. Lets get started with the creation of secure TLS connections for our clients.

To create our cert we could use tools like openssl or cfssl, but since we’re working with Go we’ll create the cert using idiomatic go. Though, we’ll still use legacy tools to give you a flavor for what’s going on behind the scenes.

First we’ll need to install the generate cert utility if you haven’t already done so. With generate cert built and installed let’s create a self signed cert for out 3-node cluster. Choose a folder and run the generate cert utility providing the IP address, hostname and flag to self sign the certificate.

# generate self signed certificate for all 3-nodes in our cluster
generate_cert --host \
   viper,192.168.1.167,cobra,192.168.1.184,sidewinder,192.168.1.182 --ca

Generate cert will create cert.pem and key.pem. If you have openssl installed you can verify the issuer is the same as the subject (self signed) like this.

openssl x509 -in cert.pem -inform PEM -noout -subject -issuer

If you’re running on windows the Git bash shell will let you use openssl

If you have java installed you can use keytool to list the certificate info.

# use java keytool to list the cert
keytool -printcert -file cert.pem

With the cert created copy it to a location you will reference it from on node1.

Here’s the original run script we used for Node1:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	 --initial-advertise-peer-urls http://${THIS_IP}:2380 \
	 --listen-peer-urls http://${THIS_IP}:2380 \
	 --advertise-client-urls http://${THIS_IP}:2379 \
	 --listen-client-urls http://${THIS_IP}:2379 \
	 --initial-cluster ${CLUSTER} \
	 --initial-cluster-state ${CLUSTER_STATE} \
	 --initial-cluster-token ${TOKEN}

We’re going to modify the script above to change our client references to use secure TLS socket connections to communicate with node1.

# changes to the etcd run script for TLS

# Added env var references to location of cert and key
CERT_FILE=${HOME}/src/certs/167/cert.pem
CERT_KEY=${HOME}/src/certs/167/key.pem

# note: advertise-client-urls and listen-client-urls changed
#       to use https
etcd --data-dir=data.etcd --name ${THIS_NAME} \
        --cert-file ${CERT_FILE} --key-file ${CERT_KEY} \
        --initial-advertise-peer-urls http://${THIS_IP}:2380 \
        --listen-peer-urls http://${THIS_IP}:2380 \
        --advertise-client-urls https://${THIS_IP}:2379 \
        --listen-client-urls https://${THIS_IP}:2379 \
        --initial-cluster ${CLUSTER} \
        --initial-cluster-state ${CLUSTER_STATE} \
        --initial-cluster-token ${TOKEN}

Our changes to the script above now include references to the cert and key file we created with generate cert as well as changes from http to https for the client.

With these changes in place you can stop and start the etcd on node1 and we should be able to connect from our client using https. I’ll be using httpie, a curl like utility we described in our post getting started with etcd.

# here's a curl example:
#  curl --cacert cert.pem -L https://viper:2379/v3/kv/range \
#    -X POST -d '{"key": "Zm9v"}'

# connect to etcd using https and httpie client
http --verify=cert.pem POST https://viper:2379/v3/kv/range key=Zm9V

HTTP/1.1 200 OK
Access-Control-Allow-Headers: accept, content-type, authorization
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE
Access-Control-Allow-Origin: *
Content-Length: 114
Content-Type: application/json
Date: Sun, 26 Jan 2020 18:33:31 GMT
Grpc-Metadata-Access-Control-Allow-Headers: accept, content-type, authorization
Grpc-Metadata-Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE
Grpc-Metadata-Access-Control-Allow-Origin: *
Grpc-Metadata-Content-Type: application/grpc
Grpc-Metadata-Trailer: Grpc-Status
Grpc-Metadata-Trailer: Grpc-Message
Grpc-Metadata-Trailer: Grpc-Status-Details-Bin

{
    "header": {
        "cluster_id": "8542102317057920171",
        "member_id": "16840390794084916863",
        "raft_term": "95",
        "revision": "5"
    }
}

If you try the same request using http you should now get a connection error.

http POST http://viper:2379/v3/kv/range key=Zm9V

http: error: ConnectionError: ('Connection aborted.', ConnectionResetError(10054
, 'An existing connection was forcibly closed by the remote host', None, 10054,
None)) while doing POST request to URL: http://viper:2379/v3/kv/range

However, since you haven’t configured node2 and node3 yet for https you should still be able to connect with http.

http POST http://cobra:2379/v3/kv/range key=Zm9V

HTTP/1.1 200 OK
Access-Control-Allow-Headers: accept, content-type, authorization
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE
Access-Control-Allow-Origin: *
Content-Length: 113
Content-Type: application/json
Date: Sun, 26 Jan 2020 18:37:55 GMT
Grpc-Metadata-Content-Type: application/grpc

{
    "header": {
        "cluster_id": "8542102317057920171",
        "member_id": "2181031141463563501",
        "raft_term": "95",
        "revision": "5"
    }
}

You should go ahead and repeat the steps above to secure the client interaction on node2 and node3. At this point you should feel confident in configuring the connections between the cluster members to use TLS. Etcd will also permit you to secure 2-way TLS connections by providing client certs to the cluster members.

Our DevSecOps hardening is moving along well and our cluster security is tightening up.

I hope you enjoyed this post and it’s helped to make your own etcd configuration more secure!