Logstash Authentication with SSL certificates

If you want to have a remote logstash instance available through the internet, you need to make sure only allowed clients are able to connect. Since the lumberjack protocol is not HTTP based, you cannot fall back to proxy through an nginx with http basic auth and SSL configured. Instead you need to setup your own authentification based on SSL certificates. The official logstash documentation doesn't go into details about this and it took me some time to figure out what I exactly need to make that work. This is why I provide a step-by-step guide here. The openssl commands and configs are mostly copied from this excellent guide.

First check your openssl version. I ran into some problems using flags other openssl versions didn't understand. Everything in this guide was generated with that version.

$ openssl version
LibreSSL 2.2.7

For logstash and filebeats, I used the version 6.3.1.

Also I never made it work with curl to check if the logstash server is working correctly but instead I tested successfully with filebeats. If you run into errors like

routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
routines:OPENSSL_internal:WRONG_VERSION_NUMBER
routines:OPENSSL_internal:TLSV1_ALERT_UNKNOWN_CA
routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
routines:SSL3_READ_BYTES:sslv3 alert handshake failure

or similar, this guide is for you.

Basic workflow

Basically this is how it is working: You need to create a common root CA certificate, which you then you to both sign the certificates for logstash and filebeats (or any other beat). With this logstash can verify if the connection comes from some known client.

Generate CA cert

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt

Generate logstash cert

Create a file with the following content and save it as 1ogstash.conf.

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
countryName                     = XX
stateOrProvinceName             = XXXXXX
localityName                    = XXXXXX
postalCode                      = XXXXXX
organizationName                = XXXXXX
organizationalUnitName          = XXXXXX
commonName                      = XXXXXX
emailAddress                    = XXXXXX

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = DOMAIN_1
DNS.2 = DOMAIN_2
DNS.3 = DOMAIN_3
DNS.4 = DOMAIN_4

Instead of DOMAIN_1 you need to add your actual domain where the logstash will be reachable later on. If you want to use this cert on multiple machines, add these domains to, otherwise just delete the DNS.x entries you don't need.

Now run

openssl genrsa -out logstash.key 2048
openssl req -sha512 -new -key logstash.key -out logstash.csr -config logstash.conf

Now get the serial of the CA and save it in a file. With

openssl x509 -in ca.crt -text -noout -serial

you will see something like serial=AEE7043158EFBA8F in the last line. Put the id into a file with

echo "AEE7043158EFBA8F" > serial

Now you can use that to create and sign your logstash cert with it

openssl x509 -days 3650 -req -sha512 -in logstash.csr -CAserial serial -CA ca.crt -CAkey ca.key -out logstash.crt -extensions v3_req -extfile logstash.conf
mv logstash.key logstash.key.pem && openssl pkcs8 -in logstash.key.pem -topk8 -nocrypt -out logstash.key

Generate beats cert

Now that we are done with the logstash side, we need to create another certificate, which can be used by beats, for example filebeats.

Create another file called beat.conf with the following content:

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
countryName                     = XX
stateOrProvinceName             = XXXXXX
localityName                    = XXXXXX
postalCode                      = XXXXXX
organizationName                = XXXXXX
organizationalUnitName          = XXXXXX
commonName                      = XXXXXX
emailAddress                    = XXXXXX

[ usr_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, server
nsComment = "OpenSSL FileBeat Server / Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation
extendedKeyUsage = serverAuth, clientAuth

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth

Here you don't need to change anything if you don't really want to. Now create the key and sign it again.

openssl genrsa -out beat.key 2048
openssl req -sha512 -new -key beat.key -out beat.csr -config beat.conf
openssl x509 -days 3650 -req -sha512 -in beat.csr -CAserial serial -CA ca.crt -CAkey ca.key -out beat.crt -extensions v3_req -extensions usr_cert  -extfile beat.conf

Configure logstash pipeline

This is an example pipeline, which shows you how logstash needs to be setup to use all the certs.

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
    ssl_key => "/etc/pki/tls/private/logstash.key"
    ssl_verify_mode => "force_peer"
    ssl_certificate_authorities => ["/etc/ca.crt"]
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Start your logstash and make sure it is available under the same domain specified in the cert.

Configure beats

Since curl didn't work for me to verify my logstash, I used filebeats for it. Save this in a file called filebeat.yml.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /test.log
output.logstash:
  hosts: ["logstash.example.com:5044"]
  ssl:
    certificate_authorities: ["/etc/ca.crt"]
    certificate: "/etc/beat.crt"
    key: "/etc/beat.key"

Change the domain to your logstash domain and then start filebeat. I used it on docker with

docker run --rm -v $(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml -v $(pwd)/ca.crt:/etc/ca.crt -v $(pwd)/beat.crt:/etc/beat.crt -v $(pwd)/beat.key:/etc/beat.key -v $(pwd)/test.log:/test.log docker.elastic.co/beats/filebeat:6.3.1

As soon as I do echo "hello world" >> test.log from another terminal, I can see logstash getting messages like

{
    "@timestamp" => 2018-07-08T13:35:11.263Z,
        "source" => "/test.log",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "beat" => {
            "name" => "410db3eac54f",
         "version" => "6.3.1",
        "hostname" => "410db3eac54f"
    },
       "message" => "hello world",
      "@version" => "1",
    "prospector" => {
        "type" => "log"
    },
         "input" => {
        "type" => "log"
    },
        "offset" => 12,
          "host" => {
        "name" => "410db3eac54f"
    }
}

Hooray!