• Beau Steward

Kibana, Load Balancer, and Too Many Redirects

I'm building a new ELK stack for work and I ran into a problem enabling security. Our old one has a hacked together concept of security and the new stack, with upgraded Elastic software, will be using the built-in features provided through our licensing. However, I ran into an interesting problem configuring Kibana and a software load balancer.


Logging in resulted in a too many redirects error and being dumped back to the login page.


I had initially started with haproxy as my load balancer. Thinking there might be a wonky issue with haproxy, I switched to nginx, but the issue persisted.


If you search the internet on this problem, you can find a fairly wide array of solutions, none of which helped me. They either didn't apply to my situation (I wasn't using rewrites) or just didn't work. So, here is what worked for me.


Now, I mentioned I'm using a load balancer, not specifically using the terminology around a reverse proxy. This means I have multiple Kibana deployments behind the load balancer. As it turns out, the issue was one Kibana not being able to read the cookie created by another Kibana and giving up.


Why was this happening? Kibana created the cookie, so why can't another Kibana read it? Well, that's because it's encrypted. And the 2 Kibanas didn't have the same key. Because I didn't define an encryption key for the cookie, each Kibana was, separately, generating a random key on startup.


To solve this problem, you use the following configuration parameter:


xpack.security.encryptionKey

This parameter is documented, but I had to find it by reading about parameters rather than researching the problem. The value of this key should be 32 characters or larger. I used a randomly generated UUID.


I run this in a docker swarm, so the parameter, ultimately, looks like this:


environment:
  - ELASTICSEARCH_HOSTS=["https://server1:9200","https://server2:9200","https://server3:9200"]
  - SERVER_HOST=0.0.0.0
  - SERVER_NAME={{.Node.Hostname}}
  - XPACK_SECURITY_ENABLED=true
  - XPACK_SECURITY_ENCRYPTIONKEY="9f22a2cb-5e80-4545-bc05-9ba041fa2b34"
  - ELASTICSEARCH_USERNAME=kibana
  - ELASTICSEARCH_PASSWORD=password
  - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/config/certs/ca/ca.crt

Simple enough fix and makes sense. Giving both Kibanas the same encryption key means either can read the cookie.

0 views0 comments

Recent Posts

See All

Drop+THX Panda Still Has Problems

I just trashed a previous article I wrote but hadn't published. I was prepared to write about the Panda being a failure because, once again, firmware updates to solve its problems were delayed. But...