wasting 100s of hours on setting up elasticsearch and kibana.
One of the most frustrating realizations I’ve had is that not every version of Elasticsearch works seamlessly with every version of Kibana—especially when running both in Docker containers.
Helm: Not Always the Solution
If you've tried using Helm charts to set up Elasticsearch, you’ll know it’s resource-intensive. Unless you have a beast of a machine, Helm can be overkill. Moreover, customizing your Elasticsearch setup via Helm isn’t exactly beginner-friendly.
Minikube: A Manual Approach
Minikube is another option, but preparing the manifest for both Elasticsearch and Kibana yourself can be a tedious process. AI tools for manifest generation weren’t helpful for me either. The constant pod creation and failure cycles were exhausting.
The Project That Got Me Hooked
I was inspired by a project (link) that involved analyzing Gmail data and extracting insights. The idea of visualizing this data in Kibana instead of querying via curl was appealing, but connecting Kibana to Elasticsearch turned out to be quite a challenge.
The Versioning Nightmare
Initially, I used the Elasticsearch version specified in the GitHub repo and spun up a Kibana container—only to be asked for an enrollment token that never appeared when I launched the Elasticsearch container. I wasted hours searching logs and Googling, only to find out that older Elasticsearch versions don’t generate this token.
When I switched to a newer version of Elasticsearch, I finally received the token, but Kibana still wouldn’t connect. After some digging, I realized both containers needed to be on the same Docker network, and that fixed the connection issue. However, the newer version of Elasticsearch used HTTPS, requiring additional code changes for authentication.
Schema Woes
After addressing the authentication, I hit another roadblock—schema incompatibility between the old and new Elasticsearch versions. I had no experience with index creation or bulk indexing, and the schema changes were overwhelming. At this point, I was ready to throw in the towel.
The YouTube Solution
But then I had an idea: Why not watch some tutorials on YouTube? I found a random guide on setting up Elasticsearch and Kibana, and miraculously, it worked on the first try. After all that frustration, I finally had the system up and running. Ironically, by the time it worked, I was too exhausted to go through my Gmail data.
Key Takeaway
The lesson? Persistence pays off, and sometimes, a simple YouTube video can save you hours of headaches. While reading documentation is important, knowing when to pivot and learn from others can be just as valuable.