r/elasticsearch • • 4d ago

Best Practices for Email Alerts and Monitoring in Elastic Stack SIEM

9 Upvotes

Hi,

I have an Elastic Stack SIEM cluster (Kibana, Elasticsearch, Fleet, Filebeat, and Logstash), and I'm wondering if anyone has recommendations for setting up email alerts when something goes wrong with the system.

How do you handle monitoring and alerting in your environment? What would you consider best practices for this kind of setup?

Thanks in advance!


r/elasticsearch • • 6d ago

Truth about elastic job market?

5 Upvotes

For reference, I am an elastic certified engineer with couple YOE in deployment and management of elastic stack for multiple organizations for search, observability and security use case.

I am finding it extremely difficult to land a role related to elastic. With elastic's partnership with OpenAI, I thought this will open up new opportunities but no. I rarely find any role and even after applying there is no response back.

What should I do? What's your opinion on current job market especially related to elastic?


r/elasticsearch • • 6d ago

Truth about elastic job market?

1 Upvotes

For reference, I am an Elastic Certified Engineer with couple of years of experience in deployment and management of Elastic clusters for search, observability and security use cases.

I loved working with elastic so I went ahead and took the exam and eventually passed it. Since then I've been looking to land a full time role related to elastic but having no luck.

So, I wanted to ask you guys about your opinions on current job market especially related to elastic.

I thought elastic's partnership with openAI will make some difference but no. Roles related to elastic are very rare to find in my opinion.

So again, what's your take on this?


r/elasticsearch • • 13d ago

AI Agents constantly make terrible mistakes

2 Upvotes

I'm messing about with the Agent builder for my company. Mainly just testing the waters and see what its capable of. At elasticon 2025 and 2026 the agents were presented as this sort of supertool.

My experience with it has been very frustrating. It keeps making the most terrible errors in simple querys. I have given it A LOT of context now and it sure does help. But more than half the time it just gives back complete bogus.

Im using Claude 4.6 sonnet.

Was wondering if people are experiencing the same thing or if im just doing it completely wrong. Would love some tips!


r/elasticsearch • • 15d ago

How do you organize Kibana Spaces, indices, and telemetry across multiple domains to monitor from a service perspective rather than a component perspective?

Thumbnail reddit.com
2 Upvotes

r/elasticsearch • • 15d ago

New feature "Streams" issue

4 Upvotes

Hi,

I’ve discovered a new feature called Streams, and I’m wondering how I can resolve an issue with a log index that has a data quality status of Poor.

I’ve attached a screenshot showing what it looks like.

Could you please advise me on how to fix this?

Thanks for help!


r/elasticsearch • • 19d ago

Testing Elastic SIEM detections after pipeline changes

2 Upvotes

Windows process events are still in Discover after an ingest pipeline update, but our PowerShell EQL rule stopped generating alerts. `process.command_line` is present, though `event.category` and a couple of other ECS fields changed. I can find the activity with a manual query, but the scheduled rule shows successful execution with zero alerts. Atomic Red Team gave us test activity, so at least we know what should have shown up. Has anyone tried cymulate for repeatable detection tests with Elastic? Not sure how much it would help here if the issue is ECS normalization rather than the rule itself. Now I'm wondering what else we broke with that pipeline update. How are you regression testing detections after changes like this?

Open to any suggestions and what you r using for SIEM detections.


r/elasticsearch • • 20d ago

Certified Engineer Exam – Completed but got result as failed inspite of Trueability declared as Uncounted attempt...

Thumbnail reddit.com
5 Upvotes

An update for everyone ...

I got the reattempt from elastic for 9.x version. Scary for me as the syallabus changed , the documentation was hell different but I managed to study for that as well.

Thanks to Elastic team for giving attention to this case.

I prepared and took exam on 16 oct , got the result yesterday and it's passed!

A piece of advise those who r preparing for ECE certification ... Please be well versed of official doc ... Nothing is beyond that ...that's enough to crack the exam... I love to do self study ... I really read every topic line by line .... It's helps u to go through the edge cases , situations ....

Thanks !!


r/elasticsearch • • 20d ago

Managed Elasticsearch for small workloads without the VCUs

0 Upvotes

Disclosure up front: I'm one of the people building this, so take it with the right amount of salt.

Many here know me - my team and I have been running Elasticsearch for a lot of different orgs over the years. One complaint keeps coming up: small and spiky workloads cost far more than they should. Think a site search with a few GB of data, a RAG side project, an internal tool. Elastic Cloud Hosted starts around $99/mo for a cluster you have to size. Elastic Cloud Serverless is a big improvement, but search projects still keep a baseline running billed in VCU-hours. Elastic's own pricing examples put 2 GB at roughly $24-27/mo and 20 GB at $190-210/mo, and there's no published way to estimate VCUs per query. OpenSearch Serverless is worse at the low end, because the OCU floor alone is around $350/mo.

So we built WarpSearch (https://warpsearch.io/): serverless search that bills per GB stored, per GB written and per request. There's no compute baseline. An idle collection costs only its storage, and it scales to zero after some idle.

It's not Elasticsearch under the hood. It's our own cloud-native C++ engine (full-text, vectors, hybrid, facets). It serves the Elasticsearch REST API at a compat endpoint, so the official clients, Logstash, Beats, Fluent Bit, Vector and Grafana's ES datasource connect with just a URL change:

Try it out - add our MCP and tell your agent to index data for you: https://warpsearch.io/docs/agent-quickstart

Numbers: median keyword search of 1-6 ms (2.1 ms on 2.8M docs), and kNN at 6.2 ms median with 0.964 recall@10 on 10M vectors. Writes are searchable in seconds, or immediately with commit=true.

Pricing: a permanent free tier then $10/mo minimum. There's a line-by-line cost comparison against ES Serverless, Elastic Cloud, OpenSearch Serverless and a couple of vector DBs here: https://warpsearch.io/compare/elasticsearch-serverless

It's not just for small workloads - since it's cloud native, we are able to do quite a bunch of nice things with it when it coems to scale and cold storage. But it's still WIP.

I'd really like feedback from people who run ES in production:

- Which missing APIs would be a dealbreaker for you?

- For small workloads, is idle cost actually your pain point, or is it ops/upgrades?

Happy to answer anything technical in the comments.


r/elasticsearch • • 21d ago

Looking for Advice on Improving My Elastic Security Environment

0 Upvotes

A Few Questions About Elastic Security / SIEM

Hi everyone,

I have a few questions about Elastic Security and would appreciate hearing about your experiences, recommendations, and best practices.

1. Threat Intelligence

Which free threat intelligence sources would you recommend besides Abuse.ch?

As a general question, do you feed threat intelligence data directly into your SIEM, or do you use API integrations to query external threat intelligence sources?

2. Reporting

Reporting is one area where I feel Elastic could be improved. I know it is possible to take screenshots or export reports to PDF, but scheduled reports would be a really useful feature. How do you handle reporting in your environment?

3. SOC Dashboards

In my opinion, Elastic's default dashboards could be better. Has anyone built custom SOC dashboards that they would be willing to share? I'd love to see some examples or get inspiration for building my own.

4. Vulnerability Management

Has anyone found a good way to integrate vulnerability management with Elastic Defend? So far, I haven't found a solution that fully meets my needs. I'd be interested in hearing how you approach this.

5. Elasticsearch Health and Configuration Auditing

Is there a health-check tool or script that can help verify whether an Elasticsearch environment is configured correctly, rather than simply checking whether the cluster is healthy?

For example, I'd like to identify issues such as incorrectly parsed logs, inefficient mappings or schemas, unnecessary indexing overhead, and misconfigured ILM policies. Ideally, I want to validate that my indices, data streams, mappings, and lifecycle policies are all configured correctly and efficiently.

What tools or approaches do you use for this?

6. Entity Analytics

Could someone explain what Entity Analytics actually does and how it can be used in practice? Are you using it in your environment? If so, I'd appreciate a real-world example of how it helps with threat detection or investigations.

7. Sigma Rules and Alert Noise

After installing and enabling Sigma rules, I started seeing a large number of alerts in my environment.

Is it common to run Sigma rules in production, and do you use them in your own environments? How do you handle false positives, tune the rules, and decide which ones are worth keeping enabled?

I'd be interested in hearing your recommendations for deploying and maintaining Sigma rules in a production environment without generating too much alert noise.

8. Machine Learning Jobs

How can I verify that my Elastic Security machine learning jobs are configured correctly and running as expected? Are there any recommended checks or best practices for monitoring their status, detecting failures, and validating that they are producing useful results?

Thanks in advance for sharing your experience and recommendations!


r/elasticsearch • • 23d ago

What’s your rule of thumb for OpenSearch shard sizes before things break?

0 Upvotes

We’re currently ingesting about 80GB of application logs a day into an OpenSearch cluster, and our shard count is getting out of hand.

Right now, we create daily indices with 3 primary shards each. The problem is that after 30 days, we have hundreds of tiny shards (some are only 10GB-15GB), and our cluster is starting to show JVM memory pressure even when search traffic is quiet.

The official docs recommend aiming for 30GB–50GB per shard, but I'm curious what people actually aim for in the real world. Do you rollover by size rather than time, or do you run shrink/merge tasks on older indices?


r/elasticsearch • • 27d ago

Which methods have proven effective for improving Elasticsearch search relevance?

2 Upvotes

Getting search results quickly is one challenge; ensuring that the useful results appear first is another challenge. As the dataset grows, things like analyzers, mappings, tokenization, and query design can noticeably affect Elasticsearch search relevance.
For developers who use Elasticsearch in production, which techniques have made the difference in Elasticsearch search relevance?
I would especially like to hear lessons from projects where the default query did not produce the Elasticsearch search relevance users expected. What changes did you make? How did you measure whether the Elasticsearch changes improved the user experience?


r/elasticsearch • • 28d ago

Using Elasticsearch with AI and document search

0 Upvotes

I have been looking into ways to bring Elasticsearch with local AI for projects that need to search through big amounts of documents while still having control, over the data.

Elasticsearch offers text and vector search features, which makes it appealing for retrieval-augmented generation (RAG) processes. I have also been checking out LM-Kit for AI and document processing. I am curious about how these parts can come together in a real-world application.

For people who have created systems, how do you connect Elasticsearch with an AI environment? I would especially like to know what works well for retrieving results, indexing documents, and making sure search results stay relevant as the data gets bigger.


r/elasticsearch • • Oct 19 '26

Simplest way to run a cross-region read replica of a small ECK-managed Elasticsearch cluster

1 Upvotes

Hey guys,

looking for real-world advice before I commit to an approach.

Current setup:

- self-managed Elasticsearch (9.4.x), deployed via ECK on Kubernetes

- 3-node cluster, 8 vCPU / 16GB RAM per node, 150GB PVC per node

- free/basic license

- single region today, its an internal backend service

I want to have a read copy of this cluster running in a second region, mainly for cutting the latency for callers that are geo located farther from the primary region. This should be a prototype, so I would prefer reliability and easy setup over complex solutions. The data lag of a few minutes is completely acceptable, i dont need anything close to realtime.

What I have already ruled out:

- cross-cluster replication: needs a higher tier of license, and the cluster is small, so it feels like overkill

- app dual-write, I dont want to put my engineers in charge of consistency, failure backfills, and so on

What Im thinking about:

- scheduled snapshot/restore (I already have the s3 plugin installed). restore to a new index each cycle + alias swap. The downside i realized, each restore cycle is a full copy since the target index starts empty, so I believe it makes sense at a slower frequency (~30 minutes), not continuous

- the cronjob, doing filtered remote _reindex, using "last-modified" as a cursor, seems like quite a simple pipeline: 1. read the saved cursor 2. reindex only docs after it 3. save the new cursor (small index). The gotcha I see, the deletes wont propagate, the last-modified query on reindex only sees the updated or new docs.

The delete operation that i need to handle, is the reason i think snapshot/restore is a simpler and more reliable choice for a prototype.

Appreciate help, trying to avoid reinventing something that has its own failure modes. Happy to share details if that helps you to give better advice.


r/elasticsearch • • Oct 17 '26

Elasticsearch for RAG

7 Upvotes

I am new in Elasticsearch. In company we are planning to index about 10TB data for RAG application (keyword + vectors) locally. Before buying any hardware I was learning about appropriate spec for our use. I have read Elasticsearch docs where there is only about shard size 10-50GB and heap 31GB max. But no data about each Node size.
Our expected parallel user request is 20.
I read a post that 1-2TB data with two node parallel requests 250 takes 5-7s on light load and about 20s on heavy loads.
So would it be proper to have 5Node + one Master node for 10TB of data.(before index,vector and token)


r/elasticsearch • • Oct 15 '26

Long term retention | Historical dashboard data for archive purposes

4 Upvotes

I want to maintain historical dashboard data for archive purposes, beyond our data source's (Prometheus) retention period.

Mimir and Thanos feel like overkill for this. Add infrastructure, storage costs, but all i need is to look back at historical data.

Any lighter approaches that is working for you?


r/elasticsearch • • Oct 14 '26

New to ElasticSearch - PDF keyword search

0 Upvotes

I’m new to elastic search and curious if this fits my use case, or if it’s overkill.

I have a low volume of PDF files (<500), and low number of queries per month (<5000). I simply need to find the PDF files that have matching words of the user query.

My main question is do I have to chunk and upload the PDFs as seperate documents, or would I upload each PDF entirely? Since I’m not using embedding I figure it could handle the entire document. Most PDFs are modest in size, usually 5-20 pages.

If I’ve missed a resource to learn more, please feel free to point me in that direction.


r/elasticsearch • • Oct 14 '26

Elastic101 – Best Practice #005: Use numeric suffix for ILM rollover

Thumbnail
gallery
0 Upvotes

Almost 10 years of Elasticsearch in production, and this is still one of the most common index lifecycle management (ILM) mistakes I see.

The rule: for a regular (non data stream) index, the ILM rollover action needs the index name to end with - and digits (^.*-\d+$), like logs-000001. Rollover creates the next index by incrementing that number. It also needs index.lifecycle.rollover_alias set, pointing to the index.

logs, logs-prod and logs-2026.10.12 all fail this check. Attaching a rollover policy to daily date-named indices is a typical trigger.

Miss it and the index lands in the ERROR step (illegal_argument_exception: index name [...] does not match pattern '^.*-\d+$'). No later phase runs after that, so the index stays in hot and is never deleted. It's a silent disk leak.

Want a date in the name? Use date math. The name still ends with digits, so rollover works:

PUT %3Ctest-logs-%7Bnow%2Fd%7D-000001%3E
{
  "settings": {
    "index.lifecycle.name": "my-policy",
    "index.lifecycle.rollover_alias": "test-logs"
  },
  "aliases": { "test-logs": { "is_write_index": true } }
}

This creates the index test-logs-2026.10.14-000001. Read and write through the test-logs alias. The date is the creation date, not the data date.

Pro tip: find stuck indices with GET */_ilm/explain?only_errors=true. ILM retries failed steps on its own, but it fails again until the cause is fixed. POST <index>/_ilm/retry runs the step immediately after you fix it. You can't rename an index, so for existing ones reindex into a numeric name. For new setups, use data streams: no naming pattern, no rollover alias to maintain.

Note: data streams are great for time-based data, but they're append-only. You can't send update or delete requests for existing documents directly to the stream (update by query, delete by query and requests to the backing index still work). If you update documents frequently, an alias with a write index, as above, is the better fit.

Previous Elastic101 best practices:

  1. #001 – HTTP traffic

  2. #002 – shard size

  3. #003 – bulk indexing

  4. #004 – primary shard count

If you'd like me to continue this series, an upvote would be appreciated. 🙂


r/elasticsearch • • Oct 13 '26

Plug in a native C++ search engine and run Elasticsearch queries faster

10 Upvotes

Hi everyone, I’m on the team at Vesiro, and today we’re announcing the beta of VesiroSearch.

VesiroSearch is a purpose-built native C++ search engine that accelerates Elasticsearch query performance by up to 2x. It installs as a standard plugin, which only takes about a minute per node.

We built it to help teams get more search capacity from the infrastructure they use to run Elasticsearch. For workloads where query processing drives capacity needs, that can mean serving the same traffic on fewer or smaller servers, reducing infrastructure costs while meeting response-time requirements.

VesiroSearch doesn’t change the query DSL in Elasticsearch, which means that you do not have to change your queries or mappings to run it. No other end points are affected either, and VesiroSearch works with your current indices, so your setup remains the same.The beta lets you evaluate performance with your own data and queries. You can enable or bypass VesiroSearch for individual requests to compare results in the same test environment.

We would love to get feedback from teams that run Elasticsearch extensively. Learning which workloads that benefit from our acceleration, and which workloads we should add in our product roadmap would be super helpful. Also, if you have any feedback on compatibility or what would make evaluation easier, that would be greatly appreciated as well.

The documentation covers compatibility and installation. 

You can read more and request beta access here: www.vesiro.com


r/elasticsearch • • Oct 10 '26

Rebuilding my Elastic stack with Hermes Agent + GPT-5.5

Thumbnail reddit.com
5 Upvotes

r/elasticsearch • • Oct 02 '26

Show & Tell I made ELK a one-click deploy on Railway (private networking, auth, retention, live demo)

11 Upvotes

Every time I needed centralized logs for a side project, I'd lose an evening wiring Elasticsearch, Logstash and Kibana together: passwords, TLS, which port is public, why Kibana can't reach ES. So I turned it into a one-click Railway template and tested it the boring way.

What you get

  • 3 separate services, grouped: Elasticsearch, Logstash, Kibana

  • Only Kibana is public. ES and Logstash talk over the private network

  • Passwords generated on deploy, Logstash intake behind basic auth (no-auth gets a 401)

  • A restricted writer user for Logstash, so your app never holds the admin password

  • Persistent volumes + log retention configured, so it doesn't eat your disk

  • Copy-paste senders for Node, Python and Go (no SDK, plain HTTP)

What I actually tested on fresh deploys: sent logs, searched them in Discover, restarted and redeployed every service, checked nothing was lost.

Cost, measured not guessed: about $27-34/month of compute depending on load.

Try it before deploying, read-only live demo with logs flowing in real time: https://kibana-production-43ee.up.railway.app (login: demo / PublicDemo2026!)

Single node, so it's for small/medium workloads, not a 10TB cluster. Feedback on the pipeline config and retention defaults very welcome.


r/elasticsearch • • Sep 28 '26

Discussion How do you prove an Elasticsearch node is safe to retire after shard migration finishes?

0 Upvotes

A green cluster and zero shards on a node are necessary before retirement, but they do not prove that clients, ingest pipelines, snapshots, monitoring, or discovery settings have stopped depending on its address. A coordinating-only workload or a stale seed-host list can survive a shard drain and fail only after the process disappears.

What belongs in the retirement gate? I am considering allocation exclusion, waiting for relocation and recovery activity to finish, checking cluster state for zero shard ownership, and then watching connection logs and network traffic to the old address through more than the longest batch interval. Client sniffing, seed hosts, remote-cluster settings, snapshot repositories, exporters, and alerting targets would be inventoried separately.

Is there a dependable way to attribute remaining connections to specific clients before shutdown? What final test do you use to distinguish a truly unused node from one that is still serving coordination or discovery traffic, and which checks should be repeated after its voting configuration and persistent settings are cleaned up?


r/elasticsearch • • Sep 25 '26

Discussion Optimal elsaticsearch setup and how it impacts search?

2 Upvotes

Context: Hi all, I'm trying to find out how our current elasticsearch cluster is set up and how can it be optimally expanded to index now types of documents. I work for an archival collection and we have rdf metadata describing our doucments. So we generally need to translate relevant metadata fields into json documents to be indexed.
At the moment we have indexed a newspaper collection in ES with the current set up: one node, ~40 indices (one for each publication) each index 1 shard and some thousands documents in an index. Each document corresponds to a newspaper page and has a full_text field with the searchable ocr text + other specific fields such as date/title/spatial coverage for precise clustering. These indices are grouped under an alias.

Question 1: As I see best practices are shard sizes between 10GB and 50GB, Keep the number of documents on each shard below 200 million,
I wonder if all this collection could be one single index, with maybe 2 shards (we add 1 node so that we have 1 shard primary and one replica on the second node) with an extra field in each document to define which publication (the name of the newspaper, which now is simply the index name).

Which one of these two set ups works best in performance and how could our search functionalities potentially be impacted?

Question 2: We now have to add a new collection with other types of documents and I am wondering several things:

2a. My colleague wants to have always the same documents structure in different indices, suggesting to make it as general as possible for different types of context, aggregating all the "text" fields in our metadata into a full_text field, instead of having for instance, three fields "title", "label", "author". Doing this, I understand we could not do aggregations on specific fields, but are there any other shortcomings? Is a single full_text field going to be more efficient to index and query?

2b. What is the best way to go with adding new material? keep adding indices to the same ES instance (maybe adding more shards and nodes) and aggregate them under an alias? or Add new ES instances for new material? What does it change and how does it impact search?

2c. If we look at how we want to set up our UI search app, we ideally want a first text search where people can filter on type of collection to look into or not. If they don't, they will get results across different types of materials (so querying ALL indices), can this be done if indices have different document structures?

Thank you for reading all of this and apologies if it was long. ES feels like a steep learning curve for me and I really did not want to ask AI :)