Understand Lucene to Understand ElasticSearch

The purpose of this article is to promote a better understanding of how things work within ElasticSearch. If you are reading this article, you probably already know how ElasticSearch works, if that is not true, you can learn a little more by reading the following article:

Lucene.

Lucene is the base of ElasticSearch, but you don’t interract directly with him, as you drive your car, but you don’t ask direct to your engine to start. But what if your car break, don’t you think is a good idea to know how your engine works?

Lucene Indexing

You have a big a mount of files, and you need to find a specificy file, wich contais a certain word, how to be quickly to do this? how to be scalable? Here’s where indexing comes in: to search large amounts of text quickly, you must first index that text and convert it into a format that will let you search it rapidly. This conversion process is called indexing, and its output is called an index.

Lucene Index.

The index is composed of one or more segments, and each segment is composed of several indexes, confusing, right? When it is created, it is separated into smaller segments, or you can see it as sub-indices, where each index is not completely independent.

Lucene Segments.

Segments are immutable .Each segment contains one or more Lucene Documents

At some point, on your journey through ES, you have been in the situation of having to delete a document. And apparently that was no problem, but what is going on behind the scenes?

When you delete a document, it is only “marked” as deleted and a new version of the document is added to the segment. Its real execution is only carried out from time to time when a “joining” of larger segments occurs.

In the meantime, documents continue to take up disk space.

Lucene Merge.

Over time the index will accumulate many segments. Periodically, segments are merged into a single new segment and removing the old segments.

But wait, whats it the benefits Merge my segments?

Basically, because two important things, discard old documents and as result reduce our index space on disk, and the second one is the old segments are remove, and a new bigger segment are create, incresing your search speed.

Conclusion

Understand Lunce is essential to understand Elasticsearch, of course are difference beetwen them, but the concept and how things works, is the same. This kind of concepts is hard to find in ElasticSearch documents, because they abstract all this job for you. I hope i can explain some things about the core of Lucene, to show how ElasticSearch trully works.

References and Inspirations:

https://www.manning.com/books/lucene-in-action-second-edition