Introduction to Elasticsearch

Going through my previous blogs it will be clearly evident that I am very fond of writing about the topic of "Full Text Search". This time also, I would like to discuss on a topic related to Searching and Indexing. That is, about a document based NoSQL database named Elasticsearch which uses an underlying full-text search engine to store data as indexes and query it efficiently. Recently, I had an opportunity to work on this product and was fascinated by its concept and features.

I don't wish to get into the theory of NoSQL databases and Polyglot persistence here. For that you can refer to the book NoSQL Distilled by Martin Fowler, which gives a good introduction to this subject. Here I would like to give a brief introduction about ElasticSearch and explain how ElasticSearch is related to the concept of full text search.

A little bit of history (as per Wikipedia):

Shay Banon created Compass Framework in 2004. (You can read about Compass from my previous article here ). While thinking about the third version of Compass he realized that it would be necessary to rewrite big parts of Compass to "create a scalable search solution". So he created "a solution built from the ground up to be distributed" and used a common interface, JSON over HTTP, suitable for programming languages other than Java as well. Shay Banon released the first version of Elasticsearch in February 2010.

By the way, what is Elasticsearch?:

Elasticsearch is a distributed RESTful search engine built for the cloud. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. You can Download Elasticsearch from the official distribution site.

How is Elasticsearch related to Full Text Search?:

ElasticSearch is powered by Lucene, a powerful open-source full-text search library, under the hood. ElasticSearch uses Apache Lucene to create and manage an inverted index and instead of searching the text directly, it searches an index instead. Hence ElasticSearch is able to achieve fast search responses. This is like retrieving pages in a book related to a keyword by scanning the index at the back of a book, as opposed to searching every word of every page of the book.

My Experiences with Elasticsearch:

Recently, I was involved in a migration of Data-warehouse from relational star based schema to Elasticsearch NoSQL data store. This rewrite simplified the schema definition for analytics significantly and gave amazing performance benefits for storing (indexing) and querying (searching) data. ElasticSearch also gave us the benefits of horizontal scaling (with the concept of shrads) which is excellent to handle big data.

But one challenge with NoSQL database is to deal with the non-conventional style of queries and resultset (usually a JSON based notation). It might seem non-intuitive to Developers, who are much familiar with the intuitive and standard notation of Structured Query Language (SQL) for relational databases. This is clearly a disadvantage for projects using NoSQL datastores, as a lot of time is spend by Developers in writing code for translating SQL statements to its NoSQL counterpart statements.

Due to the complexity and learning curve associated with the non-conventional style of querying, firing even simple queries against NoSQL databases becomes cumbersome and time consuming. When I was working on the migration project, I was searching for an open source tool that could give a SQL interface for the underlying NoSQL data store, so that I could verify data easily outside the application. But I couldn't find one, so I had to use the web-based Marvel plugin for Elasticsearch and had to spent lot of time learning the syntax for creating expressions to query data in the format Elasticsearch understands.

So after finishing the migration project, I started working an open source project for developing a web based query tool for querying Elasticsearch using standard SQL syntax. This project is currently publicly hosted in GitHub under name sql-query-browser-for-elasticsearch. This project is still under development (with daily check-ins), but at it current state, it can be used as a thin web client for Elasticsearch Server running on any machine to execute simple queries against a hosted Elasticsearch server. The project can be downloaded using the Git client and run locally.

Emerging visualization trends with Elasticsearch:

Elastic Search, Logstash and Kibana – the ELK Stack – is emerging as the best technology stack to collect, manage and visualize big data. These technology stack provides good visualization tools and relieves developers from the need to write custom queries to fetch data.

Conclusion:

The world of NoSQL is still in its infancy and need to go a long way to get to a position where relational databases are positioned now. Maybe in near future, we can expect a standardization in the language for getting information from and updating NoSQL databases, similar to what SQL provides now for relational databases. Till then, we need a community involvement in producing more open-source, user friendly query tools to interface with these feature rich databases.

References:

1. Wikipedia entry for Elasticsearch

2. Github project for Elasticsearch

Full-Text Search using Oracle Text

This is in continuation with my previous post on Full Text Search in which I discussed about MySQL’s built-in Full Text Search engine and external Open Source Full Text Search engines as options for performing integrating full-text search features in java applications. This time I want to share information on building Full-Text Search Applications with Oracle Text.

Oracle Text:

Oracle Text is a powerful search technology built into all Oracle Database editions, including the free Express Edition (XE). The development APIs provided by Oracle Text allow software developers to easily implement full-featured content search applications.

Oracle Text is suitable for a wide variety of search-related use cases and storage structures. Application areas for Text include e-business, document and records management as well as issue tracking just to name a few. Retrievable text can reside in a structured form inside the database or in unstructured form either in a local file system or on the Web.

Oracle Text can be used to search structured and unstructured documents complementing the SQL wildcard matching. Oracle Text provides a complete SQL-based search API that consists of custom query operators, DDL syntax extensions, a set of PL/SQL procedures and database views. Text API gives the application developer full control over indexing, queries, security, presentation, and software configuration that is sometimes required. Oracle Text is also programming-language agnostic and works equally well for PHP as well as Java applications.

Setting Up Oracle Text:

Oracle Text is installed with an Oracle Database XE installation by default. With other database editions, you need to install the Oracle Text feature yourself. Once the feature is present, you only need to create a normal database user and grant the CTXAPP role to the user. This will allow the user to execute certain index management procedures:

Indexing Process and Searching

Oracle Text indexes retrievable data items before users are able to find content with search. This is a common approach used to ensure adequate search performance. The Oracle Text indexing process is modeled after a pipeline, where data items retrieved from a data store pass through a series of transformations before their keywords are added to the index. The indexing process is split into multiple phases, where each phase is handled by a separate entity and configurable by the application developer.

Oracle Text has different index types that are suitable for different purposes. For full-text search with large documents, the CONTEXT index is the appropriate index type. The indexing process includes the following phases:

1. Data Retrieval: Data is simply fetched from a data store, for example, a Web page, database large object, or local file system, and passed as a stream of data to the next phase.
2. Filtering: The filters are responsible for converting data in different file formats to plain text. The other components in the indexing pipeline only process plain text data and don't know about file formats such as Microsoft Word or Excel.
3. Sectioning: The sectioner adds metadata about the structure of the original data item.
4. Lexing: A stream of characters is split into words based on the language of the item.
5. Indexing: In this final phase, the keywords are added to the actual index.

Once the index has been built, an application can use plain SQL queries to execute a search entered by an end user.

Searching

The CONTAINS operator is used for searching CONTEXT indexes.

Index Maintenance

Because base table data is replicated by the index, the data needs to be periodically synchronized to the index. Index maintenance procedures can be found in the CTX_DDL PL/SQL package.

Summary

Oracle Text allows users to create full-text index on a single column / multiple columns in a single table as well across multiple tables in a database. Details on creating the index, searching and index maintence is discussed comprehensively in the OTN Developer article on full text indexing.

References:

1. OTN Developer article on full text indexing

2. OTN Discussion Forum - Topic on multi-table indexing

3. Thread on full-text indexing

Integrating Lucene with Spring Framework & Hibernate

While looking for integration support for Lucene with Spring Framework & Hibernate, I have come across a full-blown open source Java Search Engine Framework called Compass Framework which is built on top of the Lucene Search Engine and provides seamless integration support to popular development frameworks like Hibernate and Spring Framework.

Why do we need yet another framework for implementing search functionality?

Lucene is a low level API which implies that it can easily cause coupling problems especially with the domain objects. This way of directly coding the Lucene API into the application maybe a performance killer and can also become a cause of maintenance nightmare in future (with domain model changes). Looking for other options for integrating Lucene with our Spring based application, I came across two alternatives that exist in the open-source arena:

1. Lucene Spring Modules

One option is using the "Lucene Spring Modules", which is a part of "Spring Modules project" which tries to extend the functionalities of Spring Framework to include other open-source tools. The project is intended to facilitate integration between Spring Framework and other projects without cluttering or expanding the Spring core.

2. Compass Framework

Another option is to use Compass Framework which provides a declarative way to map the domain model to the search engine. Compass provides a high level abstraction on top of the Lucene's low level API which supports a declarative mapping of domain objects. It externalizes all dependencies and coupling in a compass meta data file and thus provides a declarative technique to map the domain objects. Compass also implements fast index operations and optimization which increases the application performance.

Compass Framework provides a module named "Compass::Spring" which is intended to provide closer integration with the Spring Framework. It supports IoC using Spring's Application Context and provides support for Hibernate Session Factory. CF claims to support complex applications with bigger domain models easily. Compass also claims to bring maintenance and performance down to negligible values. Compass comes with a sample project (the old petclinic sample with additional search functionalities using Compass Framework) that demonstrates its integration support with Spring Framework & Hibernate. The product is also quite mature with much elaborate documentation. The current stable version is compass version 1.1M2.

More about Compass Framework

Compass is a first class open source Java Search Engine Framework, enabling the power of Search Engine semantics to your application stack decoratively. Compass is a powerful, transactional Object to Search Engine Mapping (OSEM) Java framework which allows you to declaratively map your Object domain model to the underlying Search Engine, synchronizing data changes between Index and different datasources. Compass provides a high level abstraction on top of the Lucene low level API. Compass also implements fast index operations and optimization and introduce transaction capabilities to the Search Engine.

In recent versions, compass provides a Lucene Jdbc Directory implementation, allowing storing Lucene index within a database for both pure Lucene applications and Compass enabled applications. Compass also provides support to SpringHibernate Gps Device (configured in Spring context file using IoC) which utilizes Compass OSEM feature (Object to Search Engine Mappings) and Hibernate ORM feature (Object to Relational Mappings) to provide simple database indexing. All the OSEM mappings are defined in a compass meta-data file and the SpringHibernate Gps Device intercepts the Hibernate session factory object to index data transparently. The Gps Device also provide real time mirroring of data changes done through Hibernate so you didn't have to explicitly re-index data after a store/update/delete. The path data travels through the system are: Database -- Hibernate -- Objects -- Compass::Gps -- Compass::Core (Search Engine). The compass returns the ids of objects matched along with a tag that identifies the class of object it belongs.

Dear readers don’t forget to read about the origin of compass framework as described by the author Shay Banon’s on his blog. It is well written and I bet you will surely enjoy the narration!!!!!!!!

References:

Open Symphony's Page
Shay Banon’s Blog

Full Text Search

In this article I have tried to evaluate some of the options for integrating full-text search features in java applications.

MySQL’s built-in Full Text Search engine

From my initial search what I could find was that MySQL’s built-in Text Search Engine surprisingly does effective full-text searching if the dataset is small. Also it has the least cost to implement since the search criteria can be specified as a part of query itself. But as the size of dataset grows its efficiency becomes dependent on the system resources like CPU, RAM etc.

Open Source Full Text Search engines

Most of the external full text search engines work by keeping a separate index of the table data which will be updated at frequent intervals (maybe with some amount of caching) so that time spend on the database server is less for searching for information. This approach will certainly lessen the load on the database server.

A complete list of popular full text search engines is available at WikiMedia site

1. Sphinx

Sphinx is a full-text search engine, distributed under GPL version 2. Generally, it's a standalone search engine, meant to provide fast, size-efficient and relevant full-text search functions to other applications. Sphinx was specially designed to integrate well with SQL databases and scripting languages. Currently built-in data source drivers support fetching data either via direct connection to MySQL, PostgreSQL, or from a pipe in a custom XML format.

2. Lucene

Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java with features like Scalable High-Performance Indexing, Powerful Accurate and Efficient Search Algorithms etc.

As a full-text search engine, Lucene needs little introduction. Lucene, an open source project hosted by Apache, aims to produce high-performance full-text indexing and search software. The Java Lucene product itself is a high-performance, high capacity, full-text search tool used by many popular Websites such as the Wikipedia online encyclopedia and TheServerSide.com, as well as in many, many Java applications. It is a fast, reliable tool that has proved its value in countless demanding production environments.

Although Lucene is well known for its full-text indexing, many developers are less aware that it can also provide powerful complementary searching, filtering, and sorting functionalities. Indeed, many searches involve combining full-text searches with filters on different fields or criteria. For example, you may want to search a database of books or articles using a full-text search, but with the possibility to limit the results to certain types of books. Traditionally, this type of criteria-based searching is in the realm of the relational database. However, Lucene offers numerous powerful features that let you efficiently combine full-text searches with criteria-based searches and sorts.

Bench Marks

The results of benchmarking the most popular full text search engines (MySQL’s built-in Text Search engine, Sphinix Text Search engine plug-in for MySQL and Lucene) is published in the PlanetMySQL site.

Conclusion

Lucene is “the most” popular full text search solution available now to conduct efficient full text searches on database compared to MySQL’s built-in Text Search engine and Sphinix plug-in for MySQL. Lucene is just a java API so it provides seamless integration with other Java programs as compared to Sphinix written in pearl. It is a set of tools that allows us to create an index and then search it. So we need to manually handle the index creation/updation and searches using the API. But the good news is that Spring & Hibernate supports integration of Lucene through various support classes. So go ahead and use it in your java projects. Have a great time searching inside your applications with Lucene!!!

Reference:

Java World Article on Integrating Lucene