Getting Lucene Down to Business With Lucid Imagination

On Jan. 26, Lucid Imagination opened for business as the commercial entity for the Apache Lucene-Solr ecosystem. The new company hopes to unite both the Lucene and the open source developer communities under its wing by offering product support, training, consulting services and value-added software for enterprises using Lucene- and Solr-based search solutions.

The Lucene Search Library was developed some 10 years ago by Doug Cutting, who released it as an open source project under the Apache license. It became one of the top five Apache projects, with more than 4,000 installations and 6,000 downloads per day, according to Eric Gries, CEO of Lucid. With its adoption by companies including Netflix, Monster, AOL, Comcast, Apple, LinkedIn, MySpace and Wikipedia, Lucene’s use has grown ten-fold in the last three years.

The Solr Search Server is a Web-based search service that sits on top of Lucene. It was originally developed at Cnet by Yonik Seeley to add a schema, administrative tools, cache management, replication and faceted browse features to the Lucene Search Library. It is the fastest growing Lucene sub-project, noted Gries.

“Lucene was one of the most underestimated players in the enterprise search market. There came a point where there was a very large installed base. It is in use as a mission-critical application. This was the right timing to introduce a commercial entity,” Gries told LinuxInsider.

Seeding an Idea

It took a solid decade for Lucene to become the search tool of choice for so many enterprise users. Although it quietly gathered momentum over the last 10 years, it only came into its own over the last two years, with a tremendous increase in the number of downloads. If you ask Gries, it’s now as good or better than what is available commercially.

“Enterprise search requires constant caring. You have to take care of it all the time to prevent the accuracy of the results from going down. All that was available before our commercial launch was the volunteer support from the community,” Gries said.

His new company will provide that care and special handling by providing subscribers an annual or semiannual health check of the enterprise system along with development support with code architectural system reviews as the document search and handling subsystems are built into enterprise networks.

Gaining Attention

Lucid Imagination Founder and CTO Marc Krellenstein is an expert on search technologies with more than 20 years of experience. He stumbled upon Lucene about seven years ago while evaluating search engines for another company. At that point, he was only mildly impressed with it.

“I had never head of it and wasn’t sure how many installations were running. I thought it was very good but not quite ready for prime time. Features were missing, and it had some bugs. It was mature, but I didn’t know how long it would be around. So I standardized the company on another product,” Krellenstein told LinuxInsider.

Three years later, Krellenstein adjusted his assessment. But his experience in trying to move forward with its use typifies a problem that enterprises often see when using open source projects.

The Spark Kindles

Clearly, Lucene was getting better — it was free and had good performance characteristics, Krellenstein surmised.

“I tried to move the company to Lucene, but it was hard because there was no company behind Lucene and nobody was openly supporting it. It was well written and fairly self supporting. But nobody at the company liked the idea that there was nobody on the other end to support it,” he explained.

Perhaps that was the spark that smoldered until he joined forces with Gries to fill that void with Lucene. One thing lead to another. It was a natural development to form a company behind Lucene, he said.

Krellenstein founded Lucid Imagination in August 2007. [*correction] The company secured US$6 million in Series A funding last September from Granite Ventures and Walden International Domain.

No Brainer

The decision to form Lucid Imagination seemed as logical as following a road map. They were at the intersection of two of the most exciting market sectors, Gries said. Better, that intersection of enterprise and open source search wasn’t crowded.

“The next step was to start recruiting the major committers to work for the company and arrange funding,” he said.

The numbers were impressive, even if the field already had notable players. The top competition included Autonomy, Google Appliance, Microsoft/Fast, IBM and Endeca, according to Gries.

However, the potential was clearly calling for an open source commercial entity. The market size was bulging at $1.8 billion. The growth rate was streaking at 28 percent, according to IDC Research.

Basic Product

Unlike some other startups, Lucid Imagination may have an easier time getting up and running. It was tweaking an existing product rather than starting from scratch with its own research and development.

Lucene, explained Krellenstein, is one of the first new generation search technologies. Its developers added missing features and fixed some problems.

“The Apache license helped a great deal. It’s a great vehicle for getting software developed. People can do with it what they want,” he said.

Solr’s entrance helped Lucene’s growth path in the last two years as well, according to Krellenstein. On its own, at a technical level, Lucene was really a Java-class library, bare-bones in terms of what was needed to turn it into a full-fledged search engine.

That did not stop its adopters, who had been using it for a long time. But it was a limitation for wide-scale adoption. Solr’s contribution changed all that, Krellenstein said.

Fast Track

Lucid Imagination opened its doors to a ready-made customer list. All it had to do was sell the support package to make using Lucene-Solr a better deal than the established enterprise search companies offered.

Lucene already had widespread penetration, according to Krellenstein.

“Now with 4,000 installations and being so good and so free, I think it’s really going to become part of the standard stack for a lot of the companies if it isn’t already,” he said.

Waiting Customers

Most of the people Lucid Imagination will contact in the near future have already installed and are using the Lucene search engine. So the company instantly has a target audience to pitch.

“We see the market growing dramatically. With the economy going through a meltdown, you have a segment here that is continuing to grow. Because we’re coming from the open source angle, we are extremely attractive for customers to use,” predicted Gries.

As the company goes after this installed base, it can also approach other potential clients replacing commercial enterprise search with open source.

Gries’ market plan, in summary: “Why do you want to pay a few hundred thousand dollars per year for software you can get for free and just pay for support?”

The Market

Generally, all search engines did well recently, big and small companies alike. Last year, the entire market grew despite the recession, noted Sue Feldman, vice president for search and discovery technologies for IDC. She does not expect the search market to be hit quite as hard as the other IT markets.

“There are a lot of reasons for this growth. For companies like Autonomy, compliance is driving the market. You’ve got top-level officials running scared. It is just too expensive not to automate the process now that federal rules apply. Once the dust settles, you can expect that everybody will be suing everybody else,” Feldman told LinuxInsider.

In the short term, all of the government stimulus spending could spur this market tremendously. All of the infrastructure projects proposed by the new administration will have a search component. In the long term, Feldman predicted a shift away from enterprise resource planning (ERP) transaction-based platforms and toward information management and access platform as the next big investment.

Roots Taking Hold

Lucid Imagination brings a new factor to the enterprise search industry. Its roots in open source could help its potential success.

“Having that expertise behind Lucene and Solr are the building blocks, but they are not the application. What Lucene and Solr give Lucid Imagination is open source, non-proprietary code upon which they can build. And that will appeal to a certain portion of the market,” said Feldman.

Lucid Imagination’s roots will appeal to two key segments. The first one is software vendors who have to imbed search in a larger applications as more and more applications require a search component. The second segment is the population of very large, content-rich organizations such as client aggregators, content distributors and content suppliers, according to Feldman. These companies need to have a heavy-duty application that can handle large volumes of queries and documents and keep them up to date very quickly.