Full-text search for a community site

Some online services can’t do without full-text search. On one project

we needed to look into integrating a third-party search engine. It’s a community site

where search is a key feature. I did a little research of my own, and

this post is part of it.

The main requirement is that the tool have a web crawler that can run against the site and

build an index. We don’t need to index local documents or

anything like that. Another constraint is that we can’t use third-party search with ads

like AdWords or the like.

Basic requirements (in order of importance)

• Relevance of results
• Search speed
• Support for searching in several languages
• Easy integration
• Customizable search results
• A good price

Google is synonymous with search for me, so my first stop was Google Mini.

Google Mini

Pros

• Proven search quality
• Simple plug-and-play installation and administration through a web interface
• Google technical support

Cons

• No way to try it out, apart from the one-month money-back guarantee
• A limit of 100,000 documents
• Relatively expensive at $2,995

Next up was the Apache Lucene engine, which

I found through Dion Almaer’s I Love Lucene (TSS discussion).

Apache Lucene

Pros

• Completely free and open source
• No indexing limits
• Advanced integration through an API

Cons

• Completely untested for our purposes
• Complicated configuration that requires programming
• No web crawler

Put simply, Lucene is a piece of raw material that needs work before it can be

deployed. Still, there’s a great deal of

software built on Lucene, including familiar names like JIRA (an issue tracker), Eclipse and Roller (similar to

Blogger). Two web search engines, Nutch and Red-Piranha, are also based directly on Lucene. Both are open source and both solve

the web crawler problem.

Being a patriot, I also thought of Jyxo, specifically Jyxo Data. I use Jyxo when Google fails to find purely

Czech content.

Jyxo

Pros

• Fairly well-proven search quality
• No indexing limits
• Easy integration through XML

Cons

• Support for other languages (English, French)
• Unknown pricing

I’d love to hear your experiences, observations and opinions in the comments.