Query Construction API
Zend_Search_Lucene
Programmer's Reference Guide

Character Set

UTF-8 and single-byte character set support

Zend_Search_Lucene works with the UTF-8 charset internally. Index files store unicode data in Java's "modified UTF-8 encoding". Zend_Search_Lucene core completely supports this encoding with one exception. [1] Zend_Search_Lucene

Actual input data encoding may be specified through Zend_Search_Lucene API. Data will be automatically converted into UTF-8 encoding.

Default text analyzer

However, the default text analyzer (which is also used within query parser) uses ctype_alpha() for tokenizing text and queries.

ctype_alpha() is not UTF-8 compatible, so the analyzer converts text to 'ASCII//TRANSLIT' encoding before indexing. The same processing is transparently performed during query parsing. [2]

Note: Default analyzer doesn't treats numbers as parts of terms. Use corresponding 'Num' analyzer if you don't want words to be broken by numbers.

UTF-8 compatible text analyzers

Zend_Search_Lucene also contains a set of UTF-8 compatible analyzers: Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8, Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8Num, Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8_CaseInsensitive, Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8Num_CaseInsensitive.

Any of this analyzers can be enabled with the code like this:

  1. Zend_Search_Lucene_Analysis_Analyzer::setDefault(
  2.     new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8());
Warning

UTF-8 compatible analyzers were improved in Zend Framework 1.5. Early versions of analyzers assumed all non-ascii characters are letters. New analyzers implementation has more accurate behavior.

This may need you to re-build index to have data and search queries tokenized in the same way, otherwise search engine may return wrong result sets.

All of these analyzers need PCRE (Perl-compatible regular expressions) library to be compiled with UTF-8 support turned on. PCRE UTF-8 support is turned on for the PCRE library sources bundled with PHP source code distribution, but if shared library is used instead of bundled with PHP sources, then UTF-8 support state may depend on you operating system.

Use the following code to check, if PCRE UTF-8 support is enabled:

  1. if (@preg_match('/\pL/u', 'a') == 1) {
  2.     echo "PCRE unicode support is turned on.\n";
  3. } else {
  4.     echo "PCRE unicode support is turned off.\n";
  5. }

Case insensitive versions of UTF-8 compatible analyzers also need » mbstring extension to be enabled.

If you don't want mbstring extension to be turned on, but need case insensitive search, you may use the following approach: normalize source data before indexing and query string before searching by converting them to lowercase:

  1. // Indexing
  2. setlocale(LC_CTYPE, 'de_DE.iso-8859-1');
  3.  
  4. ...
  5.  
  6. Zend_Search_Lucene_Analysis_Analyzer::setDefault(
  7.     new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8());
  8.  
  9. ...
  10.  
  11. $doc = new Zend_Search_Lucene_Document();
  12.  
  13. $doc->addField(Zend_Search_Lucene_Field::UnStored('contents',
  14.                                                   strtolower($contents)));
  15.  
  16. // Title field for search through (indexed, unstored)
  17. $doc->addField(Zend_Search_Lucene_Field::UnStored('title',
  18.                                                   strtolower($title)));
  19.  
  20. // Title field for retrieving (unindexed, stored)
  21. $doc->addField(Zend_Search_Lucene_Field::UnIndexed('_title', $title));
  1. // Searching
  2. setlocale(LC_CTYPE, 'de_DE.iso-8859-1');
  3.  
  4. ...
  5.  
  6. Zend_Search_Lucene_Analysis_Analyzer::setDefault(
  7.     new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8());
  8.  
  9. ...
  10.  
  11. $hits = $index->find(strtolower($query));
[1] supports only Basic Multilingual Plane (BMP) characters (from 0x0000 to 0xFFFF) and doesn't support "supplementary characters" (characters whose code points are greater than 0xFFFF) Java 2 represents these characters as a pair of char (16-bit) values, the first from the high-surrogates range (0xD800-0xDBFF), the second from the low-surrogates range (0xDC00-0xDFFF). Then they are encoded as usual UTF-8 characters in six bytes. Standard UTF-8 representation uses four bytes for supplementary characters.
[2] Conversion to 'ASCII//TRANSLIT' may depend on current locale and OS.
Query Construction API
Zend_Search_Lucene
Programmer's Reference Guide