Author: Zhang Yan. Foreword: In July 2008, I wrote an article titled “Architecture Design of a Ten-Million-Scale Full-Text Search (Search Engine) Based on Sphinx+MySQL“. Quite a few readers wanted to read the full text, so I cleaned up the document and shared it. Unzipped, the document is 7.33M and 19 pages. Download link for this site: http://blog.zyan.cc/book/sphinx/sphinx_mysql.zip Sina mirror: http://ishare.iask.sina.com.cn/f/6728201.html The limitations of that document’s architecture, I already pointed out in my December 2008 article “Architecture Design of a High-Concurrency General-Purpose Search Engine for Hundred-Million-Scale Data“: first, MySQL’s own concurrency is limited — under 200-300 concurrent connections, queries and updates get slow; second, because MySQL table primary keys map one-to-one to Sphinx index IDs, you can’t build a site-wide query across multiple tables, and adding a new category means editing the config file, which is a hassle; third, because it’s integrated with MySQL, it can’t bring out Sphinx’s advantages. Even so, for search applications with modest write volume it’s already good enough, and it will probably help a lot of people.
Main text: After that, the Sphinx distributed general-purpose site search engine platform I developed based on “Architecture Design of a High-Concurrency General-Purpose Search Engine for Hundred-Million-Scale Data“ has been running in production for over 9 months. Through continuous refinement and improvement in operations, it has now become a scalable distributed general-purpose site search engine framework. Add, delete, and update operations from products such as CMS, video, and forums write their text content in real time to the self-developed HTTPSQS high-performance simple message queue service, and a queue controller updates the index and storage. It provides API query interfaces supporting XML and JSON, and supports hundred-million-scale indexing, distribution, Chinese word segmentation, highlighting, automatic summarization, and near-real-time (within 1 minute) incremental index updates.
Below are some introductions to how the technical key points of the new Sphinx search architecture are implemented, shared here for discussion: 1. Combining unigram segmentation with Chinese word segmentation: ① The unigram segmentation lives in the index update module. The Sphinx indexing engine supports unigram segmentation for CJK (Chinese, Japanese, Korean) languages (which must be UTF-8 encoded). Take the sentence 【反恐行动是国产主视角射击网络游戏】 — Sphinx splits it into 【反 恐 行 动 是 国 产 主 视 角 射 击 网 络 游 戏】, then builds a reverse index for each character. If you combine characters from this sentence into a word that doesn’t exist, such as 【恐动】, it will also be found, so when searching you need to add quotes — for example searching 【”反恐 行动”】 matches exactly the four characters together, while the non-contiguous 【”恐动”】 will not be found. But this still leaves a problem: searching 【”反恐行动游戏”】 or 【”国产网络游戏”】 finds nothing. To handle that, Chinese word segmentation in the search query module is used. The configuration for UTF-8 Chinese unigram segmentation in the sphinx.conf config file is as follows:
…omitted…
index t_source_main {
source = t_source_main
path = /data0/search/sphinx/data/t_source_main
docinfo = extern
mlock = 0
morphology = none
min_word_len = 1
charset_type = utf-8
min_prefix_len = 0
html_strip = 1
charset_table = 0..9, A..Z->a..z, \_, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F
ngram_len = 1
ngram_chars = U+3000..U+2FA1F
}
…omitted…
② Chinese word segmentation lives in the search query module. When searching “反恐行动游戏” or “国产网络游戏”, an independent Chinese word segmentation system is called first, splitting them into “反恐行动 游戏” and “国产 网络游戏” respectively. Then quotes are added around the space-separated terms, and you search Sphinx for 【”反恐行动” “游戏”】 or 【”国产” “网络游戏”】, and that record will be found. When the Chinese word segmentation dictionary has additions, deletions, or changes, there’s no need to rebuild the entire Sphinx search index.
2. Use the self-developed HTTPSQS (http://code.google.com/p/httpsqs) open-source simple queue service to buffer high-concurrency data writes. For add, delete, and update operations on news, forum posts, customer service announcements, SNS communities, and so on, the text content is written in real time through the update interface into the HTTPSQS queue, and then a queue controller updates it into the Sphinx search engine index.
3. A small issue where Sphinx can’t strictly sort by field. If you don’t want to use weighting and only want strict sorting by time, primary key, etc., while the matching mode is not SPH_MATCH_BOOLEAN (the more commonly used ones are SPH_MATCH_ALL and SPH_MATCH_EXTENDED), then Sphinx search results on a given page will be sorted somewhat inaccurately. For example: sorting in descending order by UNIX timestamp, with 0,20 as the first page and 20,40 as the second page — the minimum timestamp on the first page will certainly be greater than the maximum timestamp on the second page, but the 0,20 records within the first page will not be strictly sorted by timestamp, and likewise for the second page. So if precise sorting is needed, when a user flips to a particular page of search results, the records on that Sphinx search result page must be sorted again separately. In my search architecture, this re-sorting is handled by the search.php query interface using the array_multisort() function. Generally a page only displays 5-30 records, so re-sorting just a few dozen records in PHP is very fast.
4. Combining “time control” with “quantity control” in the queue controller to achieve near-real-time index updates within 1 minute: ① Sphinx 0.9.9’s indexing speed in production is roughly 5.5 Mbytes/sec, 6400 documents/sec. The queue controller can be set to update the incremental index once every 10 seconds; as long as the document count in the Sphinx incremental index data source stays under 380,000, the incremental index is guaranteed to be updated within 1-60 seconds — this is control from the “time” side. ② To avoid the document count in the incremental index data source growing to 380,000, the queue controller also activates merging the incremental index into the main index once the incremental index data source exceeds 10,000 documents; the merged documents are then deleted from the incremental index data source — this is control from the “quantity” side.
5. Usage notes for the self-written “search engine query API interface”:
http://xxx.xxx.xxx.xxx/search.php?query=%E9%87%91%E5%B1%B1 (search keyword. The program can tell whether the keyword is GBK- or UTF-8-encoded, and whether the keyword has been URL-encoded.) &output=xml (output type supported: xml or json) &excerpts=1 (whether to enable highlighting and text summarization; 1 on, 0 off) &excerpts_before= (highlighting and text summarization; if the value is empty, neither is performed. The string inserted before the matched keyword.) &excerpts_after= (highlighting and text summarization; if the value is empty, neither is performed. The string inserted after the matched keyword.) &excerpts_limit=256 (highlighting and text summarization; if the value is empty, neither is performed. The maximum number of symbols (code points) the summary may contain.) &excerpts_field=c1,c2,c3,c4,c5 (highlight only the specified fields and not the rest; if this parameter is empty, all string-type fields are highlighted by default) &offset=0&limit=20 (equivalent to limit 0,20 in SQL) &max_matches=30000 (maximum number of search results) &match_mode=SPH_MATCH_EXTENDED2 &ranking_mode=SPH_RANK_PROXIMITY_BM25 &sort_mode=SPH_SORT_EXTENDED&sort_by=@relevance DESC,u1 ASC,@id DESC (sort mode: @relevance and @id are built-in variables; @relevance means the relevance weight, @id equals search_id, and u1 is a field name) &field_weights=c1,7;c2,1 (weight settings: field c1 has weight 7, field c2 has weight 1) &filter=u1:0_1_6,false;u2:4,true (integer value filter: match the result set where field u1 equals 0, 1 or 6 and field u2 does not equal 4. false means equals, true means not equals) &filter_range=u1:0,100,false;u2:50,90,true (integer range filter: field u1 >= 0 and u1 <= 100, field u2 < 50 and u2 > 90) &filter_range=u1:1.23,99.645,false; u2:1034.3,7834.56,true (floating-point range filter: field u1 >= 1.23 and u1 <= 99.645, field u2 < 1034.3 and u2 > 7834.56)
6. Example of the search result front-end page:
7. One server platform and API interface, shared across all kinds of products: Examples: Kingsoft game article and video search: http://s.xoyo.com/result.php?w=%E5%89%91%E7%BD%913 Kingsoft game forum post search: http://jx3.bbs.xoyo.com/search2.php?srchtxt=%E4%B8%83%E7%A7%80&select=title Original article link: http://blog.zyan.cc/sphinx_search/
