Lucene源代码之构造自己的分词器

分词名称：TjuChineseAnalyzer.源代码如下：

1. package org.apache.lucene.analysis.tjuchinese;
3. import java.io.IOException;
4. import java.io.Reader;
5. import java.io.StringReader;
6. import java.util.Set;
8. import org.apache.lucene.analysis.Analyzer;
9. import org.apache.lucene.analysis.StopFilter;
10. import org.apache.lucene.analysis.TokenStream;
12. import com.xjt.nlp.word.ICTCLAS;
14. public final class TjuChineseAnalyzer extends Analyzer {
15.     private Set stopWords;
17.     // 可以在此扩展English stop words和Chinese stop words
18.     public static final String[] ENGLISH_STOP_WORDS = { "a", "an", "and",
19.             "are", "as", "at", "be", "but", "by", "for", "if", "in", "into",
20.             "is", "it", "no", "not", "of", "on", "or", "s", "such", "t",
21.             "that", "the", "their", "then", "there", "these", "they", "this",
22.             "to", "was", "will", "with", "我", "我们" };
24.     /** Builds an analyzer which removes words in ENGLISH_STOP_WORDS. */
25.     public TjuChineseAnalyzer() {
26.          stopWords = StopFilter.makeStopSet(ENGLISH_STOP_WORDS);
27.      }
29.     /** Builds an analyzer which removes words in the provided array. */
30.     public TjuChineseAnalyzer(String[] stopWords) {
31.         this.stopWords = StopFilter.makeStopSet(stopWords);
32.      }
34.     /** Filters LowerCaseTokenizer with StopFilter. */
35.     public TokenStream tokenStream(String fieldName, Reader reader) {
36.         try {
37.              ICTCLAS splitWord = new ICTCLAS();
38.              String inputString = FileIO.readerToString(reader);
39.              String resultString = splitWord.paragraphProcess(inputString);
40.              TokenStream result = new TjuChineseTokenizer(new StringReader(
41.                      resultString));
42.              result = new StopFilter(result, stopWords);
43.             return result;
44.             /*
45.               * return new StopFilter(new LowerCaseTokenizer(new StringReader(
46.               * resultString)), stopWords);
47.               */
48.          } catch (IOException e) {
49.              System.out.println("转换出错");
50.             return null;
51.          }
52.      }
54. }

TjuChineseTokenizer.java;

1. package org.apache.lucene.analysis.tjuchinese;
3. import java.io.Reader;
4. import org.apache.lucene.analysis.LowerCaseTokenizer;
6. public class TjuChineseTokenizer extends LowerCaseTokenizer{
7.      public TjuChineseTokenizer(Reader Input)
8.       {
9.          super(Input);
10.       }
11. }

FileIO.java;

1. package org.apache.lucene.analysis.tjuchinese;
3. import java.io.BufferedReader;
4. import java.io.IOException;
5. import java.io.Reader;
7. public class FileIO {
8.     
9.     public static String readerToString(Reader reader) throws IOException {
10.          BufferedReader br = new BufferedReader(reader);
11.          String ttt = null;
13.         // 使用 StringBuffer 类，可以提高字符串操作的效率
14.          StringBuffer tttt = new StringBuffer("");
15.         while ((ttt = br.readLine()) != null) {
16.              tttt.append(ttt);
17.          }
18.         return tttt.toString();
19.      }
20. }

eclipse里面的部署为：

搞定！！

下面测试一下，测试代码如下：

1. package org.apache.lucene.analysis.tjuchinese;
3. import java.io.IOException;
4. import java.io.StringReader;
6. import org.apache.lucene.analysis.Analyzer;
7. import org.apache.lucene.analysis.Token;
8. import org.apache.lucene.analysis.TokenStream;
10. public class testTjuChjnese {
11.     public static void main(String[] args) {
13.          String string = "hello!我爱中国人民";
14.          Analyzer analyzer = new TjuChineseAnalyzer();
15.          TokenStream ts = analyzer
16.                  .tokenStream("dummy", new StringReader(string));
18.          Token token;
19.          System.out.println("Tokens:");
21.         try {
22.             int n = 0;
23.             while ((token = ts.next()) != null) {
24.                  System.out.println((n++) + "->" + token.toString());
25.              }
26.          } catch (IOException ioe) {
27.              ioe.printStackTrace();
28.          }
29.      }
31. }

运行结果：

Tokens:
0->(hello,0,5)
1->(nx,6,8)
2->(w,12,13)
3->(r,17,18)
4->(爱,20,21)
5->(v,22,23)
6->(中国,25,27)
7->(ns,28,30)
8->(人民,32,34)
9->(n,35,36)

【难点】

1.   public CharArraySet(int startSize, boolean ignoreCase) {
2.     this.ignoreCase = ignoreCase;
3.     int size = INIT_SIZE;
4.     while(startSize + (startSize>>2) > size)
5.        size <<= 1;
6.      entries = new char[size][];
7.    }

startSize + (startSize>>2，不解？

附录：

因为本分词器要用到ICTCLAS java接口。所以要先下载下载地址http://download.csdn.net/source/778456；

将文件全部复制到工程文件中（也可以通过导入）后，得到的eclipse视图如下：

导入视图如下（仅供参考）：

如果出现这样的错误：

java.lang.UnsatisfiedLinkError: no ICTCLAS in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at com.xjt.nlp.word.ICTCLAS.<clinit>(ICTCLAS.java:37)
Exception in thread "main"
那么应该就是你缺少了某些文件，尤其是ICTCLAS.dll，另外像“classes”、“data”、“lib”源文件夹也是必需的。