[ InAction Note ] Ch4. Lucene’s analysis process - Stemming analysis

Preface: 
Our final analyzer pulls out all the stops. It has a ridiculous, yet descriptive name: PositionalPorterStopAnalyzer. This analyzer removes stop words, leaving positional holes where words are removed, and leverages a stemming filter. 

The PorterStemFilter is shown in the class hierarchy in figure 4.5, but it isn’t used by any built-in analyzer. It stems words using the Porter stemming algorithm created by Dr. Martin Porter, and it’s best defined in his own words: 

The Porter stemming algorithm (or “Porter stemmer”) is a process for removing the commoner morphological and inflexional endings from words in English. Its main use is as part of a term normalisation process that is usually done when setting up Information Retrieval systems.

In other words, the various forms of a word are reduced to a common root form. For example, the words breathe, breathes, breathing, and breathed, via the Porter stemmer, reduce to breath. 

The Porter stemmer is one of many stemming algorithms. See section 8.2.1 for coverage of an extension to Lucene that implements the Snowball algorithm (also created by Dr. Porter). KStem is another stemming algorithm that has been adapted to Lucene (search Google for KStem and Lucene). 

Next we’ll show how to use StopFilter to remove words but leave a positional hole behind, and then we’ll describe the full analyzer. 

StopFilter leaves holes: 
Stop-word removal brings up an interesting issue: what happens to the holes left by the words removed? Suppose you index “one is not enough.” The tokens emitted fromStopAnalyzer will be one and enough, with is and not thrown away. By default, StopAnalyzer accounts for the removed words by incrementing the position increment. 

This is illustrated from the output of AnalyzerUtils.displayTokensWithPositions: 

view plaincopy to clipboardprint?

1. AnalyzerUtils.displayTokensWithPositions(new StopAnalyzer(Version.LUCENE_30),  
2.         "The quick brown fox jumps over the lazy dog");  

Output: 

2: [quick]
3: [brown]
4: [fox]
5: [jump]
6: [over]
8: [lazi]
9: [dog]

Positions 1 and 7 are missing due to the removal of the. If you have a need to disable the holes so that position increment is always 1, use StopFilter’ssetEnablePositionIncrements method. But be careful when doing so: your index won’t record the deleted words, so there can be surprising effects. For example, the phrase "one enough" will match the indexed phrase "one is not enough" if you don’t preserve the holes! 

Stepping back a bit, the primary reason to remove stop words is because these words typically have no special meaning; they are the “glue” words required in any language. The problem is, because we’ve discarded them, we’ve lost some information, which may or may not be a problem for your application. For example, nonexact searches can still match the document, such as "a quick brown fox." 

There’s an interesting alternative, called shingles, which are compound tokens created from multiple adjacent tokens. Lucene has a TokenFilter called ShingleFilter in the contrib analyzers module that creates shingles during analysis. We’ll describe it in more detail in section 8.2.3. With shingles, stop words are combined with adjacent words to make new tokens, such as the-quick. At search time, the same expansion is used. This enables precise phrase matching, because the stop words aren’t discarded. Using shingles yields good search performance because the number of documents containing the-quick is far fewer than the number containing the stop word the in any context. 

Combining stemming and stop-word removal: 
This custom analyzer uses a stop-word removal filter, enabled to maintain positional gaps and fed from a LowerCaseTokenizer. The results of the stop filter are fed to the Porter stemmer. Listing 4.12 shows the full implementation of this sophisticated analyzer. LowerCaseTokenizer kicks off the analysis process, feeding tokens through the stop-word removal filter and finally stemming the words using the built-in Porter stemmer. 
- Listing 4.12 PositionalPorterStopAnalyzer: stemming and stop word removal 

view plaincopy to clipboardprint?

1. package ch4;  
2.   
3. import java.io.Reader;  
4. import java.util.Set;  
5.   
6. import org.apache.lucene.analysis.Analyzer;  
7. import org.apache.lucene.analysis.LowerCaseTokenizer;  
8. import org.apache.lucene.analysis.PorterStemFilter;  
9. import org.apache.lucene.analysis.StopAnalyzer;  
10. import org.apache.lucene.analysis.StopFilter;  
11. import org.apache.lucene.analysis.TokenStream;  
12.   
13. public class PositionalPorterStopAnalyzer extends Analyzer {  
14.     private Set stopWords;  
15.   
16.     public PositionalPorterStopAnalyzer() {  
17.         this(StopAnalyzer.ENGLISH_STOP_WORDS_SET);  
18.     }  
19.   
20.     public PositionalPorterStopAnalyzer(Set stopWords) {  
21.         this.stopWords = stopWords;  
22.     }  
23.   
24.     public TokenStream tokenStream(String fieldName, Reader reader) {  
25.         StopFilter stopFilter = new StopFilter(true, new LowerCaseTokenizer(reader), stopWords);  
26.         stopFilter.setEnablePositionIncrements(true);  
27.         return new PorterStemFilter(stopFilter);  
28.     }  
29. }  

Then you can test with below code: 

view plaincopy to clipboardprint?

1. AnalyzerUtils.displayTokensWithPositions(new PositionalPorterStopAnalyzer(), "the quick fox jumps");  

The output will be: 

2: [quick]
3: [fox]
4: [jump]

The "jumps" has been stemmed to be "jump"! For the implementation of displayTokensWithPositions, please refer here on topic Visualizing token positions.