[ InAction Note ] Ch6. Extending search - Custom filters

Preface: 
If all the information needed to perform filtering is in the index, there’s no need to write your own filter because the QueryWrapperFilter can handle it, as described insection 5.6.5. But there are good reasons to factor external information into a custom filter. In this section we tackle the following example: using our book example data and pretending we’re running an online bookstore, we want users to be able to search within our special hot deals of the day. 

You might be tempted to simply store the specials flag as an indexed field, but keeping this up-to-date might prove too costly. Rather than reindex entire documents when specials change, we’ll implement a custom filter that keeps the specials flagged in our (hypothetical) relational database. Then we’ll see how to apply our filter during searching, and finally we’ll explore an alternative option for applying the filter. 

Implementing a custom filter 
We start with abstracting away the source of our specials by defining this interface: 

view plaincopy to clipboardprint?

1. package demo.ch6;  
2.   
3. public interface SpecialsAccessor {  
4.     String[] isbns();  
5. }  

The isbns() method returns those books that are currently specials. Because we won’t have an enormous amount of specials at one time, returning all the ISBNs of the books on special will suffice. Now that we have a retrieval interface, we can create our custom filter, SpecialsFilter. Filters extend from theorg.apache.lucene.search.Filter class and must implement the getDocIdSet(IndexReader reader) method, returning a DocIdSet. Bit positions match the document numbers. Enabled bits mean the document for that position is available to be searched against the query, and unset bits mean the document won’t be considered in the search. Figure 6.2 illustrates an example SpecialsFilter that sets bits for books on special (see listing 6.14). 
 

- Listing 6.14 Retrieving filter information from external source with SpecialsFilter 

view plaincopy to clipboardprint?

1. import java.io.IOException;  
2.   
3. import org.apache.lucene.index.AtomicReader;  
4. import org.apache.lucene.index.AtomicReaderContext;  
5. import org.apache.lucene.index.DocsEnum;  
6. import org.apache.lucene.index.Term;  
7. import org.apache.lucene.search.DocIdSet;  
8. import org.apache.lucene.search.DocIdSetIterator;  
9. import org.apache.lucene.search.Filter;  
10. import org.apache.lucene.util.Bits;  
11. import org.apache.lucene.util.OpenBitSet;  
12.   
13. public class SpecialsFilter extends Filter {  
14.     private SpecialsAccessor accessor;  
15.   
16.     public SpecialsFilter(SpecialsAccessor accessor) {  
17.         this.accessor = accessor;  
18.     }  
19.   
20.     @Override  
21.     public DocIdSet getDocIdSet(AtomicReaderContext ctx, Bits bits)  
22.             throws IOException {  
23.         AtomicReader reader = ctx.reader();  
24.         OpenBitSet oBits = new OpenBitSet(reader.maxDoc());  
25.         String[] isbns = accessor.isbns();  
26.         for (String isbn : isbns)   
27.         {  
28.             DocsEnum docEnum = reader.termDocsEnum(new Term("isbn", isbn));  
29.             while(docEnum.nextDoc()!= DocIdSetIterator.NO_MORE_DOCS)  
30.             {  
31.                 if(docEnum.freq()>0)  
32.                 {  
33.                     oBits.set(docEnum.docID());  
34.                 }  
35.             }  
36.         }  
37.         return oBits;  
38.     }  
39. }  

The filter is quite straightforward. First we fetch the ISBNs of the current specials. Next, we interact with the AtomicReader API to iterate over all documents matching each ISBN; in each case it should be a single document per ISBN because this is a unique field. The document was indexed with Field.Index.NOT_ANALYZED, so we can retrieve it directly with the ISBN. Finally, we record each matching document in an OpenBitSet, which we return to Lucene. Let’s test our filter during searching. 

Using our custom filter during searching 
To test that our filter is working, we created a simple TestSpecialsAccessor to return a specified set of ISBNs, giving our test case control over the set of specials: 

view plaincopy to clipboardprint?

1. public class TestSpecialsAccessor implements SpecialsAccessor {  
2.     private String[] isbns;  
3.   
4.     public TestSpecialsAccessor(String[] isbns) {  
5.         this.isbns = isbns;  
6.     }  
7.   
8.     public String[] isbns() {  
9.         return isbns;  
10.     }  
11. }  

Here’s how we test our SpecialsFilter, using the same setUp() that the other filter tests used: 

view plaincopy to clipboardprint?

1. public void testCustomFilter() throws Exception {  
2.     Query allBooks = new TermQuery(new Term("contents", "manning"));  
3.     String[] isbns = new String[] { "1933988940", "9781935182023" };  
4.     SpecialsAccessor accessor = new TestSpecialsAccessor(isbns);  
5.     Filter filter = new SpecialsFilter(accessor);  
6.     TopDocs hits = searcher.search(allBooks, filter, 10);  
7.     assertEquals("the specials", isbns.length, hits.totalHits);  
8. }  

Note that we made an important implementation decision not to cache the DocIdSet in SpecialsFilter. Decorating SpecialsFilter with a CachingWrapperFilter frees us from that aspect. Let’s see an alternative means of applying a filter during searching. 

An alternative: FilteredQuery 
To add to the filter terminology overload, one final option is FilteredQuery. FilteredQuery inverts the situation that searching with a filter presents. Using a filter, anIndexSearcher’s search method applies a single filter during querying. Using the FilteredQuery, though, you can turn any filter into a query, which opens up neat possibilities, such as adding a filter as a clause to a BooleanQuery. 

Let’s take the SpecialsFilter as an example again. This time, we want a more sophisticated query: books in an education category on special, or books on Logo. We couldn’t accomplish this with a direct query using the techniques shown thus far, but FilteredQuery makes this possible. Had our search been only for books in the education category on special, we could’ve used the technique shown in the previous code snippet instead. 

Our test case, in listing 6.15, demonstrates the described query using a BooleanQuery with a nested TermQuery and FilteredQuery. 

Listing 6.15 Using a FilteredQuery 

view plaincopy to clipboardprint?

1. public void testFilteredQuery() throws Exception {  
2.     // 1)  
3.     String[] isbns = new String[] { "9781935182023" };        
4.       
5.     // 2)  
6.     SpecialsAccessor accessor = new TestSpecialsAccessor(isbns);          
7.     Filter filter = new SpecialsFilter(accessor);  
8.     WildcardQuery educationBooks = new WildcardQuery(new Term("category", "*education*"));  
9.     FilteredQuery edBooksOnSpecial = new FilteredQuery(educationBooks, filter);  
10.       
11.     // 3)  
12.     TermQuery logoBooks = new TermQuery(new Term("subject", "logo"));  
13.       
14.     // 4)  
15.     BooleanQuery logoOrEdBooks = new BooleanQuery();  
16.     logoOrEdBooks.add(logoBooks, BooleanClause.Occur.SHOULD);  
17.     logoOrEdBooks.add(edBooksOnSpecial, BooleanClause.Occur.SHOULD);  
18.     TopDocs hits = searcher.search(logoOrEdBooks, 10);  
19.     System.out.println(logoOrEdBooks.toString());  
20.     assertEquals("Papert and Steiner", 2, hits.totalHits);  
21. }  

1) This is the ISBN number for filtering
2) We construct a query for education books on special.
3) We construct a query for all books with logo in the subject.
4) The two queries are combined in an OR fashion.

The getDocIdSet() method of the nested Filter is called each time a FilteredQuery is used in a search, so we recommend that you use a caching filter if the query is to be used repeatedly and the results of a filter don’t change. 

Filtering is a powerful means of overriding which documents a query may match, and in this section you’ve seen how to create custom filters and use them during searching, as well as how to wrap a filter as a query so that it may be used wherever a query may be used. Filters give you a lot of flexibility for advanced searching.