程式扎記: Lucene

顯示具有 Lucene 標籤的文章。顯示所有文章

2014年7月23日星期三

[ InAction Note ] Ch6. Extending search - Custom filters

Preface:
If all the information needed to perform filtering is in the index, there’s no need to write your own filter because the QueryWrapperFilter can handle it, as described insection 5.6.5. But there are good reasons to factor external information into a custom filter. In this section we tackle the following example: using our book example data and pretending we’re running an online bookstore, we want users to be able to search within our special hot deals of the day.

You might be tempted to simply store the specials flag as an indexed field, but keeping this up-to-date might prove too costly. Rather than reindex entire documents when specials change, we’ll implement a custom filter that keeps the specials flagged in our (hypothetical) relational database. Then we’ll see how to apply our filter during searching, and finally we’ll explore an alternative option for applying the filter.

Implementing a custom filter
We start with abstracting away the source of our specials by defining this interface:

view plaincopy to clipboardprint?
package demo.ch6;  
  
public interface SpecialsAccessor {  
    String[] isbns();  
}  

The isbns() method returns those books that are currently specials. Because we won’t have an enormous amount of specials at one time, returning all the ISBNs of the books on special will suffice. Now that we have a retrieval interface, we can create our custom filter, SpecialsFilter. Filters extend from theorg.apache.lucene.search.Filter class and must implement the getDocIdSet(IndexReader reader) method, returning a DocIdSet. Bit positions match the document numbers. Enabled bits mean the document for that position is available to be searched against the query, and unset bits mean the document won’t be considered in the search. Figure 6.2 illustrates an example SpecialsFilter that sets bits for books on special (see listing 6.14).

- Listing 6.14 Retrieving filter information from external source with SpecialsFilter

view plaincopy to clipboardprint?
import java.io.IOException;  
  
import org.apache.lucene.index.AtomicReader;  
import org.apache.lucene.index.AtomicReaderContext;  
import org.apache.lucene.index.DocsEnum;  
import org.apache.lucene.index.Term;  
import org.apache.lucene.search.DocIdSet;  
import org.apache.lucene.search.DocIdSetIterator;  
import org.apache.lucene.search.Filter;  
import org.apache.lucene.util.Bits;  
import org.apache.lucene.util.OpenBitSet;  
  
public class SpecialsFilter extends Filter {  
    private SpecialsAccessor accessor;  
  
    public SpecialsFilter(SpecialsAccessor accessor) {  
        this.accessor = accessor;  
    }  
  
    @Override  
    public DocIdSet getDocIdSet(AtomicReaderContext ctx, Bits bits)  
            throws IOException {  
        AtomicReader reader = ctx.reader();  
        OpenBitSet oBits = new OpenBitSet(reader.maxDoc());  
        String[] isbns = accessor.isbns();  
        for (String isbn : isbns)   
        {  
            DocsEnum docEnum = reader.termDocsEnum(new Term("isbn", isbn));  
            while(docEnum.nextDoc()!= DocIdSetIterator.NO_MORE_DOCS)  
            {  
                if(docEnum.freq()>0)  
                {  
                    oBits.set(docEnum.docID());  
                }  
            }  
        }  
        return oBits;  
    }  
}  

The filter is quite straightforward. First we fetch the ISBNs of the current specials. Next, we interact with the AtomicReader API to iterate over all documents matching each ISBN; in each case it should be a single document per ISBN because this is a unique field. The document was indexed with Field.Index.NOT_ANALYZED, so we can retrieve it directly with the ISBN. Finally, we record each matching document in an OpenBitSet, which we return to Lucene. Let’s test our filter during searching.

Using our custom filter during searching
To test that our filter is working, we created a simple TestSpecialsAccessor to return a specified set of ISBNs, giving our test case control over the set of specials:

view plaincopy to clipboardprint?
public class TestSpecialsAccessor implements SpecialsAccessor {  
    private String[] isbns;  
  
    public TestSpecialsAccessor(String[] isbns) {  
        this.isbns = isbns;  
    }  
  
    public String[] isbns() {  
        return isbns;  
    }  
}  

Here’s how we test our SpecialsFilter, using the same setUp() that the other filter tests used:

view plaincopy to clipboardprint?
public void testCustomFilter() throws Exception {  
    Query allBooks = new TermQuery(new Term("contents", "manning"));  
    String[] isbns = new String[] { "1933988940", "9781935182023" };  
    SpecialsAccessor accessor = new TestSpecialsAccessor(isbns);  
    Filter filter = new SpecialsFilter(accessor);  
    TopDocs hits = searcher.search(allBooks, filter, 10);  
    assertEquals("the specials", isbns.length, hits.totalHits);  
}  

Note that we made an important implementation decision not to cache the DocIdSet in SpecialsFilter. Decorating SpecialsFilter with a CachingWrapperFilter frees us from that aspect. Let’s see an alternative means of applying a filter during searching.

An alternative: FilteredQuery
To add to the filter terminology overload, one final option is FilteredQuery. FilteredQuery inverts the situation that searching with a filter presents. Using a filter, anIndexSearcher’s search method applies a single filter during querying. Using the FilteredQuery, though, you can turn any filter into a query, which opens up neat possibilities, such as adding a filter as a clause to a BooleanQuery.

Let’s take the SpecialsFilter as an example again. This time, we want a more sophisticated query: books in an education category on special, or books on Logo. We couldn’t accomplish this with a direct query using the techniques shown thus far, but FilteredQuery makes this possible. Had our search been only for books in the education category on special, we could’ve used the technique shown in the previous code snippet instead.

Our test case, in listing 6.15, demonstrates the described query using a BooleanQuery with a nested TermQuery and FilteredQuery.

Listing 6.15 Using a FilteredQuery

view plaincopy to clipboardprint?
public void testFilteredQuery() throws Exception {  
    // 1)  
    String[] isbns = new String[] { "9781935182023" };        
      
    // 2)  
    SpecialsAccessor accessor = new TestSpecialsAccessor(isbns);          
    Filter filter = new SpecialsFilter(accessor);  
    WildcardQuery educationBooks = new WildcardQuery(new Term("category", "*education*"));  
    FilteredQuery edBooksOnSpecial = new FilteredQuery(educationBooks, filter);  
      
    // 3)  
    TermQuery logoBooks = new TermQuery(new Term("subject", "logo"));  
      
    // 4)  
    BooleanQuery logoOrEdBooks = new BooleanQuery();  
    logoOrEdBooks.add(logoBooks, BooleanClause.Occur.SHOULD);  
    logoOrEdBooks.add(edBooksOnSpecial, BooleanClause.Occur.SHOULD);  
    TopDocs hits = searcher.search(logoOrEdBooks, 10);  
    System.out.println(logoOrEdBooks.toString());  
    assertEquals("Papert and Steiner", 2, hits.totalHits);  
}  

1) This is the ISBN number for filtering
2) We construct a query for education books on special.
3) We construct a query for all books with logo in the subject.
4) The two queries are combined in an OR fashion.

The getDocIdSet() method of the nested Filter is called each time a FilteredQuery is used in a search, so we recommend that you use a caching filter if the query is to be used repeatedly and the results of a filter don’t change.

Filtering is a powerful means of overriding which documents a query may match, and in this section you’ve seen how to create custom filters and use them during searching, as well as how to wrap a filter as a query so that it may be used wherever a query may be used. Filters give you a lot of flexibility for advanced searching.

2014年7月16日星期三

[ InAction Note ] Ch6. Extending search - Extending QueryParser

Preface:
In section 3.5, we introduced QueryParser and showed that it has a few settings to control its behavior, such as setting the locale for date parsing and controlling the default phrase slop. QueryParser is also extensible, allowing subclassing to override parts of the query-creation process. In this section, we demonstrate subclassingQueryParser to disallow inefficient wildcard and fuzzy queries, custom date-range handling, and morphing phrase queries into SpanNearQuerys instead ofPhraseQuerys.

Customizing QueryParser’s behavior
Although QueryParser has some quirks, such as the interactions with an analyzer, it does have extensibility points that allow for customization. Table 6.2 details the methods designed for overriding and why you may want to do so.

All of the methods listed return a Query, making it possible to construct something other than the current subclass type used by the original implementations of these methods. Also, each of these methods may throw a ParseException, allowing for error handling.

QueryParser also has extensibility points for instantiating each query type. These differ from the points listed in table 6.2 in that they create the requested query type and return it. Overriding them is useful if you only want to change which Query class is used for each type of query without altering the logic of what query is constructed. These methods are newBooleanQuery, newTermQuery, newPhraseQuery, newMultiPhraseQuery, newPrefixQuery, newFuzzyQuery, newRangeQuery,newMatchAllDocsQuery and newWildcardQuery. For example, if whenever a TermQuery is created by QueryParser you’d like to instantiate your own subclass ofTermQuery, simply override newTermQuery.

Prohibiting fuzzy and wildcard queries
The subclass in listing 6.7 demonstrates a custom query parser subclass that disables fuzzy and wildcard queries by taking advantage of the ParseException option.
- Listing 6.7 Disallowing wildcard and fuzzy queries

view plaincopy to clipboardprint?
package demo.ch6;  
  
import org.apache.lucene.analysis.Analyzer;  
import org.apache.lucene.queryparser.classic.ParseException;  
import org.apache.lucene.queryparser.classic.QueryParser;  
import org.apache.lucene.search.Query;  
import org.apache.lucene.util.Version;  
  
public class CustomQueryParser extends QueryParser {  
    public CustomQueryParser(Version matchVersion, String field, Analyzer analyzer) {  
        super(matchVersion, field, analyzer);  
    }  
  
    @Override  
    protected final Query getWildcardQuery(String field, String termStr) throws ParseException {  
        throw new ParseException("Wildcard not allowed");  
    }  
  
    @Override  
    protected Query getFuzzyQuery(String field, String term, float minSimilarity) throws ParseException {  
        throw new ParseException("Fuzzy queries not allowed");  
    }  
}  

To use this custom parser and prevent users from executing wildcard and fuzzy queries, construct an instance of CustomQueryParser and use it exactly as you wouldQueryParser, as shown in listing 6.8.
- Listing 6.8 Using a custom QueryParser

view plaincopy to clipboardprint?
public void testCustomQueryParser() {  
    CustomQueryParser parser = new CustomQueryParser(VER, "field", analyzer);  
    try {  
        parser.parse("a?t");  
        fail("Wildcard queries should not be allowed");  
    } catch (ParseException expected) {  
    }  
    try {  
        parser.parse("xunit~");  
        fail("Fuzzy queries should not be allowed");  
    } catch (ParseException expected) {  
    }  
}  

With this implementation, both of these expensive query types are forbidden, giving you peace of mind in terms of performance and errors that may arise from these queries expanding into too many terms. Our next QueryParser extension enables creation of NumericRangeQuery.

Handling numeric field-range queries
As you learned in chapter 2, Lucene can handily index numeric and date values. Unfortunately, QueryParser is unable to produce the corresponding NumericRangeQueryinstances at search time. Fortunately, it’s simple to subclass QueryParser to do so, as shown in listing 6.9.
- Listing 6.9 Extending QueryParser to properly handle numeric fields

view plaincopy to clipboardprint?
package demo.ch6;  
  
import org.apache.lucene.analysis.Analyzer;  
import org.apache.lucene.queryparser.classic.ParseException;  
import org.apache.lucene.queryparser.classic.QueryParser;  
import org.apache.lucene.search.NumericRangeQuery;  
import org.apache.lucene.search.Query;  
import org.apache.lucene.search.TermRangeQuery;  
import org.apache.lucene.util.Version;  
  
public class NumericRangeQueryParser extends QueryParser {  
    public NumericRangeQueryParser(Version matchVersion, String field, Analyzer a) {  
        super(matchVersion, field, a);  
    }  
  
    @Override  
    public Query getRangeQuery(String field, String part1, String part2, boolean sInclusive, boolean eInclusive) throws ParseException {  
        TermRangeQuery query = (TermRangeQuery) super.getRangeQuery(field, part1, part2, sInclusive, eInclusive);  
        if ("price".equals(field))   
        {  
            return NumericRangeQuery.newDoubleRange("price",  
                    Double.parseDouble(query.getLowerTerm().utf8ToString()),  
                    Double.parseDouble(query.getUpperTerm().utf8ToString()),  
                    query.includesLower(), query.includesUpper());  
        }   
        else   
        {  
            return query;  
        }  
    }  
}  

Using this approach, you rely on QueryParser to first create the TermRangeQuery, and from that you construct the NumericRangeQuery as needed. Testing ourNumericQueryParser, like this:

view plaincopy to clipboardprint?
public void testNumericRangeQuery() throws Exception {  
    String expression = "price:[50 TO 90]";  
    QueryParser parser = new NumericRangeQueryParser(VER, "subject", analyzer);  
    Query query = parser.parse(expression);  
    System.out.println(expression + " parsed to " + query);  
}  

yields the expected output (note that the 50 and 90 have been turned into floating point values):

price:[50 TO 90] parsed to price:[50.0 TO 90.0]

As you’ve seen, extending QueryParser to handle numeric fields was straightforward. Let’s do the same for date fields next.

Handling date ranges
QueryParser has built-in logic to detect date ranges: if the terms are valid dates, according to DateFormat.SHORT and lenient parsing within the default or specified locale, the dates are converted to their internal textual representation. By default, this conversion will use the older DateField.dateToString method, which renders each date with millisecond precision; this is likely not what you want. If you invoke QueryParser’s setDateResolution methods to state which DateTools.Resolution your field(s) were indexed with, then QueryParser will use the newer DateTools.dateToString method to translate the dates into strings with the appropriate resolution. If either term fails to parse as a valid date, they’re both used as is for a textual range.

But despite these two built-in approaches for handling dates, QueryParsers’s date handling hasn’t been updated to handle date fields indexed as NumericField, which is the recommended approach for dates, as described in section 2.6.2. Let’s see how we can once again override newRangeQuery, this time to translate our date-based range searches into the corresponding NumericRangeQuery, shown in listing 6.10.
- Listing 6.10 Extending QueryParser to handle date fields

view plaincopy to clipboardprint?
package demo.ch6;  
  
import org.apache.lucene.analysis.Analyzer;  
import org.apache.lucene.queryparser.classic.ParseException;  
import org.apache.lucene.queryparser.classic.QueryParser;  
import org.apache.lucene.search.NumericRangeQuery;  
import org.apache.lucene.search.Query;  
import org.apache.lucene.search.TermRangeQuery;  
import org.apache.lucene.util.Version;  
  
public class NumericDateRangeQueryParser extends QueryParser {  
    public NumericDateRangeQueryParser(Version matchVersion, String field, Analyzer a) {  
        super(matchVersion, field, a);  
    }  
  
    @Override  
    public Query getRangeQuery(String field, String part1, String part2, boolean sInc, boolean eInc) throws ParseException {  
        TermRangeQuery query = (TermRangeQuery) super.getRangeQuery(field, part1, part2, sInc, eInc);  
        if ("pubmonth".equals(field))   
        {  
            DateFormat dateformat = new SimpleDateFormat("yyyyMM");  
            Resolution res = this.getDateResolution("pubmonth");  
            System.out.printf("\t[Test] Resolution=%s\n", res);  
            System.out.printf("\t[Test] Lower Term: %s\n", query.getLowerTerm().utf8ToString());  
            System.out.printf("\t[Test] Upper Term: %s\n", query.getUpperTerm().utf8ToString());  
            try  
            {  
                return NumericRangeQuery.newLongRange("pubmonth",  
                        dateformat.parse(query.getLowerTerm().utf8ToString()).getTime(),  
                        dateformat.parse(query.getUpperTerm().utf8ToString()).getTime(),  
                        query.includesLower(), query.includesUpper());  
            }  
            catch(Exception e){throw new ParseException("");}  
        } else {  
            return query;  
        }  
    }  
}  

In this case it’s still helpful to use QueryParser’s built-in logic for detecting and parsing dates. You simply build on that logic in your subclass by taking the further step to convert the query into a NumericRangeQuery. Note that in order to use this subclass you must call QueryParser.setDateResolution, so that the resulting text terms are created with DateTools, as shown in listing 6.11.

view plaincopy to clipboardprint?
public void testDateRangeQuery() throws Exception {  
    String expression = "pubmonth:[01/01/2010 TO 06/01/2010]";  
    QueryParser parser = new NumericDateRangeQueryParser(VER, "subject", analyzer);  
    parser.setDateResolution("pubmonth", DateTools.Resolution.MONTH);  
    parser.setLocale(Locale.US);  
    Query query = parser.parse(expression);  
    System.out.println(expression + " parsed to " + query);  
    TopDocs matches = searcher.search(query, 10);  
    assertTrue("expecting at least one result !", matches.totalHits > 0);  
}  

This test produces the following output:

pubmonth:[01/01/2010 TO 06/01/2010] parsed to pubmonth:[1259596800000 TO 1275321600000]

CONTROLLING THE DATE-PARSING LOCALE
To change the locale used for date parsing, construct a QueryParser'instance and call setLocale(). Typically the client’s locale would be determined and used instead of the default locale. For example, in a web application the HttpServletRequest object contains the locale set by the client browser. You can use this locale to control the locale used by date parsing in QueryParser, as shown in listing 6.12.
- Listing 6.12 Using the client locale in a web application

view plaincopy to clipboardprint?
public class SearchServletFragment extends HttpServlet {  
  protected void doGet(HttpServletRequest request,  
                       HttpServletResponse response)   
      throws ServletException, IOException {  
    QueryParser parser = new NumericDateRangeQueryParser(  
                             Version.LUCENE_30,  
                             "contents",  
                             new StandardAnalyzer(Version.LUCENE_30));  
    parser.setLocale(request.getLocale());  
    parser.setDateResolution(DateTools.Resolution.DAY);  
    Query query = null;  
    try {  
      query = parser.parse(request.getParameter("q"));  
    } catch (ParseException e) {  
      e.printStackTrace(System.err);  
    }  
    TopDocs docs = searcher.search(query, 10);  
  }  
}  

QueryParser’s setLocale is one way in which Lucene facilitates internationalization (often abbreviated as I18N) concerns. Text analysis is another, more important, place where such concerns are handled. Further I18N issues are discussed in section 4.8. Our final QueryParser customization shows how to replace the default PhraseQuerywith SpanNearQuery.

Allowing ordered phrase queries
When QueryParser parses a single term, or terms within double quotes, it delegates the construction of the Query to a getFieldQuery method. Parsing an unquoted term calls the getFieldQuery method without the slop signature (slop makes sense only on multiterm phrase query); parsing a quoted phrase calls the getFieldQuery signature with the slop factor, which internally delegates to the nonslop signature to build the query and then sets the slop appropriately. The Query returned is either aTermQuery or a PhraseQuery, by default, depending on whether one or more tokens are returned from the analyzer. Given enough slop, PhraseQuery will match terms out of order in the original text. There’s no way to force a PhraseQuery to match in order (except with slop of 0 or 1). However, SpanNearQuery does allow in-order matching. A straightforward override of getFieldQuery allows us to replace a PhraseQuery with an ordered SpanNearQuery, shown in listing 6.13.
- Listing 6.13 Translating PhraseQuery to SpanNearQuery

view plaincopy to clipboardprint?
protected Query getFieldQuery(String field, String queryText, int slop) throws ParseException {  
    Query orig = super.getFieldQuery(field, queryText, slop); // 1)  
    if (!(orig instanceof PhraseQuery)) {  
        return orig;  // 2)  
    }  
    PhraseQuery pq = (PhraseQuery) orig;  
    Term[] terms = pq.getTerms();  // 3)  
    SpanTermQuery[] clauses = new SpanTermQuery[terms.length];  
    for (int i = 0; i < terms.length; i++) {  
        clauses[i] = new SpanTermQuery(terms[i]);  
    }  
    SpanNearQuery query = new SpanNearQuery(clauses, slop, true); // 4)  
    return query;  
}  

1) We delegate to QueryParser's implementation for analysis and determination of query type.
2) We override PhraseQuery and return anything else right away.
3) We pull all terms from the original PhraseQuery.
4) We create a SpanNearQuery with all the terms from the original PhraseQuery.

Our test case shows that our custom getFieldQuery is effective in creating a SpanNearQuery:

view plaincopy to clipboardprint?
public void testPhraseQuery() throws Exception {  
    CQPWithSpanQuery parser = new CQPWithSpanQuery(VER, "contents", analyzer);  
    Query query = parser.parse("singleTerm");  
    assertTrue("TermQuery", query instanceof TermQuery);  
    query = parser.parse("\"phrase test\"");  
    System.out.printf("\t[Test] PhraseQuery: %s\n", query.getClass().getName());  
    assertTrue("SpanNearQuery", query instanceof SpanNearQuery);  
}  

Another possible enhancement would be to add a toggle switch to the custom query parser, allowing the in-order flag to be controlled by the user of the API.

Supplement:
* Ch5. Advanced search techniques - Span queries (1)

訂閱：文章 (Atom)

程式扎記

標籤

2014年7月23日星期三

[ InAction Note ] Ch6. Extending search - Custom filters

2014年7月16日星期三

[ InAction Note ] Ch6. Extending search - Extending QueryParser

[Git 常見問題] error: The following untracked working tree files would be overwritten by merge

檢舉濫用情形

學習筆記

標籤

2014年7月23日 星期三

[ InAction Note ] Ch6. Extending search - Custom filters

2014年7月16日 星期三

[ InAction Note ] Ch6. Extending search - Extending QueryParser

[Git 常見問題] error: The following untracked working tree files would be overwritten by merge

檢舉濫用情形

學習筆記

2014年7月23日星期三

2014年7月16日星期三