How to disable default scoring/boosting in Hibernate Search/Lucene?

Question

I want to serve my users the most relevant and best results. For example, I'm rewarding records that have a big title, description, attached photos, etc. For context: the records are bicycle routes, having routepoints (coordinates) and metadata like photos, reviews, etc.

Now, I have indexed these records using Hibernate and then I search within the index using Lucene in Hibernate Search. To score my results, I build queries based on the document properties and boost them (using boostedTo()) in a should BooleanJunction clause:

bj.should(qb.range().onField("descriptionLength").above(3000).createQuery()).boostedTo(3.0f);   
bj.should(qb.range().onField("views.views").above(5000).createQuery()).boostedTo(3.0f);     
bj.should(qb.range().onField("nameLength").above(20).createQuery()).boostedTo(1.0f);     
bj.should(qb.range().onField("picturesLength").above(0).createQuery()).boostedTo(5.0f);
bj.should(qb.keyword().onField("routePoints.poi.participant").matching("true").createQuery()).boostedTo(10.0f);

To try and disable Lucene's scoring, I have overridden the DefaultSimilarity class, set all the comparing to 1.0f score and enabled it via Hibernate config:

public class IgnoreScoringSimilarity extends DefaultSimilarity {
    @Override
    public float idf(long docFreq, long numDocs) {
        return 1.0f;
    }

    @Override
    public float tf(float freq) {
        return 1.0f;
    }

    @Override
    public float coord(int overlap, int maxOverlap) {
        return 1.0f;
    }

    @Override
    public float lengthNorm(FieldInvertState state) {
        return 1.0f;
    }

    @Override
    public float queryNorm(float sumOfSquaredWeights) {
        return 1.0f;
    }
} 

Hibernate config:

<property name="hibernate.search.default.similarity" value="com.search.IgnoreScoringSimilarity"/>

This approach works for 90% of the time, however, I am still seeing some weird results that seem to be out of place. The pattern I recognize is that these routes (documents) are very large in size. A normal route has about 20-30 routepoints, however these out-of-place results have 100-150. This leaves me to believe that default Lucene scoring is still happening (scoring higher because of document size).

Am I doing something wrong in disabling Lucene's scoring? Could there be another explanation?

Empanel

Not an answer, but a consideration: I would not disable Lucene's default scoring, but would work on the indexing phase. I'd build a custom Indexer for your documents that sets a (reduced) boost for big documents, instead; you can call document.setBoost() on the indexer to set a custom value, based on the number of routepoints, and check the results. Something like setBoost(100/routepoints_count), or some exponential function. – Berkey

Thanks for your comment! But wouldn't that still give (an albeit small) boost to document size by factoring in the routepoints count? That's what I don't want, because for our scoring system, it doesn't matter if a route has 2 or 200 routepoints, it should only be scored by it's metadata. – Empanel

Yes, that would, but since you are already boosting documents with big factors I don't think that would matter that much. Do you really need to index the routepoints too? Could you add a snippet of your indexer to understand the content of the index? – Berkey

Answer 1

I can suggest another approach based on custom result sorting. You can read about it in the answer. This answer is a slightly outdated, so I modified this example according to Lucene API 4.10.1. Comparator

public abstract class CustomComparator extends FieldComparator<Double> {
    double[] scoring;
    double bottom;
    double topValue;
    private FieldCache.Ints[] currentReaderValues;
    private String[] fields;

    protected abstract double getScore(int[] value);

    public CustomComparator(int hitNum, String[] fields) {
        this.fields = fields;
        scoring = new double[hitNum];
    }

    int[] fromReaders(int doc) {
        int[] result = new int[currentReaderValues.length];
        for (int i = 0; i < result.length; i++) {
            result[i] = currentReaderValues[i].get(doc);
        }
        return result;
    }

    @Override
    public int compare(int slot1, int slot2) {
        return Double.compare(scoring[slot1], scoring[slot2]);
    }

    @Override
    public void setBottom(int slot) {
        this.bottom = scoring[slot];
    }

    @Override
    public void setTopValue(Double top) {
        topValue = top;
    }

    @Override
    public int compareBottom(int doc) throws IOException {
        double v2 = getScore(fromReaders(doc));
        return Double.compare(bottom, v2);
    }

    @Override
    public int compareTop(int doc) throws IOException {
        double docValue = getScore(fromReaders(doc));
        return Double.compare(topValue, docValue);
    }

    @Override
    public void copy(int slot, int doc) throws IOException {
        scoring[slot] = getScore(fromReaders(doc));
    }

    @Override
    public FieldComparator<Double> setNextReader(AtomicReaderContext atomicReaderContext) throws IOException {
        currentReaderValues = new FieldCache.Ints[fields.length];
        for (int i = 0; i < fields.length; i++) {
            currentReaderValues[i] = FieldCache.DEFAULT.getInts(atomicReaderContext.reader(), fields[i], null, false);
        }
        return this;
    }

    @Override
    public Double value(int slot) {
        return scoring[slot];
    }
}

Example of search

public class SortExample {

    public static void main(String[] args) throws IOException {

        final String[] fields = new String[]{"descriptionLength", "views.views", "nameLength"};

        Sort sort = new Sort(
                new SortField(
                        "",
                        new FieldComparatorSource() {
                            public FieldComparator newComparator(String fieldname, int numHits, int sortPos, boolean reversed) throws IOException {
                                return new CustomComparator(numHits, fields) {
                                    @Override
                                    protected double getScore(int[] value) {
                                        int descriptionLength = value[0];
                                        int views = value[1];
                                        int nameLength = value[2];
                                        return -((descriptionLength > 2000.0 ? 5.0 : 0.0) +
                                                (views > 5000.0 ? 3.0 : 0.0) +
                                                (nameLength > 20.0 ? 1.0 : 0.0));
                                    }
                                };
                            }
                        }
                )
        );

        IndexWriterConfig indexWriterConfig = new IndexWriterConfig(Version.LUCENE_4_10_4, new StandardAnalyzer());
        Directory directory = new RAMDirectory();
        IndexWriter indexWriter = new IndexWriter(directory, indexWriterConfig);

        addDoc(indexWriter, "score 0", 1000, 1000, 10);
        addDoc(indexWriter, "score 5", 3000, 1000, 10);
        addDoc(indexWriter, "score 3", 1000, 6000, 10);
        addDoc(indexWriter, "score 1", 1000, 1000, 30);
        addDoc(indexWriter, "score 4", 1000, 6000, 30);
        addDoc(indexWriter, "score 6", 5000, 1000, 30);
        addDoc(indexWriter, "score 9", 5000, 6000, 30);

        final IndexReader indexReader = DirectoryReader.open(indexWriter, false);
        IndexSearcher indexSearcher = new IndexSearcher(indexReader);
        Query query = new TermQuery(new Term("all", "all"));
        int nDocs = 100;

        final TopDocs search = indexSearcher.search(query, null, nDocs, sort);
        System.out.println("Max " + search.scoreDocs.length + " " + search.getMaxScore());
        for (ScoreDoc sd : search.scoreDocs) {
            Document document = indexReader.document(sd.doc);
            System.out.println(document.getField("name").stringValue());
        }

    }

    private static void addDoc(IndexWriter indexWriter, String name, int descriptionLength, int views, int nameLength) throws IOException {
        Document doc = new Document();
        doc.add(new TextField("name", name, Field.Store.YES));
        doc.add(new TextField("all", "all", Field.Store.YES));
        doc.add(new IntField("descriptionLength", descriptionLength, Field.Store.YES));
        doc.add(new IntField("views.views", views, Field.Store.YES));
        doc.add(new IntField("nameLength", nameLength, Field.Store.YES));
        indexWriter.addDocument(doc);
    }
}

Code will output

score 9
score 6
score 5
score 4
score 3
score 1
score 0

Quadriceps

Thanks for the suggestion! This seems an interesting approach. However, I'm not sure if this would work for the way I want to boost. It seems the answer you linked only boosts the fields themselves? That way I cannot do things like 'boost routes with isRoundtrip = false' by x or 'boost routes with title length > 50 by x'. Please correct me if I'm wrong though! – Empanel

Instead of boost query you can change ranking formula in scoring. In example you have field1 * 0.5 + field2 * 1.4 + field3 * 1.8, you can change it to titleLength > 50.0 ? 3.0 : 0.0 + nameLength > 0.0 ? 1.0 : 0.0 and so on. In this approach you have even more control on ranking, because you can have any scoring formula. – Quadriceps

Interesting, I will experiment a bit with this approach, thanks for the suggestion, +1. – Empanel

Hope you will success with it. – Quadriceps

I'm trying to implement your suggestion, however I'm running into some compatibility issues. I'm using Hibernate Search 5.2, so I guess this uses Lucene 4.7?. What do I return in the setNextReader method, do I return this? What do I need to do in the mandatory compareTop and setTopValue methods? (need to override those to extend FieldComparator. Also wondering how I can use NumericDocValues, do I need to use it instead of the int[][] currentReaderValues? Having a hard time understanding the concept of those new properties I need to use. – Empanel

I modified example from reference question, according your version of Lucene (it 4.10.1 from dependencies list mvnrepository.com/artifact/org.hibernate/…). I think this should help you. – Quadriceps

Thanks alot! I will mark this answer as the correct one, as it simultaneously answers the question of how to disable the default sorting, and how to make a custom scoring algorithm sorting. – Empanel