Multiple languages, boosting and, stemming and KeywordRepeat
First, apologies for the weird subject line, and apologies for cross-posting, but last week it got no replies on the Solr user mailing list.
We index many languages and search over all those languages at once, but boost the language of the user's preference. To differentiate between stemmed tokens and unstemmed tokens we use KeywordRepeat and RemoveDuplicates, this works very well.
However, we just stumbled over the following example, q=australia is not stemmed in English, but its suffix is removed by the Romanian stemmer, causing the Romanian results to be returned on top of English results, despite language boosting.
This is because the Romanian part of the query consists of the stemmed and unstemmed version of the word, but the English part of the query is just one clause per field (title, content etc). Thus the Romanian results score roughtly twice that of English results.
Now, this is of course really obvious, but the 'solution' is not. To work around the problem i removed the RemoveDuplicates filter so i get two clauses for English as well, really ugly but it works. What i don't understand is the debug output, it doesn't list two identical clauses, instead, it doubled the boost on the field, so instead of: