One important question: will the typical document have a mix of English and Bangla and Hindi? If so, you would probably have them all in one collection.
Another thing to think about is the tokenizer. Are all words separated by white space? If not, then you might need to think about which tokenizer to use.
As for character sets, I think you should make sure all the inputs are in UTF-8, then there should be no problem.
There will be other things to consider but this is a start.
Cheers -- Rick
On September 10, 2017 9:32:11 AM EDT, Mugeesh Husain <[hidden email]> wrote:
>I am working on multi language search engine for english,bangla, hindi
>indonesia language. can anybody guide me how to configure solr
>1.) should i need to configure all the language in a single
>2.)should I need to configure separate shard/collection for each of
>I am looking for the suggestion about architecture level of this
>Please suggest and guide me to defining the schema and architecture.
>Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Sorry for being brief. Alternate email is rickleir at yahoo dot com
another thing to consider is what users would expect, would english user
search over english docs only ? if yes, the most important task would be to
correctly set up / create accurate per language analyzers, otherwise you
may consider to also adopt machine translation, either on the search
queries or on the resulting docs, me and a friend of mine gave a talk at
bbuzz this year .
Il giorno lun 11 set 2017 alle ore 03:46 Mugeesh Husain <[hidden email]>
> Thank you rick for your response.
> The document document have sepearte of the lanaguage instead of mix of
> Arabic, English, Bengali, Hindi, Malay.
> I coul not find any tokenizer for Malay, can you suggest me if you know
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
Having the language already separated makes it a lot easier.
You could add the language suffix (e.g. 3 letter with ISO 639-2B https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) per field where you have the different languages. Or else you could have copied an entire field to their language-analyzed fields, and hope that would be good enough for matching.