Howto search for § character

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Howto search for § character

Bernd Schmidt

Hi all,


we have defined a field named "_text_" for a full text search based on field-type "text_general":
<field name="_text_" type="text_general" multiValued="true" indexed="true" stored="false"/>"


When trying to search for the "§" character, we have strange behaviour:


q=_text_:§ AND entityClass:StructureNodeImpl  => numFound:469 (all nodes where entityClass:StructureNodeImpl)
q=_text_:§ => numFound:0


How can we search for the occurence of the § character?


Best regards, 
    Bernd

 Mit freundlichen Grüßen

 Bernd Schmidt
 SOFTWARE-ENTWICKLUNG

 [hidden email]



 eggheads GmbH
 Herner Straße 370
44807 Bochum

Fon +49 234 89397-0
Fax +49 234 89397-28
 
 www.eggheads.de
 -----------------------------------------------


Kunden DER TOURISTIK, EMSA, FRIATEC, MAMMUT, SUTTERLÜTY, SCHÄFER SHOP, THOMAS COOK, TUI, WILO SE, WÜRTH, u.v.m.


Leistungen Standardsoftware für Product Information Management, Cross Media Publishing & Multi Channel Commerce, Prozessberatung


Innovationspreis 2017 eggheads ist Sieger beim Innovationspreis-IT 2017 in der Kategorie E-Commerce. Mehr

-----------------------------------------------

Webinar Vorstellung der neuen Funktionalität der eggheads Suite am 12.12.2017. Mehr

-----------------------------------------------
Reply | Threaded
Open this post in threaded view
|

Re: Howto search for § character

Rick Leir-2
Bernd,
What is the analysis chain you have in schema.xml? The chain tokenizes text and filters characters. There is an index time chain and a query time chain. My suspicion is that your analysis chain is mapping that char to a plain ascii char. Use the SolrAdmin analysis tab to debug this.
Cheers -- Rick

On December 6, 2017 11:09:14 AM EST, Bernd Schmidt <[hidden email]> wrote:

>
>Hi all,
>
>
>we have defined a field named "_text_" for a full text search based on
>field-type "text_general":
><field name="_text_" type="text_general" multiValued="true"
>indexed="true" stored="false"/>"
>
>
>When trying to search for the "§" character, we have strange behaviour:
>
>
>q=_text_:§ AND entityClass:StructureNodeImpl  => numFound:469 (all
>nodes where entityClass:StructureNodeImpl)
>q=_text_:§ => numFound:0
>
>
>How can we search for the occurence of the § character?
>
>
>Best regards, 
>    Bernd
>
> Mit freundlichen Grüßen
>
> Bernd Schmidt
> SOFTWARE-ENTWICKLUNG
>
> [hidden email]
>
>
>
> eggheads GmbH
> Herner Straße 370
>44807 Bochum
>
>Fon +49 234 89397-0
>Fax +49 234 89397-28
>
> www.eggheads.de
> -----------------------------------------------
>
>
>Kunden DER TOURISTIK, EMSA, FRIATEC, MAMMUT, SUTTERLÜTY, SCHÄFER SHOP,
>THOMAS COOK, TUI, WILO SE, WÜRTH, u.v.m.
>
>
>Leistungen Standardsoftware für Product Information Management, Cross
>Media Publishing & Multi Channel Commerce, Prozessberatung
>
>
>Innovationspreis 2017 eggheads ist Sieger beim Innovationspreis-IT 2017
>in der Kategorie E-Commerce. Mehr
>
>-----------------------------------------------
>
>Webinar Vorstellung der neuen Funktionalität der eggheads Suite am
>12.12.2017. Mehr
>
>-----------------------------------------------

--
Sorry for being brief. Alternate email is rickleir at yahoo dot com
Reply | Threaded
Open this post in threaded view
|

Re: Howto search for § character

Shawn Heisey-2
In reply to this post by Bernd Schmidt
On 12/6/2017 9:09 AM, Bernd Schmidt wrote:
> we have defined a field named "_text_" for a full text search based on field-type "text_general":
> <field name="_text_" type="text_general" multiValued="true" indexed="true" stored="false"/>"
>
> When trying to search for the "§" character, we have strange behaviour:
>
> q=_text_:§ AND entityClass:StructureNodeImpl  => numFound:469 (all nodes where entityClass:StructureNodeImpl)
> q=_text_:§ => numFound:0
>
> How can we search for the occurence of the § character?

We can't see how your "text_general" type is defined, but if it is
anything like the same type included in Solr examples, then it probably
is using StandardTokenizerFactory.  It appears that this tokenizer
treats the § character as a word break and removes it from the token
stream.  Most likely, the reason the search with the extra clause works
is that the part with that character is removed, and the query ends up
ONLY being the extra clause.

You will need a fieldType with an analysis chain that doesn't remove the
§ character, and it's almost guaranteed that you'll need to reindex. 
Unless you do that, searching for that character is not going to be
possible.

Also keep in mind that searching for a single character may not do what
you expect if that character is not a single word in the text, and that
certain filters can end up trimming out really short terms like that.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Howto search for § character

Erick Erickson
The admin UI/(select core)/analysis page will help you see exactly
what happens. Additionally, the "schema browser" bit will show you
exactly what's in the index, i.e. the terms as they actually appear
after all the analysis chain is completed. Those will definitively
tell you what exactly happens with that character.

Best,
Erick

On Thu, Dec 7, 2017 at 7:37 AM, Shawn Heisey <[hidden email]> wrote:

> On 12/6/2017 9:09 AM, Bernd Schmidt wrote:
>> we have defined a field named "_text_" for a full text search based on field-type "text_general":
>> <field name="_text_" type="text_general" multiValued="true" indexed="true" stored="false"/>"
>>
>> When trying to search for the "§" character, we have strange behaviour:
>>
>> q=_text_:§ AND entityClass:StructureNodeImpl  => numFound:469 (all nodes where entityClass:StructureNodeImpl)
>> q=_text_:§ => numFound:0
>>
>> How can we search for the occurence of the § character?
>
> We can't see how your "text_general" type is defined, but if it is
> anything like the same type included in Solr examples, then it probably
> is using StandardTokenizerFactory.  It appears that this tokenizer
> treats the § character as a word break and removes it from the token
> stream.  Most likely, the reason the search with the extra clause works
> is that the part with that character is removed, and the query ends up
> ONLY being the extra clause.
>
> You will need a fieldType with an analysis chain that doesn't remove the
> § character, and it's almost guaranteed that you'll need to reindex.
> Unless you do that, searching for that character is not going to be
> possible.
>
> Also keep in mind that searching for a single character may not do what
> you expect if that character is not a single word in the text, and that
> certain filters can end up trimming out really short terms like that.
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: Howto search for § character

Bernd Schmidt
In reply to this post by Shawn Heisey-2

Indeed, I saw in the analysis tab of the solr admin that the § char will be removed when using type text_general.
But in this use case we want to make a full text search like "_text_:§45" or "_text_:§*" to find words starting with §.
We need a text field here, not a string field!
What is your recommended way to deal with it? 
Is it possible to remove the word break behaviour for the  § char?
Or is the best way to encode all § chars when indexing and searching?



Thanks, Bernd



 Mit freundlichen Grüßen

 Bernd Schmidt
 SOFTWARE-ENTWICKLUNG

 [hidden email]



 Von:   Shawn Heisey <[hidden email]>
 An:   <[hidden email]>
 Gesendet:   07.12.2017 16:37
 Betreff:   Re: Howto search for § character

On 12/6/2017 9:09 AM, Bernd Schmidt wrote:
> we have defined a field named "_text_" for a full text search based on field-type "text_general":
> <field name="_text_" type="text_general" multiValued="true" indexed="true" stored="false"/>"
>
> When trying to search for the "§" character, we have strange behaviour:
>
> q=_text_:§ AND entityClass:StructureNodeImpl  => numFound:469 (all nodes where entityClass:StructureNodeImpl)
> q=_text_:§ => numFound:0
>
> How can we search for the occurence of the § character?
 
We can't see how your "text_general" type is defined, but if it is
anything like the same type included in Solr examples, then it probably
is using StandardTokenizerFactory.  It appears that this tokenizer
treats the § character as a word break and removes it from the token
stream.  Most likely, the reason the search with the extra clause works
is that the part with that character is removed, and the query ends up
ONLY being the extra clause.
 
You will need a fieldType with an analysis chain that doesn't remove the
§ character, and it's almost guaranteed that you'll need to reindex. 
Unless you do that, searching for that character is not going to be
possible.
 
Also keep in mind that searching for a single character may not do what
you expect if that character is not a single word in the text, and that
certain filters can end up trimming out really short terms like that.
 
Thanks,
Shawn
 




 eggheads GmbH
 Herner Straße 370
44807 Bochum

Fon +49 234 89397-0
Fax +49 234 89397-28
 
 www.eggheads.de
 -----------------------------------------------


Kunden DER TOURISTIK, EMSA, FRIATEC, MAMMUT, SUTTERLÜTY, SCHÄFER SHOP, THOMAS COOK, TUI, WILO SE, WÜRTH, u.v.m.


Leistungen Standardsoftware für Product Information Management, Cross Media Publishing & Multi Channel Commerce, Prozessberatung


Innovationspreis 2017 eggheads ist Sieger beim Innovationspreis-IT 2017 in der Kategorie E-Commerce. Mehr

-----------------------------------------------

Webinar Vorstellung der neuen Funktionalität der eggheads Suite am 12.12.2017. Mehr

-----------------------------------------------
Reply | Threaded
Open this post in threaded view
|

Re: Howto search for § character

Erick Erickson
You have to use a different analysis chain. There are about a zillion
options, here's a _start_:
https://lucene.apache.org/solr/guide/6_6/understanding-analyzers-tokenizers-and-filters.html
You'll probably be defining one similar to how text_general is
defined, a <fieldType> then use your new type in your <field>. This is
really the heart of how you make Solr do what you want when it comes
to what's searchable and what's not.

When you use the admin/analysis page, hover over the light gray
two-letter abbreviations and it'll pop up the class used for that
transformation.

You can start with WhitespaceTokenizerFactory which will break only on
whitespace. Be aware that other filters can then also manipulate the
tokens created by the tokenizer. WhitespaceTokenizerFactory will _not_
remove punctuation for instance, so you have to deal with that. For
example periods at the end of a sentence "I Like Cake." would be
included in the emitted tokens, so you'e have
I
Like
Cake.

You can use one of the filters to deal with that.

I would be very reluctant to use the "string" type, it's not analyzed
in any way and is almost always the wrong solution for something like
this. So input like this
I Like Cake.
would match _only_ I\ Like\ Cake.
You couldn't search on just the term "like", or even "Like" but only
"*Like*" which rather defeats the purpose of using tokenized search.

Best,
Erick

On Thu, Dec 7, 2017 at 8:37 AM, Bernd Schmidt <[hidden email]> wrote:

>
> Indeed, I saw in the analysis tab of the solr admin that the § char will be removed when using type text_general.
> But in this use case we want to make a full text search like "_text_:§45" or "_text_:§*" to find words starting with §.
> We need a text field here, not a string field!
> What is your recommended way to deal with it?
> Is it possible to remove the word break behaviour for the  § char?
> Or is the best way to encode all § chars when indexing and searching?
>
>
>
> Thanks, Bernd
>
>
>
>  Mit freundlichen Grüßen
>
>  Bernd Schmidt
>  SOFTWARE-ENTWICKLUNG
>
>  [hidden email]
>
>
>
>  Von:   Shawn Heisey <[hidden email]>
>  An:   <[hidden email]>
>  Gesendet:   07.12.2017 16:37
>  Betreff:   Re: Howto search for § character
>
> On 12/6/2017 9:09 AM, Bernd Schmidt wrote:
>> we have defined a field named "_text_" for a full text search based on field-type "text_general":
>> <field name="_text_" type="text_general" multiValued="true" indexed="true" stored="false"/>"
>>
>> When trying to search for the "§" character, we have strange behaviour:
>>
>> q=_text_:§ AND entityClass:StructureNodeImpl  => numFound:469 (all nodes where entityClass:StructureNodeImpl)
>> q=_text_:§ => numFound:0
>>
>> How can we search for the occurence of the § character?
>
> We can't see how your "text_general" type is defined, but if it is
> anything like the same type included in Solr examples, then it probably
> is using StandardTokenizerFactory.  It appears that this tokenizer
> treats the § character as a word break and removes it from the token
> stream.  Most likely, the reason the search with the extra clause works
> is that the part with that character is removed, and the query ends up
> ONLY being the extra clause.
>
> You will need a fieldType with an analysis chain that doesn't remove the
> § character, and it's almost guaranteed that you'll need to reindex.
> Unless you do that, searching for that character is not going to be
> possible.
>
> Also keep in mind that searching for a single character may not do what
> you expect if that character is not a single word in the text, and that
> certain filters can end up trimming out really short terms like that.
>
> Thanks,
> Shawn
>
>
>
>
>
>  eggheads GmbH
>  Herner Straße 370
> 44807 Bochum
>
> Fon +49 234 89397-0
> Fax +49 234 89397-28
>
>  www.eggheads.de
>  -----------------------------------------------
>
>
> Kunden DER TOURISTIK, EMSA, FRIATEC, MAMMUT, SUTTERLÜTY, SCHÄFER SHOP, THOMAS COOK, TUI, WILO SE, WÜRTH, u.v.m.
>
>
> Leistungen Standardsoftware für Product Information Management, Cross Media Publishing & Multi Channel Commerce, Prozessberatung
>
>
> Innovationspreis 2017 eggheads ist Sieger beim Innovationspreis-IT 2017 in der Kategorie E-Commerce. Mehr
>
> -----------------------------------------------
>
> Webinar Vorstellung der neuen Funktionalität der eggheads Suite am 12.12.2017. Mehr
>
> -----------------------------------------------
Reply | Threaded
Open this post in threaded view
|

Re: Howto search for § character

Shawn Heisey-2
In reply to this post by Bernd Schmidt
On 12/7/2017 9:37 AM, Bernd Schmidt wrote:
> Indeed, I saw in the analysis tab of the solr admin that the § char will be removed when using type text_general.
> But in this use case we want to make a full text search like "_text_:§45" or "_text_:§*" to find words starting with §.
> We need a text field here, not a string field!
> What is your recommended way to deal with it? 
> Is it possible to remove the word break behaviour for the  § char?
> Or is the best way to encode all § chars when indexing and searching?

This character is classified by Unicode as punctuation:

http://www.fileformat.info/info/unicode/char/00a7/index.htm

Almost any example field type for full-text search that you're likely to
encounter is going to be designed to split on punctuation and remove it
from the token stream.  That's one of the most common things that
full-text search engines do.

You're going to need to design a new analysis chain that *doesn't* do
this, apply the fieldType containing that analysis to your field,
restart/reload, and reindex.

Designing analysis chains is an art form, and tends to be one of the
hardest parts of setting up a production Solr install.  It took me at
least a month of almost constant work to settle on the schema design for
the indexes that I maintain.  All of the "solr.TextField" types in my
schema are completely custom -- none of the analysis chains in Solr
examples are in that schema.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Howto search for § character

Tim Casey
My last company we ended up writing a custom analyzer to handle
punctuation.  But this was for lucent 2 or 3.  That analyzer was carried
forward as we updated and was used for all human derived text.

Although now there are way better analyzers and way better ways to hook
them up, as noted above by Erick, We really cared about how this was done
and all of the work put into the analyzer paid off.

I would expect there to be an analyzer which would maintain punctuation
tokens for search.  One of the issues which comes up is if you want
multiple-runs of punctuation to be a single token or separate tokens.  So
what happens to "§!"  or "§?" or "?§", and in the case of things like
text/email what happens to "§!!!!".

In any event, my 2 pence worth....

tim

On Thu, Dec 7, 2017 at 10:00 AM, Shawn Heisey <[hidden email]> wrote:

> On 12/7/2017 9:37 AM, Bernd Schmidt wrote:
> > Indeed, I saw in the analysis tab of the solr admin that the § char will
> be removed when using type text_general.
> > But in this use case we want to make a full text search like
> "_text_:§45" or "_text_:§*" to find words starting with §.
> > We need a text field here, not a string field!
> > What is your recommended way to deal with it?
> > Is it possible to remove the word break behaviour for the  § char?
> > Or is the best way to encode all § chars when indexing and searching?
>
> This character is classified by Unicode as punctuation:
>
> http://www.fileformat.info/info/unicode/char/00a7/index.htm
>
> Almost any example field type for full-text search that you're likely to
> encounter is going to be designed to split on punctuation and remove it
> from the token stream.  That's one of the most common things that
> full-text search engines do.
>
> You're going to need to design a new analysis chain that *doesn't* do
> this, apply the fieldType containing that analysis to your field,
> restart/reload, and reindex.
>
> Designing analysis chains is an art form, and tends to be one of the
> hardest parts of setting up a production Solr install.  It took me at
> least a month of almost constant work to settle on the schema design for
> the indexes that I maintain.  All of the "solr.TextField" types in my
> schema are completely custom -- none of the analysis chains in Solr
> examples are in that schema.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Howto search for § character

Bernd Schmidt
In reply to this post by Shawn Heisey-2
Thanks for all the infos.
That helps so far to understand the issue .


Cheers, Bernd

 Mit freundlichen Grüßen

 Bernd Schmidt
 SOFTWARE-ENTWICKLUNG

 [hidden email]



 Von:   Shawn Heisey <[hidden email]>
 An:   <[hidden email]>
 Gesendet:   07.12.2017 19:00
 Betreff:   Re: Howto search for § character

On 12/7/2017 9:37 AM, Bernd Schmidt wrote:
> Indeed, I saw in the analysis tab of the solr admin that the § char will be removed when using type text_general.
> But in this use case we want to make a full text search like "_text_:§45" or "_text_:§*" to find words starting with §.
> We need a text field here, not a string field!
> What is your recommended way to deal with it? 
> Is it possible to remove the word break behaviour for the  § char?
> Or is the best way to encode all § chars when indexing and searching?
 
This character is classified by Unicode as punctuation:
 
http://www.fileformat.info/info/unicode/char/00a7/index.htm 
 
Almost any example field type for full-text search that you're likely to
encounter is going to be designed to split on punctuation and remove it
from the token stream.  That's one of the most common things that
full-text search engines do.
 
You're going to need to design a new analysis chain that *doesn't* do
this, apply the fieldType containing that analysis to your field,
restart/reload, and reindex.
 
Designing analysis chains is an art form, and tends to be one of the
hardest parts of setting up a production Solr install.  It took me at
least a month of almost constant work to settle on the schema design for
the indexes that I maintain.  All of the "solr.TextField" types in my
schema are completely custom -- none of the analysis chains in Solr
examples are in that schema.
 
Thanks,
Shawn
 




 eggheads GmbH
 Herner Straße 370
44807 Bochum

Fon +49 234 89397-0
Fax +49 234 89397-28
 
 www.eggheads.de
 -----------------------------------------------


Kunden DER TOURISTIK, EMSA, FRIATEC, MAMMUT, SUTTERLÜTY, SCHÄFER SHOP, THOMAS COOK, TUI, WILO SE, WÜRTH, u.v.m.


Leistungen Standardsoftware für Product Information Management, Cross Media Publishing & Multi Channel Commerce, Prozessberatung


Innovationspreis 2017 eggheads ist Sieger beim Innovationspreis-IT 2017 in der Kategorie E-Commerce. Mehr

-----------------------------------------------

Webinar Vorstellung der neuen Funktionalität der eggheads Suite am 12.12.2017. Mehr

-----------------------------------------------