Custom sort function

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Custom sort function

Sripra deep
Hi Team,
  I wrote a custom sort function that will read the field value and parse
and returns a float value that will be used for sorting. this field is
indexed, stored and docvalues enabled. my latency increases drastically
from 10ms when the filter loads 50 documents and 200ms when it loads 250
documents and 1sec when it loads 1000 documents.

This is my function:

@Override
public FunctionValues getValues(Map context, LeafReaderContext reader)
throws IOException {
final FunctionValues vals = source.getValues(context, reader);
return new FunctionValues() {
        public float floatVal(int doc) {
        float toRet = isasc ? Float.MAX_VALUE : Float.MIN_VALUE;
        /*
        My custom logic in reading a string array and returning a value out
of some condition checks
        */
        }
    }
}

Can you suggest me some suggestions on this ?

Thanks,
Sripradeep P
Reply | Threaded
Open this post in threaded view
|

Re: Custom sort function

Jörn Franke
What methods do you use for your condition checks? Regexes ? Then you could for instance precompile the regexes (saves a lot of time). Any other method? I don’t ask about the exact condition check but only the methods you use within those checks.

> Am 27.11.2019 um 07:52 schrieb Sripra deep <[hidden email]>:
>
> Hi Team,
>  I wrote a custom sort function that will read the field value and parse
> and returns a float value that will be used for sorting. this field is
> indexed, stored and docvalues enabled. my latency increases drastically
> from 10ms when the filter loads 50 documents and 200ms when it loads 250
> documents and 1sec when it loads 1000 documents.
>
> This is my function:
>
> @Override
> public FunctionValues getValues(Map context, LeafReaderContext reader)
> throws IOException {
> final FunctionValues vals = source.getValues(context, reader);
> return new FunctionValues() {
>        public float floatVal(int doc) {
>        float toRet = isasc ? Float.MAX_VALUE : Float.MIN_VALUE;
>        /*
>        My custom logic in reading a string array and returning a value out
> of some condition checks
>        */
>        }
>    }
> }
>
> Can you suggest me some suggestions on this ?
>
> Thanks,
> Sripradeep P
Reply | Threaded
Open this post in threaded view
|

Re: Custom sort function

Jörn Franke
In reply to this post by Sripra deep
And have you tried how fast it is if you don’t do anything in this method?

> Am 27.11.2019 um 07:52 schrieb Sripra deep <[hidden email]>:
>
> Hi Team,
>  I wrote a custom sort function that will read the field value and parse
> and returns a float value that will be used for sorting. this field is
> indexed, stored and docvalues enabled. my latency increases drastically
> from 10ms when the filter loads 50 documents and 200ms when it loads 250
> documents and 1sec when it loads 1000 documents.
>
> This is my function:
>
> @Override
> public FunctionValues getValues(Map context, LeafReaderContext reader)
> throws IOException {
> final FunctionValues vals = source.getValues(context, reader);
> return new FunctionValues() {
>        public float floatVal(int doc) {
>        float toRet = isasc ? Float.MAX_VALUE : Float.MIN_VALUE;
>        /*
>        My custom logic in reading a string array and returning a value out
> of some condition checks
>        */
>        }
>    }
> }
>
> Can you suggest me some suggestions on this ?
>
> Thanks,
> Sripradeep P
Reply | Threaded
Open this post in threaded view
|

Re: Custom sort function

Sripra deep
Hi Jörn Franke,

  I modified the custom function to just return a constant value as 1.0 for
all the docs and ran the load again, the latency is worst like more than
20sec. The filter I am using will fetch 15k documents (so this function is
called 15k times). And if I don't call this function in my query then
latency is less than 10ms.

So it looks like some fundamental issue with this approach and I believe
its been used in many business scenarios. Looking out for what's going
 wrong.

Thanks,
Sripradeep P


On Wed, Nov 27, 2019 at 1:00 PM Jörn Franke <[hidden email]> wrote:

> And have you tried how fast it is if you don’t do anything in this method?
>
> > Am 27.11.2019 um 07:52 schrieb Sripra deep <[hidden email]
> >:
> >
> > Hi Team,
> >  I wrote a custom sort function that will read the field value and parse
> > and returns a float value that will be used for sorting. this field is
> > indexed, stored and docvalues enabled. my latency increases drastically
> > from 10ms when the filter loads 50 documents and 200ms when it loads 250
> > documents and 1sec when it loads 1000 documents.
> >
> > This is my function:
> >
> > @Override
> > public FunctionValues getValues(Map context, LeafReaderContext reader)
> > throws IOException {
> > final FunctionValues vals = source.getValues(context, reader);
> > return new FunctionValues() {
> >        public float floatVal(int doc) {
> >        float toRet = isasc ? Float.MAX_VALUE : Float.MIN_VALUE;
> >        /*
> >        My custom logic in reading a string array and returning a value
> out
> > of some condition checks
> >        */
> >        }
> >    }
> > }
> >
> > Can you suggest me some suggestions on this ?
> >
> > Thanks,
> > Sripradeep P
>
Reply | Threaded
Open this post in threaded view
|

Re: Custom sort function

Jörn Franke
Maybe can you do it as part of the loading to calculate the score?
Which Solr version are you using?
Are you doing some heavily lifting into the constructor or your filter?

> Am 27.11.2019 um 09:34 schrieb Sripra deep <[hidden email]>:
>
> 
> Hi Jörn Franke,
>
>   I modified the custom function to just return a constant value as 1.0 for all the docs and ran the load again, the latency is worst like more than 20sec. The filter I am using will fetch 15k documents (so this function is called 15k times). And if I don't call this function in my query then latency is less than 10ms.
>
> So it looks like some fundamental issue with this approach and I believe its been used in many business scenarios. Looking out for what's going wrong.
>
> Thanks,
> Sripradeep P
>
>
>> On Wed, Nov 27, 2019 at 1:00 PM Jörn Franke <[hidden email]> wrote:
>> And have you tried how fast it is if you don’t do anything in this method?
>>
>> > Am 27.11.2019 um 07:52 schrieb Sripra deep <[hidden email]>:
>> >
>> > Hi Team,
>> >  I wrote a custom sort function that will read the field value and parse
>> > and returns a float value that will be used for sorting. this field is
>> > indexed, stored and docvalues enabled. my latency increases drastically
>> > from 10ms when the filter loads 50 documents and 200ms when it loads 250
>> > documents and 1sec when it loads 1000 documents.
>> >
>> > This is my function:
>> >
>> > @Override
>> > public FunctionValues getValues(Map context, LeafReaderContext reader)
>> > throws IOException {
>> > final FunctionValues vals = source.getValues(context, reader);
>> > return new FunctionValues() {
>> >        public float floatVal(int doc) {
>> >        float toRet = isasc ? Float.MAX_VALUE : Float.MIN_VALUE;
>> >        /*
>> >        My custom logic in reading a string array and returning a value out
>> > of some condition checks
>> >        */
>> >        }
>> >    }
>> > }
>> >
>> > Can you suggest me some suggestions on this ?
>> >
>> > Thanks,
>> > Sripradeep P
Reply | Threaded
Open this post in threaded view
|

Re: Custom sort function

Sripra deep
My exact requirement is,  I will have a new field for a set of documents
for sorting. So the number of fields is growing like 10k
fields(custom_sort1,custom_sort2.,etc), for 10k groups, each group grouping
a set of documents. Since the documents can reside in more than one group I
cannot use a single field.
So I tried to use a single field for each document in the format as
"custom_field:group1=1$group2=3". During query time I will filter all the
group1 documents and parse this field and return the respective sort
number.

My solr version is *7.1.0*.

*This is my custom Class:*

public class CustomSortValueSource extends ValueSource{

protected final boolean isasc;
protected final String s1;
protected final ValueSource source;


public CustomSort(ValueSource sources, String s1, String order ) {
if (sources == null || s1 ==null || order == null) {
     throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
         "One or more inputs missing ");
   }

this.s1 = s1;
this.source = sources;
this.isasc = order.equals("asc") ? true : false;
}

@Override
public FunctionValues getValues(Map context, LeafReaderContext reader)
throws IOException {
final FunctionValues vals = source.getValues(context, reader);
return new FunctionValues() {
       public float floatVal(int doc) {
        float toRet = isasc ? Float.MAX_VALUE : Float.MIN_VALUE;
        /*
        My custom logic in reading a string array and returning a value out
of some condition checks
        */
        }
    }
}
}

There is nothing happening in the constructor other than assigning.

Thanks,
Sripradeep P


On Wed, Nov 27, 2019 at 2:30 PM Jörn Franke <[hidden email]> wrote:

> Maybe can you do it as part of the loading to calculate the score?
> Which Solr version are you using?
> Are you doing some heavily lifting into the constructor or your filter?
>
> Am 27.11.2019 um 09:34 schrieb Sripra deep <[hidden email]>:
>
> 
> Hi Jörn Franke,
>
>   I modified the custom function to just return a constant value as 1.0
> for all the docs and ran the load again, the latency is worst like more
> than 20sec. The filter I am using will fetch 15k documents (so this
> function is called 15k times). And if I don't call this function in my
> query then latency is less than 10ms.
>
> So it looks like some fundamental issue with this approach and I believe
> its been used in many business scenarios. Looking out for what's going
>  wrong.
>
> Thanks,
> Sripradeep P
>
>
> On Wed, Nov 27, 2019 at 1:00 PM Jörn Franke <[hidden email]> wrote:
>
>> And have you tried how fast it is if you don’t do anything in this
>> method?
>>
>> > Am 27.11.2019 um 07:52 schrieb Sripra deep <[hidden email]
>> >:
>> >
>> > Hi Team,
>> >  I wrote a custom sort function that will read the field value and parse
>> > and returns a float value that will be used for sorting. this field is
>> > indexed, stored and docvalues enabled. my latency increases drastically
>> > from 10ms when the filter loads 50 documents and 200ms when it loads 250
>> > documents and 1sec when it loads 1000 documents.
>> >
>> > This is my function:
>> >
>> > @Override
>> > public FunctionValues getValues(Map context, LeafReaderContext reader)
>> > throws IOException {
>> > final FunctionValues vals = source.getValues(context, reader);
>> > return new FunctionValues() {
>> >        public float floatVal(int doc) {
>> >        float toRet = isasc ? Float.MAX_VALUE : Float.MIN_VALUE;
>> >        /*
>> >        My custom logic in reading a string array and returning a value
>> out
>> > of some condition checks
>> >        */
>> >        }
>> >    }
>> > }
>> >
>> > Can you suggest me some suggestions on this ?
>> >
>> > Thanks,
>> > Sripradeep P
>>
>