storing results

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

storing results

Joan Codina

I'm using solr, to build a search engine, and it works great!!
Thanks for the job,guys!

but...
I need to build a searcher that must allow to perform a "search process" for a collection of documents. And
this search process may last several days,  and be composed of several queries that I must be able to store and reuse.
My doubts are:
How can I store the search results?
How can I add previously stored search results to a new search?

Thanks!!
--
signUPF

Joan Codina Filbà
Departament de Tecnologia
Universitat Pompeu Fabra

Reply | Threaded
Open this post in threaded view
|

Re: storing results

Mike Klaas
On 3/27/07, Joan Codina <[hidden email]> wrote:
>  I'm using solr, to build a search engine, and it works great!!
>  Thanks for the job,guys!

Glad it is working for you.

>  but...
>  I need to build a searcher that must allow to perform a "search process"
> for a collection of documents. And this search process may last several
> days,  and be composed of several queries that I must be able to store and
> reuse.
>  My doubts are:
>  How can I store the search results?
>  How can I add previously stored search results to a new search?

I'm not sure what you mean by "search process", nor really what you
mean by the functionality requirements you list.  You can certainly
store search results and combine them together however you wish, but
it seems that that is more on the post-query data-analysis side of
things rather than having anything to do with Solr.

Could you clarify what you're looking for Solr to do for you?

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: storing results

Joan Codina
I would like to store the results of a query someway,. Then after the
user analyzes some of  the documents (and he/she can take some days to
do it),  the user can try to make a query refinement over the previous
result, getting a subset of it.
To do so I need to  store the results as a filter, with a given name, so
the user can use it later on. But I need to store this in disk, as I can
not trust on the cache or the web session.
The user should  then indicate that the query that is doing now has a
filter (a previous query) and this filter should be added to the query
(this is allowed in solr,  i think) but as filter_ID, to be loaded to
solve the query.

I hope now its clearer what I'm trying to do

Joan


Mike Klaas wrote:

> On 3/27/07, Joan Codina <[hidden email]> wrote:
>>  I'm using solr, to build a search engine, and it works great!!
>>  Thanks for the job,guys!
>
> Glad it is working for you.
>
>>  but...
>>  I need to build a searcher that must allow to perform a "search
>> process"
>> for a collection of documents. And this search process may last several
>> days,  and be composed of several queries that I must be able to
>> store and
>> reuse.
>>  My doubts are:
>>  How can I store the search results?
>>  How can I add previously stored search results to a new search?
>
> I'm not sure what you mean by "search process", nor really what you
> mean by the functionality requirements you list.  You can certainly
> store search results and combine them together however you wish, but
> it seems that that is more on the post-query data-analysis side of
> things rather than having anything to do with Solr.
>
> Could you clarify what you're looking for Solr to do for you?
>
> -Mike


Reply | Threaded
Open this post in threaded view
|

Re: storing results

Erik Hatcher
Joan,

What you're after is something custom at a layer above Solr, not  
something that really fits as something built into Solr.

For example, I've implemented "saved searches" in both Collex <http://
www.nines.org/collex> and Flare (it's a proof-of-concept in Flare,  
saved only in Rails session scope).

For what you're after, I encourage you to build a simple application  
around Solr.  Rails + solr-ruby make a really fast and clean way to  
do this, *wink*. The application would allow users to enter queries  
or build them up from clicking on facets, print, and save that query  
to some persistent storage (I cringe saying the words "relational  
database").  The user would return, pick a previously saved query and  
refine it.

And indeed Solr does support this sort of refinement/filtering of  
queries.

        Erik


On Mar 27, 2007, at 7:20 PM, Joan Codina wrote:

> I would like to store the results of a query someway,. Then after  
> the user analyzes some of  the documents (and he/she can take some  
> days to do it),  the user can try to make a query refinement over  
> the previous result, getting a subset of it.
> To do so I need to  store the results as a filter, with a given  
> name, so the user can use it later on. But I need to store this in  
> disk, as I can not trust on the cache or the web session.
> The user should  then indicate that the query that is doing now has  
> a filter (a previous query) and this filter should be added to the  
> query (this is allowed in solr,  i think) but as filter_ID, to be  
> loaded to solve the query.
>
> I hope now its clearer what I'm trying to do
>
> Joan
>
>
> Mike Klaas wrote:
>> On 3/27/07, Joan Codina <[hidden email]> wrote:
>>>  I'm using solr, to build a search engine, and it works great!!
>>>  Thanks for the job,guys!
>>
>> Glad it is working for you.
>>
>>>  but...
>>>  I need to build a searcher that must allow to perform a "search  
>>> process"
>>> for a collection of documents. And this search process may last  
>>> several
>>> days,  and be composed of several queries that I must be able to  
>>> store and
>>> reuse.
>>>  My doubts are:
>>>  How can I store the search results?
>>>  How can I add previously stored search results to a new search?
>>
>> I'm not sure what you mean by "search process", nor really what you
>> mean by the functionality requirements you list.  You can certainly
>> store search results and combine them together however you wish, but
>> it seems that that is more on the post-query data-analysis side of
>> things rather than having anything to do with Solr.
>>
>> Could you clarify what you're looking for Solr to do for you?
>>
>> -Mike
>

Reply | Threaded
Open this post in threaded view
|

Re: storing results

Chris Hostetter-3
In reply to this post by Joan Codina

: To do so I need to  store the results as a filter, with a given name, so
: the user can use it later on. But I need to store this in disk, as I can
: not trust on the cache or the web session.
: The user should  then indicate that the query that is doing now has a
: filter (a previous query) and this filter should be added to the query
: (this is allowed in solr,  i think) but as filter_ID, to be loaded to
: solve the query.

if you really want to be ableto refer to these previous searches using
some sort of identifier, and have them persist for an indefinite amount of
time, it's really out of Solr's hands -- if someone were to try and add a
fewture like this to Solr, how would it know which queries to remember and
generate names for? how long would it store each name? ... these are the
kidns of questsions that your app can understand more easily ... you could
concievable use Solr to store the name=>querystring mappings in little
custom solr docs if you wanted, but you have to decide when to create
those mappings and when to expire them.

in general though, all you really need to remember is hte query string,
remembering all of the results really isn't neccessary.  The next time
your user wants to "refine" his search -- wether it's 10 seconds latter or
10 days later -- just take the old query string and combine it with the
new query string.  how you combine it depends on how you want the scoring
to work, use an "fq" param if the old query shouldn't affect the score,
just define the super set of docs, use a BooleanQuery if you want all the
clauses from both searches to impact the score.

it's important to understand that trying to keep track of the actual
results would be very, very, very bad  (especially if you want remember
them for a really long time) because when the user comes back, the index
may have chagned, docs may have shifted ids, or been edited so they no
longer match the criteria, or have been deleted completely.



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: storing results

Joan Codina
Thanks for your answers,

Yes, its true that with boolen queries things are much easier...
+(query1) +(query2) should do an and
or

(query1) (query2) should do an or

and this does not need a special ability to parse the queries
I like the dismax approach, I think is interesting.... but then to merge queries is a bit difficult, and I think for this option is better to try the filter, but then there is no OR option, only , AND. And bq (from what I understand from de docs) seems that also performs and AND

It's a pity I have to do it for a demo in a few days (and nights) and  I don't think the resulting code will be nice enough to be shown.

Joan

Chris Hostetter wrote:
: To do so I need to  store the results as a filter, with a given name, so
: the user can use it later on. But I need to store this in disk, as I can
: not trust on the cache or the web session.
: The user should  then indicate that the query that is doing now has a
: filter (a previous query) and this filter should be added to the query
: (this is allowed in solr,  i think) but as filter_ID, to be loaded to
: solve the query.

if you really want to be ableto refer to these previous searches using
some sort of identifier, and have them persist for an indefinite amount of
time, it's really out of Solr's hands -- if someone were to try and add a
fewture like this to Solr, how would it know which queries to remember and
generate names for? how long would it store each name? ... these are the
kidns of questsions that your app can understand more easily ... you could
concievable use Solr to store the name=>querystring mappings in little
custom solr docs if you wanted, but you have to decide when to create
those mappings and when to expire them.

in general though, all you really need to remember is hte query string,
remembering all of the results really isn't neccessary.  The next time
your user wants to "refine" his search -- wether it's 10 seconds latter or
10 days later -- just take the old query string and combine it with the
new query string.  how you combine it depends on how you want the scoring
to work, use an "fq" param if the old query shouldn't affect the score,
just define the super set of docs, use a BooleanQuery if you want all the
clauses from both searches to impact the score.

it's important to understand that trying to keep track of the actual
results would be very, very, very bad  (especially if you want remember
them for a really long time) because when the user comes back, the index
may have chagned, docs may have shifted ids, or been edited so they no
longer match the criteria, or have been deleted completely.



-Hoss

  

--
signUPF

Joan Codina Filbà
Departament de Tecnologia
Universitat Pompeu Fabra
_______________________________________________________________________________

Abans d'imprimir aquest e-mail, pensa si realment és necessari, i en cas de que ho sigui, pensa que si ho fas a doble cara estalvies un 25% del paper, els arbres t'ho agrairan.
_______________________________________________________________________________

La informació d'aquest missatge electrònic és confidencial, personal i intransferible i només està dirigida a la/les adreça/ces indicades a dalt. Si vostè llegeix aquest missatge per equivocació, l'informem que queda prohibida la seva divulgació, ús o distribució, completa o en part, i li preguem esborri el missatge original juntament amb els seus fitxers annexos sense llegir-lo ni gravar-lo.

Gràcies.