[ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

Ian Holsman (Lists)

I just saw this on twitter, and thought you guys would be interested.. I
haven't tried it, but it looks interesting.

http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin

Thanks for the RT Shalin!
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

Don Werve-3
Too bad it requires integer (long) primary keys... :/

2010/3/8 Ian Holsman <[hidden email]>

>
> I just saw this on twitter, and thought you guys would be interested.. I
> haven't tried it, but it looks interesting.
>
> http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin
>
> Thanks for the RT Shalin!
>
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

Lance Norskog-2
Solr unique ids can be any type. The QueryElevateComponent complains
if the unique id is not a string, but you can comment out the QEC.  I
have one benchmark test with 2 billion documents with an integer id.
Works great.

On Mon, Mar 8, 2010 at 5:06 PM, Don Werve <[hidden email]> wrote:

> Too bad it requires integer (long) primary keys... :/
>
> 2010/3/8 Ian Holsman <[hidden email]>
>
>>
>> I just saw this on twitter, and thought you guys would be interested.. I
>> haven't tried it, but it looks interesting.
>>
>> http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin
>>
>> Thanks for the RT Shalin!
>>
>



--
Lance Norskog
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

Shalin Shekhar Mangar
I think Don is talking about Zoie - it requires a long uniqueKey.

On Tue, Mar 9, 2010 at 10:18 AM, Lance Norskog <[hidden email]> wrote:

> Solr unique ids can be any type. The QueryElevateComponent complains
> if the unique id is not a string, but you can comment out the QEC.  I
> have one benchmark test with 2 billion documents with an integer id.
> Works great.
>
> On Mon, Mar 8, 2010 at 5:06 PM, Don Werve <[hidden email]> wrote:
> > Too bad it requires integer (long) primary keys... :/
> >
> > 2010/3/8 Ian Holsman <[hidden email]>
> >
> >>
> >> I just saw this on twitter, and thought you guys would be interested.. I
> >> haven't tried it, but it looks interesting.
> >>
> >> http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Solr+Plugin
> >>
> >> Thanks for the RT Shalin!
> >>
> >
>
>
>
> --
> Lance Norskog
> [hidden email]
>



--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

Don Werve-3
2010/3/9 Shalin Shekhar Mangar <[hidden email]>

> I think Don is talking about Zoie - it requires a long uniqueKey.
>

Yep; we're using UUIDs.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

brad anderson-3
Tried following their tutorial for plugging zoie into solr:
    http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Server

It appears it only allows you to search on documents after you do a commit?
Am I missing something here, or does plugin not doing anything.

Their tutorial tells you to do a commit when you index the docs:

curl http://localhost:8983/solr/update/csv?commit=true --data-binary
@books.csv -H 'Content-type:text/plain; charset=utf-8'


When I don't do the commit, I cannot search the documents I've indexed.

Thanks,
Brad

On 9 March 2010 23:34, Don Werve <[hidden email]> wrote:

> 2010/3/9 Shalin Shekhar Mangar <[hidden email]>
>
> > I think Don is talking about Zoie - it requires a long uniqueKey.
> >
>
> Yep; we're using UUIDs.
>
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

Erik Hatcher-4
"When I don't do the commit, I cannot search the documents I've  
indexed." - that's exactly how Solr without Zoie works, and it's how  
Lucene itself works.  Gotta commit to see the documents indexed.

        Erik


On Mar 18, 2010, at 5:41 PM, brad anderson wrote:

> Tried following their tutorial for plugging zoie into solr:
>    http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Server
>
> It appears it only allows you to search on documents after you do a  
> commit?
> Am I missing something here, or does plugin not doing anything.
>
> Their tutorial tells you to do a commit when you index the docs:
>
> curl http://localhost:8983/solr/update/csv?commit=true --data-binary
> @books.csv -H 'Content-type:text/plain; charset=utf-8'
>
>
> When I don't do the commit, I cannot search the documents I've  
> indexed.
>
> Thanks,
> Brad
>
> On 9 March 2010 23:34, Don Werve <[hidden email]> wrote:
>
>> 2010/3/9 Shalin Shekhar Mangar <[hidden email]>
>>
>>> I think Don is talking about Zoie - it requires a long uniqueKey.
>>>
>>
>> Yep; we're using UUIDs.
>>

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

brad anderson-3
Indeed, which is why I'm wondering what is Zoie adding if you still need to
commit to search recent documents. Does anyone know?

Thanks,
Brad

On 18 March 2010 19:41, Erik Hatcher <[hidden email]> wrote:

> "When I don't do the commit, I cannot search the documents I've indexed." -
> that's exactly how Solr without Zoie works, and it's how Lucene itself
> works.  Gotta commit to see the documents indexed.
>
>        Erik
>
>
>
> On Mar 18, 2010, at 5:41 PM, brad anderson wrote:
>
>  Tried following their tutorial for plugging zoie into solr:
>>   http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Server
>>
>> It appears it only allows you to search on documents after you do a
>> commit?
>> Am I missing something here, or does plugin not doing anything.
>>
>> Their tutorial tells you to do a commit when you index the docs:
>>
>> curl http://localhost:8983/solr/update/csv?commit=true --data-binary
>> @books.csv -H 'Content-type:text/plain; charset=utf-8'
>>
>>
>> When I don't do the commit, I cannot search the documents I've indexed.
>>
>> Thanks,
>> Brad
>>
>> On 9 March 2010 23:34, Don Werve <[hidden email]> wrote:
>>
>>  2010/3/9 Shalin Shekhar Mangar <[hidden email]>
>>>
>>>  I think Don is talking about Zoie - it requires a long uniqueKey.
>>>>
>>>>
>>> Yep; we're using UUIDs.
>>>
>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

Janne Majaranta
To my understanding it adds a in-memory index which holds the recent  
commits and which is flushed to the main index based on the config  
options. Not sure if it helps to get solr near real time. I am  
evaluating it currently, and I am really not sure if it adds anything  
because of the cache regeneration of solr on every commit ??

-Janne

Lähetetty iPodista

brad anderson <[hidden email]> kirjoitti 19.3.2010 kello 20.53:

> Indeed, which is why I'm wondering what is Zoie adding if you still  
> need to
> commit to search recent documents. Does anyone know?
>
> Thanks,
> Brad
>
> On 18 March 2010 19:41, Erik Hatcher <[hidden email]> wrote:
>
>> "When I don't do the commit, I cannot search the documents I've  
>> indexed." -
>> that's exactly how Solr without Zoie works, and it's how Lucene  
>> itself
>> works.  Gotta commit to see the documents indexed.
>>
>>       Erik
>>
>>
>>
>> On Mar 18, 2010, at 5:41 PM, brad anderson wrote:
>>
>> Tried following their tutorial for plugging zoie into solr:
>>>  http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Server
>>>
>>> It appears it only allows you to search on documents after you do a
>>> commit?
>>> Am I missing something here, or does plugin not doing anything.
>>>
>>> Their tutorial tells you to do a commit when you index the docs:
>>>
>>> curl http://localhost:8983/solr/update/csv?commit=true --data-binary
>>> @books.csv -H 'Content-type:text/plain; charset=utf-8'
>>>
>>>
>>> When I don't do the commit, I cannot search the documents I've  
>>> indexed.
>>>
>>> Thanks,
>>> Brad
>>>
>>> On 9 March 2010 23:34, Don Werve <[hidden email]> wrote:
>>>
>>> 2010/3/9 Shalin Shekhar Mangar <[hidden email]>
>>>>
>>>> I think Don is talking about Zoie - it requires a long uniqueKey.
>>>>>
>>>>>
>>>> Yep; we're using UUIDs.
>>>>
>>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

brad anderson-3
I see, so when you do a commit it adds it to Zoie's ramdirectory. So, could
you just commit after every document without having a performance impact and
have real time search?

Thanks,
Brad

On 20 March 2010 00:34, Janne Majaranta <[hidden email]> wrote:

> To my understanding it adds a in-memory index which holds the recent
> commits and which is flushed to the main index based on the config options.
> Not sure if it helps to get solr near real time. I am evaluating it
> currently, and I am really not sure if it adds anything because of the cache
> regeneration of solr on every commit ??
>
> -Janne
>
> Lähetetty iPodista
>
> brad anderson <[hidden email]> kirjoitti 19.3.2010 kello 20.53:
>
>
>  Indeed, which is why I'm wondering what is Zoie adding if you still need
>> to
>> commit to search recent documents. Does anyone know?
>>
>> Thanks,
>> Brad
>>
>> On 18 March 2010 19:41, Erik Hatcher <[hidden email]> wrote:
>>
>>  "When I don't do the commit, I cannot search the documents I've indexed."
>>> -
>>> that's exactly how Solr without Zoie works, and it's how Lucene itself
>>> works.  Gotta commit to see the documents indexed.
>>>
>>>      Erik
>>>
>>>
>>>
>>> On Mar 18, 2010, at 5:41 PM, brad anderson wrote:
>>>
>>> Tried following their tutorial for plugging zoie into solr:
>>>
>>>>  http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Server
>>>>
>>>> It appears it only allows you to search on documents after you do a
>>>> commit?
>>>> Am I missing something here, or does plugin not doing anything.
>>>>
>>>> Their tutorial tells you to do a commit when you index the docs:
>>>>
>>>> curl http://localhost:8983/solr/update/csv?commit=true --data-binary
>>>> @books.csv -H 'Content-type:text/plain; charset=utf-8'
>>>>
>>>>
>>>> When I don't do the commit, I cannot search the documents I've indexed.
>>>>
>>>> Thanks,
>>>> Brad
>>>>
>>>> On 9 March 2010 23:34, Don Werve <[hidden email]> wrote:
>>>>
>>>> 2010/3/9 Shalin Shekhar Mangar <[hidden email]>
>>>>
>>>>>
>>>>> I think Don is talking about Zoie - it requires a long uniqueKey.
>>>>>
>>>>>>
>>>>>>
>>>>>>  Yep; we're using UUIDs.
>>>>>
>>>>>
>>>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

Grant Ingersoll-2

On Mar 23, 2010, at 7:29 PM, brad anderson wrote:

> I see, so when you do a commit it adds it to Zoie's ramdirectory. So, could
> you just commit after every document without having a performance impact and
> have real time search?
>

Not likely, maybe on really, really small indexes.  Zoie also does a writethrough, AIUI, to a file based index.


> Thanks,
> Brad
>
> On 20 March 2010 00:34, Janne Majaranta <[hidden email]> wrote:
>
>> To my understanding it adds a in-memory index which holds the recent
>> commits and which is flushed to the main index based on the config options.
>> Not sure if it helps to get solr near real time. I am evaluating it
>> currently, and I am really not sure if it adds anything because of the cache
>> regeneration of solr on every commit ??
>>
>> -Janne
>>
>> Lähetetty iPodista
>>
>> brad anderson <[hidden email]> kirjoitti 19.3.2010 kello 20.53:
>>
>>
>> Indeed, which is why I'm wondering what is Zoie adding if you still need
>>> to
>>> commit to search recent documents. Does anyone know?
>>>
>>> Thanks,
>>> Brad
>>>
>>> On 18 March 2010 19:41, Erik Hatcher <[hidden email]> wrote:
>>>
>>> "When I don't do the commit, I cannot search the documents I've indexed."
>>>> -
>>>> that's exactly how Solr without Zoie works, and it's how Lucene itself
>>>> works.  Gotta commit to see the documents indexed.
>>>>
>>>>     Erik
>>>>
>>>>
>>>>
>>>> On Mar 18, 2010, at 5:41 PM, brad anderson wrote:
>>>>
>>>> Tried following their tutorial for plugging zoie into solr:
>>>>
>>>>> http://snaprojects.jira.com/wiki/display/ZOIE/Zoie+Server
>>>>>
>>>>> It appears it only allows you to search on documents after you do a
>>>>> commit?
>>>>> Am I missing something here, or does plugin not doing anything.
>>>>>
>>>>> Their tutorial tells you to do a commit when you index the docs:
>>>>>
>>>>> curl http://localhost:8983/solr/update/csv?commit=true --data-binary
>>>>> @books.csv -H 'Content-type:text/plain; charset=utf-8'
>>>>>
>>>>>
>>>>> When I don't do the commit, I cannot search the documents I've indexed.
>>>>>
>>>>> Thanks,
>>>>> Brad
>>>>>
>>>>> On 9 March 2010 23:34, Don Werve <[hidden email]> wrote:
>>>>>
>>>>> 2010/3/9 Shalin Shekhar Mangar <[hidden email]>
>>>>>
>>>>>>
>>>>>> I think Don is talking about Zoie - it requires a long uniqueKey.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Yep; we're using UUIDs.
>>>>>>
>>>>>>
>>>>>>
>>>>

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search

qaz
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

qaz
do you mean that the plugin for solr doesn't have all the functionalities of the standalone zoie or do you mean zoie just simply cannot handle large indexes? by really really small, what exactly are we talking about here? are there any better ways for NRT? maybe in solr 1.5?

thanks
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Zoie Solr Plugin - Zoie Solr Plugin enables real-time update functionality for Apache Solr 1.4+

John Wang-9
Sorry, have completely missed this thread until someone pointed out to me
just now.

Some clarifications:

commit=true is for the plugin only. I added it to make it consistent with
Solr's behavior.

Stand-alone Zoie does not require this, it is in more of an autocommit mode
since it assumes a constant and busy stream of indexing events.

So I think what we will do is add an autocommit parameter to the
ZoieUpdateHandler and default it to true. With this, you don't need to have
commit=true to see the new docs.

-John

On Wed, Apr 14, 2010 at 5:25 PM, qaz <[hidden email]> wrote:

>
> do you mean that the plugin for solr doesn't have all the functionalities
> of
> the standalone zoie or do you mean zoie just simply cannot handle large
> indexes? by really really small, what exactly are we talking about here?
> are
> there any better ways for NRT? maybe in solr 1.5?
>
> thanks
> --
> View this message in context:
> http://n3.nabble.com/ANN-Zoie-Solr-Plugin-Zoie-Solr-Plugin-enables-real-time-update-functionality-for-Apache-Solr-1-4-tp506099p719893.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>