Replacing Google Mini Search Appliance with Solr?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Replacing Google Mini Search Appliance with Solr?

Palmer, Eric
Hello all,

Been lurking on the list for awhile.

We are at the end of life for replacing two google mini search appliances used to index our public web sites. Google is no longer selling the mini appliances and buying the big appliance is not cost beneficial.

http://search.richmond.edu/

We would run a solr replacement in linux (cents, redhat, similar) with open Java or Oracle Java.

Background
==========
~130 sites
only ~12,000 pages (at a depth of 3)
probably ~40,000 pages if we go to a depth of 4

We use key matches a lot. In solr terms these are elevated documents (elevations)

We would code a search query form in php and wrap it into our design (http://www.richmond.edu)

I have played with and love lucidworks and know that their $ solution works for our use cases but the cost model is not attractive for such a small collection.

So with solr what are my open source options and what are people's experiences crawling and indexing web sites with solr + crawler. I understand there is not a crawler with solr so that would have to be first up to get one working.

We can code in Java, PHP, Python etc. if we have to, but we don't want to write a crawler if we can avoid it.

thanks in advance for and information.

--
Eric Palmer
Web Services
U of Richmond

Reply | Threaded
Open this post in threaded view
|

RE: Replacing Google Mini Search Appliance with Solr?

Markus Jelsma-2
Hi Eric,

We have also helped some government institution to replave their expensive GSA with open source software. In our case we use Apache Nutch 1.7 to crawl the websites and index to Apache Solr. It is very effective, robust and scales easily with Hadoop if you have to. Nutch may not be the easiest tool for the job but is very stable, feature rich and has an active community here at Apache.

Cheers,
 
-----Original message-----

> From:Palmer, Eric <[hidden email]>
> Sent: Wednesday 30th October 2013 18:48
> To: [hidden email]
> Subject: Replacing Google Mini Search Appliance with Solr?
>
> Hello all,
>
> Been lurking on the list for awhile.
>
> We are at the end of life for replacing two google mini search appliances used to index our public web sites. Google is no longer selling the mini appliances and buying the big appliance is not cost beneficial.
>
> http://search.richmond.edu/
>
> We would run a solr replacement in linux (cents, redhat, similar) with open Java or Oracle Java.
>
> Background
> ==========
> ~130 sites
> only ~12,000 pages (at a depth of 3)
> probably ~40,000 pages if we go to a depth of 4
>
> We use key matches a lot. In solr terms these are elevated documents (elevations)
>
> We would code a search query form in php and wrap it into our design (http://www.richmond.edu)
>
> I have played with and love lucidworks and know that their $ solution works for our use cases but the cost model is not attractive for such a small collection.
>
> So with solr what are my open source options and what are people's experiences crawling and indexing web sites with solr + crawler. I understand there is not a crawler with solr so that would have to be first up to get one working.
>
> We can code in Java, PHP, Python etc. if we have to, but we don't want to write a crawler if we can avoid it.
>
> thanks in advance for and information.
>
> --
> Eric Palmer
> Web Services
> U of Richmond
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Google Mini Search Appliance with Solr?

Jason Hellman
Nutch is an excellent option.  It should feel very comfortable for people migrating away from the Google appliances.

Apache Droids is another possible way to approach, and I’ve found people using Heretrix or Manifold for various use cases (and usually in combination with other use cases where the extra overhead was worth the trouble).

I think the simples approach will be Nutch…it’s absolutely worth taking a shot at it.

DO NOT write a crawler!  That is a rabbit hole you do not want to peer down into :)



On Oct 30, 2013, at 10:54 AM, Markus Jelsma <[hidden email]> wrote:

> Hi Eric,
>
> We have also helped some government institution to replave their expensive GSA with open source software. In our case we use Apache Nutch 1.7 to crawl the websites and index to Apache Solr. It is very effective, robust and scales easily with Hadoop if you have to. Nutch may not be the easiest tool for the job but is very stable, feature rich and has an active community here at Apache.
>
> Cheers,
>
> -----Original message-----
>> From:Palmer, Eric <[hidden email]>
>> Sent: Wednesday 30th October 2013 18:48
>> To: [hidden email]
>> Subject: Replacing Google Mini Search Appliance with Solr?
>>
>> Hello all,
>>
>> Been lurking on the list for awhile.
>>
>> We are at the end of life for replacing two google mini search appliances used to index our public web sites. Google is no longer selling the mini appliances and buying the big appliance is not cost beneficial.
>>
>> http://search.richmond.edu/
>>
>> We would run a solr replacement in linux (cents, redhat, similar) with open Java or Oracle Java.
>>
>> Background
>> ==========
>> ~130 sites
>> only ~12,000 pages (at a depth of 3)
>> probably ~40,000 pages if we go to a depth of 4
>>
>> We use key matches a lot. In solr terms these are elevated documents (elevations)
>>
>> We would code a search query form in php and wrap it into our design (http://www.richmond.edu)
>>
>> I have played with and love lucidworks and know that their $ solution works for our use cases but the cost model is not attractive for such a small collection.
>>
>> So with solr what are my open source options and what are people's experiences crawling and indexing web sites with solr + crawler. I understand there is not a crawler with solr so that would have to be first up to get one working.
>>
>> We can code in Java, PHP, Python etc. if we have to, but we don't want to write a crawler if we can avoid it.
>>
>> thanks in advance for and information.
>>
>> --
>> Eric Palmer
>> Web Services
>> U of Richmond
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Replacing Google Mini Search Appliance with Solr?

Palmer, Eric
Markus and Jason

thanks for the info.

I will start to research Nutch.  Writing a crawler, agree it is a rabbit
hole.


--
Eric Palmer

Web Services
U of Richmond

To report technical issues, obtain technical support or make requests for
enhancements please visit
http://web.richmond.edu/contact/technical-support.html





On 10/30/13 2:53 PM, "Jason Hellman" <[hidden email]>
wrote:

>Nutch is an excellent option.  It should feel very comfortable for people
>migrating away from the Google appliances.
>
>Apache Droids is another possible way to approach, and I¹ve found people
>using Heretrix or Manifold for various use cases (and usually in
>combination with other use cases where the extra overhead was worth the
>trouble).
>
>I think the simples approach will be NutchŠit¹s absolutely worth taking a
>shot at it.
>
>DO NOT write a crawler!  That is a rabbit hole you do not want to peer
>down into :)
>
>
>
>On Oct 30, 2013, at 10:54 AM, Markus Jelsma <[hidden email]>
>wrote:
>
>> Hi Eric,
>>
>> We have also helped some government institution to replave their
>>expensive GSA with open source software. In our case we use Apache Nutch
>>1.7 to crawl the websites and index to Apache Solr. It is very
>>effective, robust and scales easily with Hadoop if you have to. Nutch
>>may not be the easiest tool for the job but is very stable, feature rich
>>and has an active community here at Apache.
>>
>> Cheers,
>>
>> -----Original message-----
>>> From:Palmer, Eric <[hidden email]>
>>> Sent: Wednesday 30th October 2013 18:48
>>> To: [hidden email]
>>> Subject: Replacing Google Mini Search Appliance with Solr?
>>>
>>> Hello all,
>>>
>>> Been lurking on the list for awhile.
>>>
>>> We are at the end of life for replacing two google mini search
>>>appliances used to index our public web sites. Google is no longer
>>>selling the mini appliances and buying the big appliance is not cost
>>>beneficial.
>>>
>>> http://search.richmond.edu/
>>>
>>> We would run a solr replacement in linux (cents, redhat, similar) with
>>>open Java or Oracle Java.
>>>
>>> Background
>>> ==========
>>> ~130 sites
>>> only ~12,000 pages (at a depth of 3)
>>> probably ~40,000 pages if we go to a depth of 4
>>>
>>> We use key matches a lot. In solr terms these are elevated documents
>>>(elevations)
>>>
>>> We would code a search query form in php and wrap it into our design
>>>(http://www.richmond.edu)
>>>
>>> I have played with and love lucidworks and know that their $ solution
>>>works for our use cases but the cost model is not attractive for such a
>>>small collection.
>>>
>>> So with solr what are my open source options and what are people's
>>>experiences crawling and indexing web sites with solr + crawler. I
>>>understand there is not a crawler with solr so that would have to be
>>>first up to get one working.
>>>
>>> We can code in Java, PHP, Python etc. if we have to, but we don't want
>>>to write a crawler if we can avoid it.
>>>
>>> thanks in advance for and information.
>>>
>>> --
>>> Eric Palmer
>>> Web Services
>>> U of Richmond
>>>
>>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Replacing Google Mini Search Appliance with Solr?

Rajinimaski
Hi Eric,

  I have also developed mini-applications replacing GSA for some of our
clients using Apache Nutch + Solr to crawl multi lingual sites and enable
multi-lingual search. Nutch+Solr is very stable and Nutch mailing list
provides a good support.

Reference link to start:
https://sites.google.com/site/profilerajanimaski/webcrawlers/apache-nutch

Thanks
Rajani




On Thu, Oct 31, 2013 at 12:27 AM, Palmer, Eric <[hidden email]> wrote:

> Markus and Jason
>
> thanks for the info.
>
> I will start to research Nutch.  Writing a crawler, agree it is a rabbit
> hole.
>
>
> --
> Eric Palmer
>
> Web Services
> U of Richmond
>
> To report technical issues, obtain technical support or make requests for
> enhancements please visit
> http://web.richmond.edu/contact/technical-support.html
>
>
>
>
>
> On 10/30/13 2:53 PM, "Jason Hellman" <[hidden email]>
> wrote:
>
> >Nutch is an excellent option.  It should feel very comfortable for people
> >migrating away from the Google appliances.
> >
> >Apache Droids is another possible way to approach, and I¹ve found people
> >using Heretrix or Manifold for various use cases (and usually in
> >combination with other use cases where the extra overhead was worth the
> >trouble).
> >
> >I think the simples approach will be NutchŠit¹s absolutely worth taking a
> >shot at it.
> >
> >DO NOT write a crawler!  That is a rabbit hole you do not want to peer
> >down into :)
> >
> >
> >
> >On Oct 30, 2013, at 10:54 AM, Markus Jelsma <[hidden email]>
> >wrote:
> >
> >> Hi Eric,
> >>
> >> We have also helped some government institution to replave their
> >>expensive GSA with open source software. In our case we use Apache Nutch
> >>1.7 to crawl the websites and index to Apache Solr. It is very
> >>effective, robust and scales easily with Hadoop if you have to. Nutch
> >>may not be the easiest tool for the job but is very stable, feature rich
> >>and has an active community here at Apache.
> >>
> >> Cheers,
> >>
> >> -----Original message-----
> >>> From:Palmer, Eric <[hidden email]>
> >>> Sent: Wednesday 30th October 2013 18:48
> >>> To: [hidden email]
> >>> Subject: Replacing Google Mini Search Appliance with Solr?
> >>>
> >>> Hello all,
> >>>
> >>> Been lurking on the list for awhile.
> >>>
> >>> We are at the end of life for replacing two google mini search
> >>>appliances used to index our public web sites. Google is no longer
> >>>selling the mini appliances and buying the big appliance is not cost
> >>>beneficial.
> >>>
> >>> http://search.richmond.edu/
> >>>
> >>> We would run a solr replacement in linux (cents, redhat, similar) with
> >>>open Java or Oracle Java.
> >>>
> >>> Background
> >>> ==========
> >>> ~130 sites
> >>> only ~12,000 pages (at a depth of 3)
> >>> probably ~40,000 pages if we go to a depth of 4
> >>>
> >>> We use key matches a lot. In solr terms these are elevated documents
> >>>(elevations)
> >>>
> >>> We would code a search query form in php and wrap it into our design
> >>>(http://www.richmond.edu)
> >>>
> >>> I have played with and love lucidworks and know that their $ solution
> >>>works for our use cases but the cost model is not attractive for such a
> >>>small collection.
> >>>
> >>> So with solr what are my open source options and what are people's
> >>>experiences crawling and indexing web sites with solr + crawler. I
> >>>understand there is not a crawler with solr so that would have to be
> >>>first up to get one working.
> >>>
> >>> We can code in Java, PHP, Python etc. if we have to, but we don't want
> >>>to write a crawler if we can avoid it.
> >>>
> >>> thanks in advance for and information.
> >>>
> >>> --
> >>> Eric Palmer
> >>> Web Services
> >>> U of Richmond
> >>>
> >>>
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Google Mini Search Appliance with Solr?

Palmer, Eric
Thanks for the link

Sent from my iPhone

On Oct 30, 2013, at 4:06 PM, "Rajani Maski" <[hidden email]> wrote:

> Hi Eric,
>
>  I have also developed mini-applications replacing GSA for some of our
> clients using Apache Nutch + Solr to crawl multi lingual sites and enable
> multi-lingual search. Nutch+Solr is very stable and Nutch mailing list
> provides a good support.
>
> Reference link to start:
> https://sites.google.com/site/profilerajanimaski/webcrawlers/apache-nutch
>
> Thanks
> Rajani
>
>
>
>
> On Thu, Oct 31, 2013 at 12:27 AM, Palmer, Eric <[hidden email]> wrote:
>
>> Markus and Jason
>>
>> thanks for the info.
>>
>> I will start to research Nutch.  Writing a crawler, agree it is a rabbit
>> hole.
>>
>>
>> --
>> Eric Palmer
>>
>> Web Services
>> U of Richmond
>>
>> To report technical issues, obtain technical support or make requests for
>> enhancements please visit
>> http://web.richmond.edu/contact/technical-support.html
>>
>>
>>
>>
>>
>> On 10/30/13 2:53 PM, "Jason Hellman" <[hidden email]>
>> wrote:
>>
>>> Nutch is an excellent option.  It should feel very comfortable for people
>>> migrating away from the Google appliances.
>>>
>>> Apache Droids is another possible way to approach, and I¹ve found people
>>> using Heretrix or Manifold for various use cases (and usually in
>>> combination with other use cases where the extra overhead was worth the
>>> trouble).
>>>
>>> I think the simples approach will be NutchŠit¹s absolutely worth taking a
>>> shot at it.
>>>
>>> DO NOT write a crawler!  That is a rabbit hole you do not want to peer
>>> down into :)
>>>
>>>
>>>
>>> On Oct 30, 2013, at 10:54 AM, Markus Jelsma <[hidden email]>
>>> wrote:
>>>
>>>> Hi Eric,
>>>>
>>>> We have also helped some government institution to replave their
>>>> expensive GSA with open source software. In our case we use Apache Nutch
>>>> 1.7 to crawl the websites and index to Apache Solr. It is very
>>>> effective, robust and scales easily with Hadoop if you have to. Nutch
>>>> may not be the easiest tool for the job but is very stable, feature rich
>>>> and has an active community here at Apache.
>>>>
>>>> Cheers,
>>>>
>>>> -----Original message-----
>>>>> From:Palmer, Eric <[hidden email]>
>>>>> Sent: Wednesday 30th October 2013 18:48
>>>>> To: [hidden email]
>>>>> Subject: Replacing Google Mini Search Appliance with Solr?
>>>>>
>>>>> Hello all,
>>>>>
>>>>> Been lurking on the list for awhile.
>>>>>
>>>>> We are at the end of life for replacing two google mini search
>>>>> appliances used to index our public web sites. Google is no longer
>>>>> selling the mini appliances and buying the big appliance is not cost
>>>>> beneficial.
>>>>>
>>>>> http://search.richmond.edu/
>>>>>
>>>>> We would run a solr replacement in linux (cents, redhat, similar) with
>>>>> open Java or Oracle Java.
>>>>>
>>>>> Background
>>>>> ==========
>>>>> ~130 sites
>>>>> only ~12,000 pages (at a depth of 3)
>>>>> probably ~40,000 pages if we go to a depth of 4
>>>>>
>>>>> We use key matches a lot. In solr terms these are elevated documents
>>>>> (elevations)
>>>>>
>>>>> We would code a search query form in php and wrap it into our design
>>>>> (http://www.richmond.edu)
>>>>>
>>>>> I have played with and love lucidworks and know that their $ solution
>>>>> works for our use cases but the cost model is not attractive for such a
>>>>> small collection.
>>>>>
>>>>> So with solr what are my open source options and what are people's
>>>>> experiences crawling and indexing web sites with solr + crawler. I
>>>>> understand there is not a crawler with solr so that would have to be
>>>>> first up to get one working.
>>>>>
>>>>> We can code in Java, PHP, Python etc. if we have to, but we don't want
>>>>> to write a crawler if we can avoid it.
>>>>>
>>>>> thanks in advance for and information.
>>>>>
>>>>> --
>>>>> Eric Palmer
>>>>> Web Services
>>>>> U of Richmond
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Google Mini Search Appliance with Solr?

kamaci
We also use Nutch at our environment.  Nutch crawls the data and it to Solr
for indexing. I have implemented a custom search API that interacts with my
Solr indexes cos of I don't want to expose my indexes directly to outside.
You can easily configure and build up what you want with such kind of
combination.

30 Ekim 2013 Çarşamba tarihinde Palmer, Eric <[hidden email]> adlı
kullanıcı şöyle yazdı:

> Thanks for the link
>
> Sent from my iPhone
>
> On Oct 30, 2013, at 4:06 PM, "Rajani Maski" <[hidden email]> wrote:
>
>> Hi Eric,
>>
>>  I have also developed mini-applications replacing GSA for some of our
>> clients using Apache Nutch + Solr to crawl multi lingual sites and enable
>> multi-lingual search. Nutch+Solr is very stable and Nutch mailing list
>> provides a good support.
>>
>> Reference link to start:
>> apache nutch | profilerajanimaski
>>
>> Thanks
>> Rajani
>>
>>
>>
>>
>> On Thu, Oct 31, 2013 at 12:27 AM, Palmer, Eric <[hidden email]>
wrote:

>>
>>> Markus and Jason
>>>
>>> thanks for the info.
>>>
>>> I will start to research Nutch.  Writing a crawler, agree it is a rabbit
>>> hole.
>>>
>>>
>>> --
>>> Eric Palmer
>>>
>>> Web Services
>>> U of Richmond
>>>
>>> To report technical issues, obtain technical support or make requests
for

>>> enhancements please visit
>>> http://web.richmond.edu/contact/technical-support.html
>>>
>>>
>>>
>>>
>>>
>>> On 10/30/13 2:53 PM, "Jason Hellman" <[hidden email]>
>>> wrote:
>>>
>>>> Nutch is an excellent option.  It should feel very comfortable for
people
>>>> migrating away from the Google appliances.
>>>>
>>>> Apache Droids is another possible way to approach, and I¹ve found
people
>>>> using Heretrix or Manifold for various use cases (and usually in
>>>> combination with other use cases where the extra overhead was worth the
>>>> trouble).
>>>>
>>>> I think the simples approach will be NutchŠit¹s absolutely worth
taking a

>>>> shot at it.
>>>>
>>>> DO NOT write a crawler!  That is a rabbit hole you do not want to peer
>>>> down into :)
>>>>
>>>>
>>>>
>>>> On Oct 30, 2013, at 10:54 AM, Markus Jelsma <[hidden email]
>
>>>> wrote:
>>>>
>>>>> Hi Eric,
>>>>>
>>>>> We have also helped some government institution to replave their
>>>>> expensive GSA with open source software. In our case we use Apache
Nutch
>>>>> 1.7 to crawl the websites and index to Apache Solr. It is very
>>>>> effective, robust and scales easily with Hadoop if you have to. Nutch
>>>>> may not be the easiest tool for the job but is very stable, feature
rich

>>>>> and has an active community here at Apache.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> -----Original message-----
>>>>>> From:Palmer, Eric <[hidden email]>
>>>>>> Sent: Wednesday 30th October 2013 18:48
>>>>>> To: [hidden email]
>>>>>> Subject: Replacing Google Mini Search Appliance with Solr?
>>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>> Been lurking on the list for awhile.
>>>>>>
>>>>>> We are at the end of life for replacing two google mini search
>>>>>> appliances used to index our public web sites. Google is no longer
>>>>>> selling the mini appliances and buying the big appliance is not cost
>>>>>> beneficial.
>>>>>>
>>>>>> http://search.richmond.edu/
>>>>>>
>>>>>> We would run a solr replacement in linux (cents, redhat, similar)
with
>>>>>> open Java or Oracle Java.
>>>>>>
>>>>>> Background
>>>>>> ==========
>>>>>> ~130 sites
>