How can I make better project than Lucene?

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

How can I make better project than Lucene?

sangwook DEV
I’m developing search engine, Fastcatsearch. http://github <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch

Lucene is widely known and famous project and I cannot beat Lucene for now.

But is there any chance to beat Lucene?

Anything like features, performance.

Please, let me know what to do to make better product than Lucene.

Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

John Wang-9
Why not spend the effort improving Lucene?

John
On Nov 14, 2014 5:46 PM, "swsong_dev" <[hidden email]> wrote:

> I’m developing search engine, Fastcatsearch. http://github
> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>
> Lucene is widely known and famous project and I cannot beat Lucene for now.
>
> But is there any chance to beat Lucene?
>
> Anything like features, performance.
>
> Please, let me know what to do to make better product than Lucene.
>
> Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

Saurabh Tiwari
+1 John wang

On Saturday, November 15, 2014, John Wang <[hidden email]> wrote:

> Why not spend the effort improving Lucene?
>
> John
> On Nov 14, 2014 5:46 PM, "swsong_dev" <[hidden email]
> <javascript:;>> wrote:
>
> > I’m developing search engine, Fastcatsearch. http://github
> > <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
> >
> > Lucene is widely known and famous project and I cannot beat Lucene for
> now.
> >
> > But is there any chance to beat Lucene?
> >
> > Anything like features, performance.
> >
> > Please, let me know what to do to make better product than Lucene.
> >
> > Thank you.
>


--
Saurabh
Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

Michael McCandless-2
In reply to this post by sangwook DEV
Actually I think competing projects is very healthy for open source development.

There are many things you could explore to "contrast" with Lucene,
e.g. write your new search engine in Go not Java: Java has many
problems, maybe Go fixes them.  Go also has a low-latency garbage
collector in development ... and Java's GC options still can't scale
to the heap sizes that are practical now.

Lucene has many limitations, so your competing engine could focus on
them.  E.g. the "schemalessness" of Lucene has become a big problem,
and near impossible to fix at this point, and prevents new important
features like LUCENE-5879 from being possible, so you could give your
engine a "gentle" schema from the start.

The Lucene Filter/Query situation is a mess: one should extend the other.

Lucene has weak support for proximity queries (SpanQuery is slow and
does not get much attention).

Lucene is showing its age, missing some compelling features like a
builtin transaction log, "core" support for numerics (they are sort of
hacked on top), optimistic concurrency support (sequence ids,
versions, something), distributed support (near real time replication,
etc.), multi-tenancy, an example server implementation, so the search
servers on top of Lucene have had to fill these gaps.  Maybe you could
make your engine distributed from the start (Go is a great match for
that, from what little I know).

All 3 highlighter options have problems.

The analysis chain (attributes) is overly complex.

In your competing engine you can borrow/copy/steal from Lucene's good
parts to get started...


Mike McCandless

http://blog.mikemccandless.com


On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:

> I’m developing search engine, Fastcatsearch. http://github <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>
> Lucene is widely known and famous project and I cannot beat Lucene for now.
>
> But is there any chance to beat Lucene?
>
> Anything like features, performance.
>
> Please, let me know what to do to make better product than Lucene.
>
> Thank you.
Reply | Threaded
Open this post in threaded view
|

RE: How can I make better project than Lucene?

wmartinusa
Thank you for a solid list of things to look into in Lucene Michael.

If the poster wants to make a better lucene, I presume he has considered the logistics and science needed; a rather daunting view imo.

-----Original Message-----
From: Michael McCandless [mailto:[hidden email]]
Sent: Saturday, November 15, 2014 6:22 AM
To: [hidden email]
Subject: Re: How can I make better project than Lucene?

Actually I think competing projects is very healthy for open source development.

There are many things you could explore to "contrast" with Lucene, e.g. write your new search engine in Go not Java: Java has many problems, maybe Go fixes them.  Go also has a low-latency garbage collector in development ... and Java's GC options still can't scale to the heap sizes that are practical now.

Lucene has many limitations, so your competing engine could focus on them.  E.g. the "schemalessness" of Lucene has become a big problem, and near impossible to fix at this point, and prevents new important features like LUCENE-5879 from being possible, so you could give your engine a "gentle" schema from the start.

The Lucene Filter/Query situation is a mess: one should extend the other.

Lucene has weak support for proximity queries (SpanQuery is slow and does not get much attention).

Lucene is showing its age, missing some compelling features like a builtin transaction log, "core" support for numerics (they are sort of hacked on top), optimistic concurrency support (sequence ids, versions, something), distributed support (near real time replication, etc.), multi-tenancy, an example server implementation, so the search servers on top of Lucene have had to fill these gaps.  Maybe you could make your engine distributed from the start (Go is a great match for that, from what little I know).

All 3 highlighter options have problems.

The analysis chain (attributes) is overly complex.

In your competing engine you can borrow/copy/steal from Lucene's good parts to get started...


Mike McCandless

http://blog.mikemccandless.com


On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:

> I’m developing search engine, Fastcatsearch. http://github 
> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>
> Lucene is widely known and famous project and I cannot beat Lucene for now.
>
> But is there any chance to beat Lucene?
>
> Anything like features, performance.
>
> Please, let me know what to do to make better product than Lucene.
>
> Thank you.

Reply | Threaded
Open this post in threaded view
|

RE: How can I make better project than Lucene?

wmartinusa
In reply to this post by Michael McCandless-2
Btw: SwSong should not steal code; which implies an existing license whose terms he is willing to break. Not a good first step.    ;-)

will

-----Original Message-----
From: Michael McCandless [mailto:[hidden email]]
Sent: Saturday, November 15, 2014 6:22 AM
To: [hidden email]
Subject: Re: How can I make better project than Lucene?

Actually I think competing projects is very healthy for open source development.

There are many things you could explore to "contrast" with Lucene, e.g. write your new search engine in Go not Java: Java has many problems, maybe Go fixes them.  Go also has a low-latency garbage collector in development ... and Java's GC options still can't scale to the heap sizes that are practical now.

Lucene has many limitations, so your competing engine could focus on them.  E.g. the "schemalessness" of Lucene has become a big problem, and near impossible to fix at this point, and prevents new important features like LUCENE-5879 from being possible, so you could give your engine a "gentle" schema from the start.

The Lucene Filter/Query situation is a mess: one should extend the other.

Lucene has weak support for proximity queries (SpanQuery is slow and does not get much attention).

Lucene is showing its age, missing some compelling features like a builtin transaction log, "core" support for numerics (they are sort of hacked on top), optimistic concurrency support (sequence ids, versions, something), distributed support (near real time replication, etc.), multi-tenancy, an example server implementation, so the search servers on top of Lucene have had to fill these gaps.  Maybe you could make your engine distributed from the start (Go is a great match for that, from what little I know).

All 3 highlighter options have problems.

The analysis chain (attributes) is overly complex.

In your competing engine you can borrow/copy/steal from Lucene's good parts to get started...


Mike McCandless

http://blog.mikemccandless.com


On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:

> I’m developing search engine, Fastcatsearch. http://github 
> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>
> Lucene is widely known and famous project and I cannot beat Lucene for now.
>
> But is there any chance to beat Lucene?
>
> Anything like features, performance.
>
> Please, let me know what to do to make better product than Lucene.
>
> Thank you.

Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

Siva Thumma
To build such a big product, One would obviously attribute the license.

Sent from iPhone

> On 15-Nov-2014, at 5:12 pm, Will Martin <[hidden email]> wrote:
>
> Btw: SwSong should not steal code; which implies an existing license whose terms he is willing to break. Not a good first step.    ;-)
>
> will
>
> -----Original Message-----
> From: Michael McCandless [mailto:[hidden email]]
> Sent: Saturday, November 15, 2014 6:22 AM
> To: [hidden email]
> Subject: Re: How can I make better project than Lucene?
>
> Actually I think competing projects is very healthy for open source development.
>
> There are many things you could explore to "contrast" with Lucene, e.g. write your new search engine in Go not Java: Java has many problems, maybe Go fixes them.  Go also has a low-latency garbage collector in development ... and Java's GC options still can't scale to the heap sizes that are practical now.
>
> Lucene has many limitations, so your competing engine could focus on them.  E.g. the "schemalessness" of Lucene has become a big problem, and near impossible to fix at this point, and prevents new important features like LUCENE-5879 from being possible, so you could give your engine a "gentle" schema from the start.
>
> The Lucene Filter/Query situation is a mess: one should extend the other.
>
> Lucene has weak support for proximity queries (SpanQuery is slow and does not get much attention).
>
> Lucene is showing its age, missing some compelling features like a builtin transaction log, "core" support for numerics (they are sort of hacked on top), optimistic concurrency support (sequence ids, versions, something), distributed support (near real time replication, etc.), multi-tenancy, an example server implementation, so the search servers on top of Lucene have had to fill these gaps.  Maybe you could make your engine distributed from the start (Go is a great match for that, from what little I know).
>
> All 3 highlighter options have problems.
>
> The analysis chain (attributes) is overly complex.
>
> In your competing engine you can borrow/copy/steal from Lucene's good parts to get started...
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
>> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:
>> I’m developing search engine, Fastcatsearch. http://github 
>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>
>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>
>> But is there any chance to beat Lucene?
>>
>> Anything like features, performance.
>>
>> Please, let me know what to do to make better product than Lucene.
>>
>> Thank you.
>
Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

Michael McCandless-2
In reply to this post by wmartinusa
Well the Apache Software License is very generous about poaching.

"Your ideas will go further if you don't insist on going with them."

Mike McCandless

http://blog.mikemccandless.com


On Sat, Nov 15, 2014 at 6:42 AM, Will Martin <[hidden email]> wrote:

> Btw: SwSong should not steal code; which implies an existing license whose terms he is willing to break. Not a good first step.    ;-)
>
> will
>
> -----Original Message-----
> From: Michael McCandless [mailto:[hidden email]]
> Sent: Saturday, November 15, 2014 6:22 AM
> To: [hidden email]
> Subject: Re: How can I make better project than Lucene?
>
> Actually I think competing projects is very healthy for open source development.
>
> There are many things you could explore to "contrast" with Lucene, e.g. write your new search engine in Go not Java: Java has many problems, maybe Go fixes them.  Go also has a low-latency garbage collector in development ... and Java's GC options still can't scale to the heap sizes that are practical now.
>
> Lucene has many limitations, so your competing engine could focus on them.  E.g. the "schemalessness" of Lucene has become a big problem, and near impossible to fix at this point, and prevents new important features like LUCENE-5879 from being possible, so you could give your engine a "gentle" schema from the start.
>
> The Lucene Filter/Query situation is a mess: one should extend the other.
>
> Lucene has weak support for proximity queries (SpanQuery is slow and does not get much attention).
>
> Lucene is showing its age, missing some compelling features like a builtin transaction log, "core" support for numerics (they are sort of hacked on top), optimistic concurrency support (sequence ids, versions, something), distributed support (near real time replication, etc.), multi-tenancy, an example server implementation, so the search servers on top of Lucene have had to fill these gaps.  Maybe you could make your engine distributed from the start (Go is a great match for that, from what little I know).
>
> All 3 highlighter options have problems.
>
> The analysis chain (attributes) is overly complex.
>
> In your competing engine you can borrow/copy/steal from Lucene's good parts to get started...
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:
>> I’m developing search engine, Fastcatsearch. http://github
>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>
>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>
>> But is there any chance to beat Lucene?
>>
>> Anything like features, performance.
>>
>> Please, let me know what to do to make better product than Lucene.
>>
>> Thank you.
>
Reply | Threaded
Open this post in threaded view
|

RE: How can I make better project than Lucene?

wmartinusa
Um, doesn't the Apache license require inclusion of the license? Just sayin'


-----Original Message-----
From: Michael McCandless [mailto:[hidden email]]
Sent: Saturday, November 15, 2014 8:47 AM
To: [hidden email]
Subject: Re: How can I make better project than Lucene?

Well the Apache Software License is very generous about poaching.

"Your ideas will go further if you don't insist on going with them."

Mike McCandless

http://blog.mikemccandless.com


On Sat, Nov 15, 2014 at 6:42 AM, Will Martin <[hidden email]> wrote:

> Btw: SwSong should not steal code; which implies an existing license whose terms he is willing to break. Not a good first step.    ;-)
>
> will
>
> -----Original Message-----
> From: Michael McCandless [mailto:[hidden email]]
> Sent: Saturday, November 15, 2014 6:22 AM
> To: [hidden email]
> Subject: Re: How can I make better project than Lucene?
>
> Actually I think competing projects is very healthy for open source development.
>
> There are many things you could explore to "contrast" with Lucene, e.g. write your new search engine in Go not Java: Java has many problems, maybe Go fixes them.  Go also has a low-latency garbage collector in development ... and Java's GC options still can't scale to the heap sizes that are practical now.
>
> Lucene has many limitations, so your competing engine could focus on them.  E.g. the "schemalessness" of Lucene has become a big problem, and near impossible to fix at this point, and prevents new important features like LUCENE-5879 from being possible, so you could give your engine a "gentle" schema from the start.
>
> The Lucene Filter/Query situation is a mess: one should extend the other.
>
> Lucene has weak support for proximity queries (SpanQuery is slow and does not get much attention).
>
> Lucene is showing its age, missing some compelling features like a builtin transaction log, "core" support for numerics (they are sort of hacked on top), optimistic concurrency support (sequence ids, versions, something), distributed support (near real time replication, etc.), multi-tenancy, an example server implementation, so the search servers on top of Lucene have had to fill these gaps.  Maybe you could make your engine distributed from the start (Go is a great match for that, from what little I know).
>
> All 3 highlighter options have problems.
>
> The analysis chain (attributes) is overly complex.
>
> In your competing engine you can borrow/copy/steal from Lucene's good parts to get started...
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:
>> I’m developing search engine, Fastcatsearch. http://github 
>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>
>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>
>> But is there any chance to beat Lucene?
>>
>> Anything like features, performance.
>>
>> Please, let me know what to do to make better product than Lucene.
>>
>> Thank you.
>

Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

Michael McCandless-2
Yes it does.

Mike McCandless

http://blog.mikemccandless.com


On Sat, Nov 15, 2014 at 8:53 AM, Will Martin <[hidden email]> wrote:

> Um, doesn't the Apache license require inclusion of the license? Just sayin'
>
>
> -----Original Message-----
> From: Michael McCandless [mailto:[hidden email]]
> Sent: Saturday, November 15, 2014 8:47 AM
> To: [hidden email]
> Subject: Re: How can I make better project than Lucene?
>
> Well the Apache Software License is very generous about poaching.
>
> "Your ideas will go further if you don't insist on going with them."
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Sat, Nov 15, 2014 at 6:42 AM, Will Martin <[hidden email]> wrote:
>> Btw: SwSong should not steal code; which implies an existing license whose terms he is willing to break. Not a good first step.    ;-)
>>
>> will
>>
>> -----Original Message-----
>> From: Michael McCandless [mailto:[hidden email]]
>> Sent: Saturday, November 15, 2014 6:22 AM
>> To: [hidden email]
>> Subject: Re: How can I make better project than Lucene?
>>
>> Actually I think competing projects is very healthy for open source development.
>>
>> There are many things you could explore to "contrast" with Lucene, e.g. write your new search engine in Go not Java: Java has many problems, maybe Go fixes them.  Go also has a low-latency garbage collector in development ... and Java's GC options still can't scale to the heap sizes that are practical now.
>>
>> Lucene has many limitations, so your competing engine could focus on them.  E.g. the "schemalessness" of Lucene has become a big problem, and near impossible to fix at this point, and prevents new important features like LUCENE-5879 from being possible, so you could give your engine a "gentle" schema from the start.
>>
>> The Lucene Filter/Query situation is a mess: one should extend the other.
>>
>> Lucene has weak support for proximity queries (SpanQuery is slow and does not get much attention).
>>
>> Lucene is showing its age, missing some compelling features like a builtin transaction log, "core" support for numerics (they are sort of hacked on top), optimistic concurrency support (sequence ids, versions, something), distributed support (near real time replication, etc.), multi-tenancy, an example server implementation, so the search servers on top of Lucene have had to fill these gaps.  Maybe you could make your engine distributed from the start (Go is a great match for that, from what little I know).
>>
>> All 3 highlighter options have problems.
>>
>> The analysis chain (attributes) is overly complex.
>>
>> In your competing engine you can borrow/copy/steal from Lucene's good parts to get started...
>>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:
>>> I’m developing search engine, Fastcatsearch. http://github
>>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>>
>>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>>
>>> But is there any chance to beat Lucene?
>>>
>>> Anything like features, performance.
>>>
>>> Please, let me know what to do to make better product than Lucene.
>>>
>>> Thank you.
>>
>
Reply | Threaded
Open this post in threaded view
|

RE: How can I make better project than Lucene?

wmartinusa
In reply to this post by Siva Thumma
Comments inline:

===

-----Original Message-----
From: Siva Thumma [mailto:[hidden email]]
Sent: Saturday, November 15, 2014 8:06 AM
To: [hidden email]
Subject: Re: How can I make better project than Lucene?

To build such a big product, One would obviously attribute the license.

Sent from iPhone

> On 15-Nov-2014, at 5:12 pm, Will Martin <[hidden email]> wrote:
>
> Btw: SwSong should not steal code; which implies an existing license whose terms he is willing to break. Not a good first step.    ;-)
>
> will
>
> -----Original Message-----
> From: Michael McCandless [mailto:[hidden email]]
> Sent: Saturday, November 15, 2014 6:22 AM
> To: [hidden email]
> Subject: Re: How can I make better project than Lucene?
>
> Actually I think competing projects is very healthy for open source development.
>
> There are many things you could explore to "contrast" with Lucene, e.g. write your new search engine in Go not Java: Java has many problems, maybe Go fixes them.  Go also has a low-latency garbage collector in development ... and Java's GC options still can't scale to the heap sizes that are practical now.

:::>wmartin: if there is a problem with GC for a domain then the jdk team should be contacted or our index design maybe revisited.


>
> Lucene has many limitations, so your competing engine could focus on them.  E.g. the "schemalessness" of Lucene has become a big problem, and near impossible to fix at this point, and prevents new important features like LUCENE-5879 from being possible, so you could give your engine a "gentle" schema from the start.

:::> I'm always amazed when I find references to fieldnames...rather than enums or ids. A scehma should and often does in Lucene, result in an automata or maybe (fst)....so why isn't the schema implemented as such? Too slow?




>
> The Lucene Filter/Query situation is a mess: one should extend the other.
>

:::> um doesn't IntellliJ JetBrains refactor? Is it too dumb?


> Lucene has weak support for proximity queries (SpanQuery is slow and does not get much attention).
>

:::> I wrote proximity for CPL (DataTimes, DOW-JONES, LoC, AOL). Give me more information.


> Lucene is showing its age, missing some compelling features like a builtin transaction log, "core" support for numerics (they are sort of hacked on top), optimistic concurrency support (sequence ids, versions, something), distributed support (near real time replication, etc.), multi-tenancy, an example server implementation, so the search servers on top of Lucene have had to fill these gaps.  Maybe you could make your engine distributed from the start (Go is a great match for that, from what little I know).
>
> All 3 highlighter options have problems.
>

:::> Well since Postings uses a plain-jane tune of BM25raw and uses internediaries to read posts,  its not surprising. Question is maybe the first thing that should be done Is profile the damn things.  The DAPO (DAGO) benchmark framework has lucene search and indexing. Maybe an extension to the search collector there.





> The analysis chain (attributes) is overly complex.
>
> In your competing engine you can borrow/copy/steal from Lucene's good parts to get started...
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
>> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:
>> I’m developing search engine, Fastcatsearch. http://github 
>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>
>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>
>> But is there any chance to beat Lucene?
>>
>> Anything like features, performance.
>>
>> Please, let me know what to do to make better product than Lucene.
>>
>> Thank you.
>

Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

sangwook DEV
In reply to this post by Michael McCandless-2
Yes, I borrows Lucene’s source code(analysis, store, utils ) partly.

My search engine is LGPL, and of course I leaved a Apache license text header in all source codes borrowed from Lucene.

Sang Song


> 2014. 11. 15., 오후 10:56, Michael McCandless <[hidden email]> 작성:
>
> Yes it does.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Sat, Nov 15, 2014 at 8:53 AM, Will Martin <[hidden email]> wrote:
>> Um, doesn't the Apache license require inclusion of the license? Just sayin'
>>
>>
>> -----Original Message-----
>> From: Michael McCandless [mailto:[hidden email]]
>> Sent: Saturday, November 15, 2014 8:47 AM
>> To: [hidden email]
>> Subject: Re: How can I make better project than Lucene?
>>
>> Well the Apache Software License is very generous about poaching.
>>
>> "Your ideas will go further if you don't insist on going with them."
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Sat, Nov 15, 2014 at 6:42 AM, Will Martin <[hidden email]> wrote:
>>> Btw: SwSong should not steal code; which implies an existing license whose terms he is willing to break. Not a good first step.    ;-)
>>>
>>> will
>>>
>>> -----Original Message-----
>>> From: Michael McCandless [mailto:[hidden email]]
>>> Sent: Saturday, November 15, 2014 6:22 AM
>>> To: [hidden email]
>>> Subject: Re: How can I make better project than Lucene?
>>>
>>> Actually I think competing projects is very healthy for open source development.
>>>
>>> There are many things you could explore to "contrast" with Lucene, e.g. write your new search engine in Go not Java: Java has many problems, maybe Go fixes them.  Go also has a low-latency garbage collector in development ... and Java's GC options still can't scale to the heap sizes that are practical now.
>>>
>>> Lucene has many limitations, so your competing engine could focus on them.  E.g. the "schemalessness" of Lucene has become a big problem, and near impossible to fix at this point, and prevents new important features like LUCENE-5879 from being possible, so you could give your engine a "gentle" schema from the start.
>>>
>>> The Lucene Filter/Query situation is a mess: one should extend the other.
>>>
>>> Lucene has weak support for proximity queries (SpanQuery is slow and does not get much attention).
>>>
>>> Lucene is showing its age, missing some compelling features like a builtin transaction log, "core" support for numerics (they are sort of hacked on top), optimistic concurrency support (sequence ids, versions, something), distributed support (near real time replication, etc.), multi-tenancy, an example server implementation, so the search servers on top of Lucene have had to fill these gaps.  Maybe you could make your engine distributed from the start (Go is a great match for that, from what little I know).
>>>
>>> All 3 highlighter options have problems.
>>>
>>> The analysis chain (attributes) is overly complex.
>>>
>>> In your competing engine you can borrow/copy/steal from Lucene's good parts to get started...
>>>
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>>
>>> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:
>>>> I’m developing search engine, Fastcatsearch. http://github
>>>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>>>
>>>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>>>
>>>> But is there any chance to beat Lucene?
>>>>
>>>> Anything like features, performance.
>>>>
>>>> Please, let me know what to do to make better product than Lucene.
>>>>
>>>> Thank you.
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

sangwook DEV
In reply to this post by Michael McCandless-2
Thank you for your sincere reply, Mr. McCandless.

When I posted an email in a mailing-list, I was afraid for not getting a considerable reply, but I’m now so glad I might find a way.

I agree that a new search engine in Go would be competitive. I think we all need a next generation search engine core redesigned from the start.

And, I understand Lucene’s limitations you mentioned. They are good points to get started.

I have been used a search engine for first 4 years and developed a search engines for last 6 years from the bottom, and I got feedback “It’s faster than Solr in indexing and searching”. (http://ddakker.tistory.com/248 <http://ddakker.tistory.com/248>)

===Result===
Data size : 529,188
Fastcat indexing time : 1m 26s
Solr3.5 indexing time : 5m 30s
Fastcat searching time : 48ms
Solr3.5 searching time : 73ms

It applied to Korea’s greatest shopping service(http://danawa.com/ <http://danawa.com/>) a month ago to my delight.

But my goal has been making a globally-used open source search engine.

As you suggested, now I want to make a whole-new search engine in Go.

I have made my first search engine alone, but I would not make a new search engine alone. I want to make it with global developers together.

If you plan to make a new search engine in Go, or know someone around you, could you help me gathering members for a new search engine, and guide us technically(feature requirement, efficient design)?

Or if there is already a new search engine project in Go, could you let me know?

In Korea, no one develops a search engine except people who work at a search engine solution company, and even they are very few and do not spend time to an open source project.

In my case, I found a tiny venture company for making time to develop an open source search engine 4 year ago.

I want to be involved in a next-generation search engine project. I would be happy to make a new search engine itself.

Your little help could be great for me.

Thank you.

Sang Song


> 2014. 11. 15., 오후 8:22, Michael McCandless <[hidden email]> 작성:
>
> Actually I think competing projects is very healthy for open source development.
>
> There are many things you could explore to "contrast" with Lucene,
> e.g. write your new search engine in Go not Java: Java has many
> problems, maybe Go fixes them.  Go also has a low-latency garbage
> collector in development ... and Java's GC options still can't scale
> to the heap sizes that are practical now.
>
> Lucene has many limitations, so your competing engine could focus on
> them.  E.g. the "schemalessness" of Lucene has become a big problem,
> and near impossible to fix at this point, and prevents new important
> features like LUCENE-5879 from being possible, so you could give your
> engine a "gentle" schema from the start.
>
> The Lucene Filter/Query situation is a mess: one should extend the other.
>
> Lucene has weak support for proximity queries (SpanQuery is slow and
> does not get much attention).
>
> Lucene is showing its age, missing some compelling features like a
> builtin transaction log, "core" support for numerics (they are sort of
> hacked on top), optimistic concurrency support (sequence ids,
> versions, something), distributed support (near real time replication,
> etc.), multi-tenancy, an example server implementation, so the search
> servers on top of Lucene have had to fill these gaps.  Maybe you could
> make your engine distributed from the start (Go is a great match for
> that, from what little I know).
>
> All 3 highlighter options have problems.
>
> The analysis chain (attributes) is overly complex.
>
> In your competing engine you can borrow/copy/steal from Lucene's good
> parts to get started...
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:
>> I’m developing search engine, Fastcatsearch. http://github <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>
>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>
>> But is there any chance to beat Lucene?
>>
>> Anything like features, performance.
>>
>> Please, let me know what to do to make better product than Lucene.
>>
>> Thank you.

Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

Erik Hatcher-4
I’m curious to see these benchmarks run on the latest Solr version, as the numbers you quoted were over two years ago.  Also, it’d be useful to see the indexing and searching benchmark code to make sure it takes advantage of best practices.   I’ve indexed 10M docs into Solr in only a few minutes.  500K, say in CSV format, for basic e-commerce product data, would likely take a minute or so.  The searching differences you present seem fairly negligible - 100ms is the blink of an eye, so anything under is considered quite acceptable by the largest e-commerce vendors in the world.  Along with that, perhaps an even more important benchmark is relevancy or in some way measure how good the search results are.

As Mike put so well, competition is a great thing so by all means I encourage you to carry on with your endeavor.  Sounds like you’ve built some powerful stuff and have extensive experience.  +1

        Erik



On Nov 15, 2014, at 6:23 PM, swsong_dev <[hidden email]> wrote:

> Thank you for your sincere reply, Mr. McCandless.
>
> When I posted an email in a mailing-list, I was afraid for not getting a considerable reply, but I’m now so glad I might find a way.
>
> I agree that a new search engine in Go would be competitive. I think we all need a next generation search engine core redesigned from the start.
>
> And, I understand Lucene’s limitations you mentioned. They are good points to get started.
>
> I have been used a search engine for first 4 years and developed a search engines for last 6 years from the bottom, and I got feedback “It’s faster than Solr in indexing and searching”. (http://ddakker.tistory.com/248 <http://ddakker.tistory.com/248>)
>
> ===Result===
> Data size : 529,188
> Fastcat indexing time : 1m 26s
> Solr3.5 indexing time : 5m 30s
> Fastcat searching time : 48ms
> Solr3.5 searching time : 73ms
>
> It applied to Korea’s greatest shopping service(http://danawa.com/ <http://danawa.com/>) a month ago to my delight.
>
> But my goal has been making a globally-used open source search engine.
>
> As you suggested, now I want to make a whole-new search engine in Go.
>
> I have made my first search engine alone, but I would not make a new search engine alone. I want to make it with global developers together.
>
> If you plan to make a new search engine in Go, or know someone around you, could you help me gathering members for a new search engine, and guide us technically(feature requirement, efficient design)?
>
> Or if there is already a new search engine project in Go, could you let me know?
>
> In Korea, no one develops a search engine except people who work at a search engine solution company, and even they are very few and do not spend time to an open source project.
>
> In my case, I found a tiny venture company for making time to develop an open source search engine 4 year ago.
>
> I want to be involved in a next-generation search engine project. I would be happy to make a new search engine itself.
>
> Your little help could be great for me.
>
> Thank you.
>
> Sang Song
>
>
>> 2014. 11. 15., 오후 8:22, Michael McCandless <[hidden email]> 작성:
>>
>> Actually I think competing projects is very healthy for open source development.
>>
>> There are many things you could explore to "contrast" with Lucene,
>> e.g. write your new search engine in Go not Java: Java has many
>> problems, maybe Go fixes them.  Go also has a low-latency garbage
>> collector in development ... and Java's GC options still can't scale
>> to the heap sizes that are practical now.
>>
>> Lucene has many limitations, so your competing engine could focus on
>> them.  E.g. the "schemalessness" of Lucene has become a big problem,
>> and near impossible to fix at this point, and prevents new important
>> features like LUCENE-5879 from being possible, so you could give your
>> engine a "gentle" schema from the start.
>>
>> The Lucene Filter/Query situation is a mess: one should extend the other.
>>
>> Lucene has weak support for proximity queries (SpanQuery is slow and
>> does not get much attention).
>>
>> Lucene is showing its age, missing some compelling features like a
>> builtin transaction log, "core" support for numerics (they are sort of
>> hacked on top), optimistic concurrency support (sequence ids,
>> versions, something), distributed support (near real time replication,
>> etc.), multi-tenancy, an example server implementation, so the search
>> servers on top of Lucene have had to fill these gaps.  Maybe you could
>> make your engine distributed from the start (Go is a great match for
>> that, from what little I know).
>>
>> All 3 highlighter options have problems.
>>
>> The analysis chain (attributes) is overly complex.
>>
>> In your competing engine you can borrow/copy/steal from Lucene's good
>> parts to get started...
>>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]> wrote:
>>> I’m developing search engine, Fastcatsearch. http://github <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>>
>>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>>>
>>> But is there any chance to beat Lucene?
>>>
>>> Anything like features, performance.
>>>
>>> Please, let me know what to do to make better product than Lucene.
>>>
>>> Thank you.
>

Reply | Threaded
Open this post in threaded view
|

RE: How can I make better project than Lucene?

wmartinusa
In reply to this post by sangwook DEV
Please let me know if you go the GO route, so I can choose the language I
like best for my work.  Wmartinusa   at   google online mail.

-----Original Message-----
From: swsong_dev [mailto:[hidden email]]
Sent: Saturday, November 15, 2014 6:23 PM
To: [hidden email]
Subject: Re: How can I make better project than Lucene?

Thank you for your sincere reply, Mr. McCandless.

When I posted an email in a mailing-list, I was afraid for not getting a
considerable reply, but I’m now so glad I might find a way.

I agree that a new search engine in Go would be competitive. I think we all
need a next generation search engine core redesigned from the start.

And, I understand Lucene’s limitations you mentioned. They are good points
to get started.

I have been used a search engine for first 4 years and developed a search
engines for last 6 years from the bottom, and I got feedback “It’s faster
than Solr in indexing and searching”. (http://ddakker.tistory.com/248
<http://ddakker.tistory.com/248>)

===Result===
Data size : 529,188
Fastcat indexing time : 1m 26s
Solr3.5 indexing time : 5m 30s
Fastcat searching time : 48ms
Solr3.5 searching time : 73ms

It applied to Korea’s greatest shopping service(http://danawa.com/
<http://danawa.com/>) a month ago to my delight.

But my goal has been making a globally-used open source search engine.

As you suggested, now I want to make a whole-new search engine in Go.

I have made my first search engine alone, but I would not make a new search
engine alone. I want to make it with global developers together.

If you plan to make a new search engine in Go, or know someone around you,
could you help me gathering members for a new search engine, and guide us
technically(feature requirement, efficient design)?

Or if there is already a new search engine project in Go, could you let me
know?

In Korea, no one develops a search engine except people who work at a
search engine solution company, and even they are very few and do not spend
time to an open source project.

In my case, I found a tiny venture company for making time to develop an
open source search engine 4 year ago.

I want to be involved in a next-generation search engine project. I would
be happy to make a new search engine itself.

Your little help could be great for me.

Thank you.

Sang Song


> 2014. 11. 15., 오후 8:22, Michael McCandless <[hidden email]>
작성:
>
> Actually I think competing projects is very healthy for open source
development.

>
> There are many things you could explore to "contrast" with Lucene,
> e.g. write your new search engine in Go not Java: Java has many
> problems, maybe Go fixes them.  Go also has a low-latency garbage
> collector in development ... and Java's GC options still can't scale
> to the heap sizes that are practical now.
>
> Lucene has many limitations, so your competing engine could focus on
> them.  E.g. the "schemalessness" of Lucene has become a big problem,
> and near impossible to fix at this point, and prevents new important
> features like LUCENE-5879 from being possible, so you could give your
> engine a "gentle" schema from the start.
>
> The Lucene Filter/Query situation is a mess: one should extend the other.
>
> Lucene has weak support for proximity queries (SpanQuery is slow and
> does not get much attention).
>
> Lucene is showing its age, missing some compelling features like a
> builtin transaction log, "core" support for numerics (they are sort of
> hacked on top), optimistic concurrency support (sequence ids,
> versions, something), distributed support (near real time replication,
> etc.), multi-tenancy, an example server implementation, so the search
> servers on top of Lucene have had to fill these gaps.  Maybe you could
> make your engine distributed from the start (Go is a great match for
> that, from what little I know).
>
> All 3 highlighter options have problems.
>
> The analysis chain (attributes) is overly complex.
>
> In your competing engine you can borrow/copy/steal from Lucene's good
> parts to get started...
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]>
wrote:
>> I’m developing search engine, Fastcatsearch. http://github 
>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>
>> Lucene is widely known and famous project and I cannot beat Lucene for
now.
>>
>> But is there any chance to beat Lucene?
>>
>> Anything like features, performance.
>>
>> Please, let me know what to do to make better product than Lucene.
>>
>> Thank you.


Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

sangwook DEV
Do you mean you want to make a new search engine with me in Go?
I don't understand what you mean exactly..

2014년 11월 16일 일요일, Will Martin<[hidden email]>님이 작성한 메시지:

> Please let me know if you go the GO route, so I can choose the language I
> like best for my work.  Wmartinusa   at   google online mail.
>
> -----Original Message-----
> From: swsong_dev [mailto:[hidden email] <javascript:;>]
> Sent: Saturday, November 15, 2014 6:23 PM
> To: [hidden email] <javascript:;>
> Subject: Re: How can I make better project than Lucene?
>
> Thank you for your sincere reply, Mr. McCandless.
>
> When I posted an email in a mailing-list, I was afraid for not getting a
> considerable reply, but I’m now so glad I might find a way.
>
> I agree that a new search engine in Go would be competitive. I think we all
> need a next generation search engine core redesigned from the start.
>
> And, I understand Lucene’s limitations you mentioned. They are good points
> to get started.
>
> I have been used a search engine for first 4 years and developed a search
> engines for last 6 years from the bottom, and I got feedback “It’s faster
> than Solr in indexing and searching”. (http://ddakker.tistory.com/248
> <http://ddakker.tistory.com/248>)
>
> ===Result===
> Data size : 529,188
> Fastcat indexing time : 1m 26s
> Solr3.5 indexing time : 5m 30s
> Fastcat searching time : 48ms
> Solr3.5 searching time : 73ms
>
> It applied to Korea’s greatest shopping service(http://danawa.com/
> <http://danawa.com/>) a month ago to my delight.
>
> But my goal has been making a globally-used open source search engine.
>
> As you suggested, now I want to make a whole-new search engine in Go.
>
> I have made my first search engine alone, but I would not make a new search
> engine alone. I want to make it with global developers together.
>
> If you plan to make a new search engine in Go, or know someone around you,
> could you help me gathering members for a new search engine, and guide us
> technically(feature requirement, efficient design)?
>
> Or if there is already a new search engine project in Go, could you let me
> know?
>
> In Korea, no one develops a search engine except people who work at a
> search engine solution company, and even they are very few and do not spend
> time to an open source project.
>
> In my case, I found a tiny venture company for making time to develop an
> open source search engine 4 year ago.
>
> I want to be involved in a next-generation search engine project. I would
> be happy to make a new search engine itself.
>
> Your little help could be great for me.
>
> Thank you.
>
> Sang Song
>
>
> > 2014. 11. 15., 오후 8:22, Michael McCandless <[hidden email]
> <javascript:;>>
> 작성:
> >
> > Actually I think competing projects is very healthy for open source
> development.
> >
> > There are many things you could explore to "contrast" with Lucene,
> > e.g. write your new search engine in Go not Java: Java has many
> > problems, maybe Go fixes them.  Go also has a low-latency garbage
> > collector in development ... and Java's GC options still can't scale
> > to the heap sizes that are practical now.
> >
> > Lucene has many limitations, so your competing engine could focus on
> > them.  E.g. the "schemalessness" of Lucene has become a big problem,
> > and near impossible to fix at this point, and prevents new important
> > features like LUCENE-5879 from being possible, so you could give your
> > engine a "gentle" schema from the start.
> >
> > The Lucene Filter/Query situation is a mess: one should extend the other.
> >
> > Lucene has weak support for proximity queries (SpanQuery is slow and
> > does not get much attention).
> >
> > Lucene is showing its age, missing some compelling features like a
> > builtin transaction log, "core" support for numerics (they are sort of
> > hacked on top), optimistic concurrency support (sequence ids,
> > versions, something), distributed support (near real time replication,
> > etc.), multi-tenancy, an example server implementation, so the search
> > servers on top of Lucene have had to fill these gaps.  Maybe you could
> > make your engine distributed from the start (Go is a great match for
> > that, from what little I know).
> >
> > All 3 highlighter options have problems.
> >
> > The analysis chain (attributes) is overly complex.
> >
> > In your competing engine you can borrow/copy/steal from Lucene's good
> > parts to get started...
> >
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]
> <javascript:;>>
> wrote:
> >> I’m developing search engine, Fastcatsearch. http://github
> >> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
> >>
> >> Lucene is widely known and famous project and I cannot beat Lucene for
> now.
> >>
> >> But is there any chance to beat Lucene?
> >>
> >> Anything like features, performance.
> >>
> >> Please, let me know what to do to make better product than Lucene.
> >>
> >> Thank you.
>
>
>
Reply | Threaded
Open this post in threaded view
|

RE: How can I make better project than Lucene?

wmartinusa
I don't know what I think about an engine in Go.

:-(


-----Original Message-----
From: sangwook DEV [mailto:[hidden email]]
Sent: Sunday, November 16, 2014 2:52 AM
To: [hidden email]
Subject: Re: How can I make better project than Lucene?

Do you mean you want to make a new search engine with me in Go?
I don't understand what you mean exactly..

2014년 11월 16일 일요일, Will Martin<[hidden email]>님이 작성한 메시지:

> Please let me know if you go the GO route, so I can choose the language I
> like best for my work.  Wmartinusa   at   google online mail.
>
> -----Original Message-----
> From: swsong_dev [mailto:[hidden email] <javascript:;>]
> Sent: Saturday, November 15, 2014 6:23 PM
> To: [hidden email] <javascript:;>
> Subject: Re: How can I make better project than Lucene?
>
> Thank you for your sincere reply, Mr. McCandless.
>
> When I posted an email in a mailing-list, I was afraid for not getting
> a considerable reply, but I’m now so glad I might find a way.
>
> I agree that a new search engine in Go would be competitive. I think
> we all need a next generation search engine core redesigned from the start.
>
> And, I understand Lucene’s limitations you mentioned. They are good
> points to get started.
>
> I have been used a search engine for first 4 years and developed a
> search engines for last 6 years from the bottom, and I got feedback
> “It’s faster than Solr in indexing and searching”.
> (http://ddakker.tistory.com/248
> <http://ddakker.tistory.com/248>)
>
> ===Result===
> Data size : 529,188
> Fastcat indexing time : 1m 26s
> Solr3.5 indexing time : 5m 30s
> Fastcat searching time : 48ms
> Solr3.5 searching time : 73ms
>
> It applied to Korea’s greatest shopping service(http://danawa.com/
> <http://danawa.com/>) a month ago to my delight.
>
> But my goal has been making a globally-used open source search engine.
>
> As you suggested, now I want to make a whole-new search engine in Go.
>
> I have made my first search engine alone, but I would not make a new
> search engine alone. I want to make it with global developers together.
>
> If you plan to make a new search engine in Go, or know someone around
> you, could you help me gathering members for a new search engine, and
> guide us technically(feature requirement, efficient design)?
>
> Or if there is already a new search engine project in Go, could you
> let me know?
>
> In Korea, no one develops a search engine except people who work at a
> search engine solution company, and even they are very few and do not
> spend time to an open source project.
>
> In my case, I found a tiny venture company for making time to develop
> an open source search engine 4 year ago.
>
> I want to be involved in a next-generation search engine project. I
> would be happy to make a new search engine itself.
>
> Your little help could be great for me.
>
> Thank you.
>
> Sang Song
>
>
> > 2014. 11. 15., 오후 8:22, Michael McCandless
> > <[hidden email]
> <javascript:;>>
> 작성:
> >
> > Actually I think competing projects is very healthy for open source
> development.
> >
> > There are many things you could explore to "contrast" with Lucene,
> > e.g. write your new search engine in Go not Java: Java has many
> > problems, maybe Go fixes them.  Go also has a low-latency garbage
> > collector in development ... and Java's GC options still can't scale
> > to the heap sizes that are practical now.
> >
> > Lucene has many limitations, so your competing engine could focus on
> > them.  E.g. the "schemalessness" of Lucene has become a big problem,
> > and near impossible to fix at this point, and prevents new important
> > features like LUCENE-5879 from being possible, so you could give
> > your engine a "gentle" schema from the start.
> >
> > The Lucene Filter/Query situation is a mess: one should extend the other.
> >
> > Lucene has weak support for proximity queries (SpanQuery is slow and
> > does not get much attention).
> >
> > Lucene is showing its age, missing some compelling features like a
> > builtin transaction log, "core" support for numerics (they are sort
> > of hacked on top), optimistic concurrency support (sequence ids,
> > versions, something), distributed support (near real time
> > replication, etc.), multi-tenancy, an example server implementation,
> > so the search servers on top of Lucene have had to fill these gaps.  
> > Maybe you could make your engine distributed from the start (Go is a
> > great match for that, from what little I know).
> >
> > All 3 highlighter options have problems.
> >
> > The analysis chain (attributes) is overly complex.
> >
> > In your competing engine you can borrow/copy/steal from Lucene's
> > good parts to get started...
> >
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]
> <javascript:;>>
> wrote:
> >> I’m developing search engine, Fastcatsearch. http://github 
> >> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
> >>
> >> Lucene is widely known and famous project and I cannot beat Lucene
> >> for
> now.
> >>
> >> But is there any chance to beat Lucene?
> >>
> >> Anything like features, performance.
> >>
> >> Please, let me know what to do to make better product than Lucene.
> >>
> >> Thank you.
>
>
>

Reply | Threaded
Open this post in threaded view
|

RE: How can I make better project than Lucene?

wmartinusa
In reply to this post by sangwook DEV
Actually I'm more interested in the list Mike wrote about lucene.


-----Original Message-----
From: sangwook DEV [mailto:[hidden email]]
Sent: Sunday, November 16, 2014 2:52 AM
To: [hidden email]
Subject: Re: How can I make better project than Lucene?

Do you mean you want to make a new search engine with me in Go?
I don't understand what you mean exactly..

2014년 11월 16일 일요일, Will Martin<[hidden email]>님이 작성한 메시지:

> Please let me know if you go the GO route, so I can choose the language I
> like best for my work.  Wmartinusa   at   google online mail.
>
> -----Original Message-----
> From: swsong_dev [mailto:[hidden email] <javascript:;>]
> Sent: Saturday, November 15, 2014 6:23 PM
> To: [hidden email] <javascript:;>
> Subject: Re: How can I make better project than Lucene?
>
> Thank you for your sincere reply, Mr. McCandless.
>
> When I posted an email in a mailing-list, I was afraid for not getting
> a considerable reply, but I’m now so glad I might find a way.
>
> I agree that a new search engine in Go would be competitive. I think
> we all need a next generation search engine core redesigned from the start.
>
> And, I understand Lucene’s limitations you mentioned. They are good
> points to get started.
>
> I have been used a search engine for first 4 years and developed a
> search engines for last 6 years from the bottom, and I got feedback
> “It’s faster than Solr in indexing and searching”.
> (http://ddakker.tistory.com/248
> <http://ddakker.tistory.com/248>)
>
> ===Result===
> Data size : 529,188
> Fastcat indexing time : 1m 26s
> Solr3.5 indexing time : 5m 30s
> Fastcat searching time : 48ms
> Solr3.5 searching time : 73ms
>
> It applied to Korea’s greatest shopping service(http://danawa.com/
> <http://danawa.com/>) a month ago to my delight.
>
> But my goal has been making a globally-used open source search engine.
>
> As you suggested, now I want to make a whole-new search engine in Go.
>
> I have made my first search engine alone, but I would not make a new
> search engine alone. I want to make it with global developers together.
>
> If you plan to make a new search engine in Go, or know someone around
> you, could you help me gathering members for a new search engine, and
> guide us technically(feature requirement, efficient design)?
>
> Or if there is already a new search engine project in Go, could you
> let me know?
>
> In Korea, no one develops a search engine except people who work at a
> search engine solution company, and even they are very few and do not
> spend time to an open source project.
>
> In my case, I found a tiny venture company for making time to develop
> an open source search engine 4 year ago.
>
> I want to be involved in a next-generation search engine project. I
> would be happy to make a new search engine itself.
>
> Your little help could be great for me.
>
> Thank you.
>
> Sang Song
>
>
> > 2014. 11. 15., 오후 8:22, Michael McCandless
> > <[hidden email]
> <javascript:;>>
> 작성:
> >
> > Actually I think competing projects is very healthy for open source
> development.
> >
> > There are many things you could explore to "contrast" with Lucene,
> > e.g. write your new search engine in Go not Java: Java has many
> > problems, maybe Go fixes them.  Go also has a low-latency garbage
> > collector in development ... and Java's GC options still can't scale
> > to the heap sizes that are practical now.
> >
> > Lucene has many limitations, so your competing engine could focus on
> > them.  E.g. the "schemalessness" of Lucene has become a big problem,
> > and near impossible to fix at this point, and prevents new important
> > features like LUCENE-5879 from being possible, so you could give
> > your engine a "gentle" schema from the start.
> >
> > The Lucene Filter/Query situation is a mess: one should extend the other.
> >
> > Lucene has weak support for proximity queries (SpanQuery is slow and
> > does not get much attention).
> >
> > Lucene is showing its age, missing some compelling features like a
> > builtin transaction log, "core" support for numerics (they are sort
> > of hacked on top), optimistic concurrency support (sequence ids,
> > versions, something), distributed support (near real time
> > replication, etc.), multi-tenancy, an example server implementation,
> > so the search servers on top of Lucene have had to fill these gaps.  
> > Maybe you could make your engine distributed from the start (Go is a
> > great match for that, from what little I know).
> >
> > All 3 highlighter options have problems.
> >
> > The analysis chain (attributes) is overly complex.
> >
> > In your competing engine you can borrow/copy/steal from Lucene's
> > good parts to get started...
> >
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]
> <javascript:;>>
> wrote:
> >> I’m developing search engine, Fastcatsearch. http://github 
> >> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
> >>
> >> Lucene is widely known and famous project and I cannot beat Lucene
> >> for
> now.
> >>
> >> But is there any chance to beat Lucene?
> >>
> >> Anything like features, performance.
> >>
> >> Please, let me know what to do to make better product than Lucene.
> >>
> >> Thank you.
>
>
>

Reply | Threaded
Open this post in threaded view
|

RE: How can I make better project than Lucene?

wmartinusa
In reply to this post by Saurabh Tiwari
http://dx.doi.org/10.1145/2480362.2480533


-----Original Message-----
From: Saurabh Tiwari [mailto:[hidden email]]
Sent: Saturday, November 15, 2014 2:09 AM
To: [hidden email]
Subject: Re: How can I make better project than Lucene?

+1 John wang

On Saturday, November 15, 2014, John Wang <[hidden email]> wrote:

> Why not spend the effort improving Lucene?
>
> John
> On Nov 14, 2014 5:46 PM, "swsong_dev" <[hidden email]
> <javascript:;>> wrote:
>
> > I’m developing search engine, Fastcatsearch. http://github 
> > <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
> >
> > Lucene is widely known and famous project and I cannot beat Lucene
> > for
> now.
> >
> > But is there any chance to beat Lucene?
> >
> > Anything like features, performance.
> >
> > Please, let me know what to do to make better product than Lucene.
> >
> > Thank you.
>


--
Saurabh

Reply | Threaded
Open this post in threaded view
|

Re: How can I make better project than Lucene?

Lee Goddard
In reply to this post by wmartinusa
What about Rust?

On 16/11/2014 10:06, Will Martin wrote:

> I don't know what I think about an engine in Go.
>
> :-(
>
>
> -----Original Message-----
> From: sangwook DEV [mailto:[hidden email]]
> Sent: Sunday, November 16, 2014 2:52 AM
> To: [hidden email]
> Subject: Re: How can I make better project than Lucene?
>
> Do you mean you want to make a new search engine with me in Go?
> I don't understand what you mean exactly..
>
> 2014년 11월 16일 일요일, Will Martin<[hidden email]>님이 작성한 메시지:
>
>> Please let me know if you go the GO route, so I can choose the language I
>> like best for my work.  Wmartinusa   at   google online mail.
>>
>> -----Original Message-----
>> From: swsong_dev [mailto:[hidden email] <javascript:;>]
>> Sent: Saturday, November 15, 2014 6:23 PM
>> To: [hidden email] <javascript:;>
>> Subject: Re: How can I make better project than Lucene?
>>
>> Thank you for your sincere reply, Mr. McCandless.
>>
>> When I posted an email in a mailing-list, I was afraid for not getting
>> a considerable reply, but I’m now so glad I might find a way.
>>
>> I agree that a new search engine in Go would be competitive. I think
>> we all need a next generation search engine core redesigned from the start.
>>
>> And, I understand Lucene’s limitations you mentioned. They are good
>> points to get started.
>>
>> I have been used a search engine for first 4 years and developed a
>> search engines for last 6 years from the bottom, and I got feedback
>> “It’s faster than Solr in indexing and searching”.
>> (http://ddakker.tistory.com/248
>> <http://ddakker.tistory.com/248>)
>>
>> ===Result===
>> Data size : 529,188
>> Fastcat indexing time : 1m 26s
>> Solr3.5 indexing time : 5m 30s
>> Fastcat searching time : 48ms
>> Solr3.5 searching time : 73ms
>>
>> It applied to Korea’s greatest shopping service(http://danawa.com/
>> <http://danawa.com/>) a month ago to my delight.
>>
>> But my goal has been making a globally-used open source search engine.
>>
>> As you suggested, now I want to make a whole-new search engine in Go.
>>
>> I have made my first search engine alone, but I would not make a new
>> search engine alone. I want to make it with global developers together.
>>
>> If you plan to make a new search engine in Go, or know someone around
>> you, could you help me gathering members for a new search engine, and
>> guide us technically(feature requirement, efficient design)?
>>
>> Or if there is already a new search engine project in Go, could you
>> let me know?
>>
>> In Korea, no one develops a search engine except people who work at a
>> search engine solution company, and even they are very few and do not
>> spend time to an open source project.
>>
>> In my case, I found a tiny venture company for making time to develop
>> an open source search engine 4 year ago.
>>
>> I want to be involved in a next-generation search engine project. I
>> would be happy to make a new search engine itself.
>>
>> Your little help could be great for me.
>>
>> Thank you.
>>
>> Sang Song
>>
>>
>>> 2014. 11. 15., 오후 8:22, Michael McCandless
>>> <[hidden email]
>> <javascript:;>>
>> 작성:
>>> Actually I think competing projects is very healthy for open source
>> development.
>>> There are many things you could explore to "contrast" with Lucene,
>>> e.g. write your new search engine in Go not Java: Java has many
>>> problems, maybe Go fixes them.  Go also has a low-latency garbage
>>> collector in development ... and Java's GC options still can't scale
>>> to the heap sizes that are practical now.
>>>
>>> Lucene has many limitations, so your competing engine could focus on
>>> them.  E.g. the "schemalessness" of Lucene has become a big problem,
>>> and near impossible to fix at this point, and prevents new important
>>> features like LUCENE-5879 from being possible, so you could give
>>> your engine a "gentle" schema from the start.
>>>
>>> The Lucene Filter/Query situation is a mess: one should extend the other.
>>>
>>> Lucene has weak support for proximity queries (SpanQuery is slow and
>>> does not get much attention).
>>>
>>> Lucene is showing its age, missing some compelling features like a
>>> builtin transaction log, "core" support for numerics (they are sort
>>> of hacked on top), optimistic concurrency support (sequence ids,
>>> versions, something), distributed support (near real time
>>> replication, etc.), multi-tenancy, an example server implementation,
>>> so the search servers on top of Lucene have had to fill these gaps.
>>> Maybe you could make your engine distributed from the start (Go is a
>>> great match for that, from what little I know).
>>>
>>> All 3 highlighter options have problems.
>>>
>>> The analysis chain (attributes) is overly complex.
>>>
>>> In your competing engine you can borrow/copy/steal from Lucene's
>>> good parts to get started...
>>>
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>>
>>> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <[hidden email]
>> <javascript:;>>
>> wrote:
>>>> I’m developing search engine, Fastcatsearch. http://github
>>>> <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>>>>
>>>> Lucene is widely known and famous project and I cannot beat Lucene
>>>> for
>> now.
>>>> But is there any chance to beat Lucene?
>>>>
>>>> Anything like features, performance.
>>>>
>>>> Please, let me know what to do to make better product than Lucene.
>>>>
>>>> Thank you.
>>
>>
>

12