[lucy-dev] Port to JavaScript with emscripten

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[lucy-dev] Port to JavaScript with emscripten

Sebastián Gurin
Hello,

I wonder if somebody tried to compile lucy to JavaScript with
http://emscripten.org (C/C++ to JavaScript compiler). There are some
successful projects like https://github.com/medialize/sass.js/ to
bring new technologies to the JavaScript world from C/C++. IN the case
of lucene-like text indexer / search - this is for sure an area that
JavaScript lacks software. Also I wonder if the clownfish thing won't
make it difficult to perform such translation. So, before I tried, I
would like to know your thoughts.

Thanks! and BTW, lucy is very good ! Keep it up!
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-dev] Port to JavaScript with emscripten

Marvin Humphrey
On Sun, Oct 29, 2017 at 12:48 PM, Sebastián Gurin
<[hidden email]> wrote:

> I wonder if somebody tried to compile lucy to JavaScript with
> http://emscripten.org (C/C++ to JavaScript compiler).

I am not aware of such an effort.

> Also I wonder if the clownfish thing won't
> make it difficult to perform such translation. So, before I tried, I
> would like to know your thoughts.

Clownfish would probably take some effort to adapt for this purpose.
If I were to take on this task (and I did not want to spend the effort
to familiarize myself with Clownfish first), I'd treat it like a loose
port: create an empty directory alongside Lucy and flesh out new .c
and .h files bit by bit, selecting out pieces to copy or munge, always
maintaining optimum emscripten target output.

Start by having an Indexer commit a Snapshot.  Then add a DocWriter
which does variable length records of key-value pairs. Next, create an
IndexReader module which delegates to a DocReader module to fetch
documents from the store.

You can leave the inverted indexing modules and locking code for
later, because the fundamental data structures and decomposition
strategies of Lucy are sound.

The result wouldn't really be Lucy, and wouldn't be immediately
contributable to Lucy. Instead, it would have the same relation to
Lucy as Lucy had to the original Lucene when it was ported. As the
exercise moves on and you become more familiar with Lucy's problem
domain and its implementation choices, you can decide what direction
you want to take.

Marvin Humphrey
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-dev] Port to JavaScript with emscripten

Sebastián Gurin
Thank you Marvin, yes that's was my plan more or less. I just disagree
with "The result wouldn't really be Lucy,". The idea of emscripten is
to be able to translate lucy automatically or semi-automatically if
possible, and the generated JavaScript code should be equivalent to
originals. Nevertheless, I was trying to get the .h files without luck
always getting the error:

/home/sg/test/prefix/bin/cfc --source=../core --source=../test
--include=/home/sg/test/prefix/share/clownfish/include --dest=autogen
--header=cfc_header
Parcel Clownfish v0.6.0 required by Lucy not found
make: *** [Makefile:522: autogen/hierarchy.json] Aborted (core dumped)


This is what I'm doing:

rm -rf /home/sg/test
mkdir -p /home/sg/test/prefix
cd /home/sg/test

git clone https://git-wip-us.apache.org/repos/asf/lucy-clownfish.git
cd lucy-clownfish/compiler/c
./configure --prefix=/home/sg/test/prefix
make
make install

cd /home/sg/test
git clone https://git-wip-us.apache.org/repos/asf/lucy.git
cd lucy/c
./configure --clownfish-prefix=/home/sg/test/prefix
--prefix=/home/sg/test/prefix
make

If you have any idea please help since I'm kind of blocked right now.
Also I tried cloning clownfish v0.6.0 with the following command byt
the same error happens:

git clone https://git-wip-us.apache.org/repos/asf/lucy-clownfish.git#cb4228524b7bef5af229c4f302eb68ce1573fa76

Thanks!

2017-10-29 17:55 GMT-03:00 Marvin Humphrey <[hidden email]>:

> On Sun, Oct 29, 2017 at 12:48 PM, Sebastián Gurin
> <[hidden email]> wrote:
>
>> I wonder if somebody tried to compile lucy to JavaScript with
>> http://emscripten.org (C/C++ to JavaScript compiler).
>
> I am not aware of such an effort.
>
>> Also I wonder if the clownfish thing won't
>> make it difficult to perform such translation. So, before I tried, I
>> would like to know your thoughts.
>
> Clownfish would probably take some effort to adapt for this purpose.
> If I were to take on this task (and I did not want to spend the effort
> to familiarize myself with Clownfish first), I'd treat it like a loose
> port: create an empty directory alongside Lucy and flesh out new .c
> and .h files bit by bit, selecting out pieces to copy or munge, always
> maintaining optimum emscripten target output.
>
> Start by having an Indexer commit a Snapshot.  Then add a DocWriter
> which does variable length records of key-value pairs. Next, create an
> IndexReader module which delegates to a DocReader module to fetch
> documents from the store.
>
> You can leave the inverted indexing modules and locking code for
> later, because the fundamental data structures and decomposition
> strategies of Lucy are sound.
>
> The result wouldn't really be Lucy, and wouldn't be immediately
> contributable to Lucy. Instead, it would have the same relation to
> Lucy as Lucy had to the original Lucene when it was ported. As the
> exercise moves on and you become more familiar with Lucy's problem
> domain and its implementation choices, you can decide what direction
> you want to take.
>
> Marvin Humphrey
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-dev] Port to JavaScript with emscripten

Nick Wellnhofer
On Oct 30, 2017, at 00:46 , Sebastián Gurin <[hidden email]> wrote:
>
> This is what I'm doing:

> cd lucy-clownfish/compiler/c

This should be

    cd lucy-clownfish/runtime/c

Nick


Reply | Threaded
Open this post in threaded view
|

Re: [lucy-dev] Port to JavaScript with emscripten

Sebastián Gurin
Aja! Thank you very much. Compiling both things, compiler and runtime
did the trick!  This is the final script (BTW would be nice to put
this in the docs since clownfish is not intuitive for newcomers:

FOLDER=/home/sg/test

mkdir -p $FOLDER/prefix
cd $FOLDER

git clone https://git-wip-us.apache.org/repos/asf/lucy-clownfish.git
cd $FOLDER/lucy-clownfish/runtime/c
./configure --prefix=$FOLDER/prefix
make
make install

cd $FOLDER/lucy-clownfish/compiler/c
./configure --prefix=$FOLDER/prefix
make
make install

cd $FOLDER
git clone https://git-wip-us.apache.org/repos/asf/lucy.git
cd $FOLDER/lucy/c
./configure --clownfish-prefix=$FOLDER/prefix --prefix=$FOLDER/prefix
make

# .h files generated $FOLDER/lucy/c/autogen/include/

2017-10-29 21:12 GMT-03:00 Nick Wellnhofer <[hidden email]>:

> On Oct 30, 2017, at 00:46 , Sebastián Gurin <[hidden email]> wrote:
>>
>> This is what I'm doing:
>
>> cd lucy-clownfish/compiler/c
>
> This should be
>
>     cd lucy-clownfish/runtime/c
>
> Nick
>
>