can't post.sh/post.jar

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

can't post.sh/post.jar

Michael Ravits
hello solrs!

I get the following error on windows when trying to index an ~60mb xml file with post.jar.

Also couldn't get post.sh to work - anyone successfully ran it on windows?

C:\solr\example\exampledocs>java -jar post.jar  flix.xml
SimplePostTool: version 1.2
SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, othe
r encodings are not currently supported
SimplePostTool: POSTing files to http://localhost:8983/solr/update..
SimplePostTool: POSTing file flix.xml
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.io.ByteArrayOutputStream.write(Unknown Source)
        at sun.net.www.http.PosterOutputStream.write(Unknown Source)
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
        at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
        at sun.nio.cs.StreamEncoder.write(Unknown Source)
        at java.io.OutputStreamWriter.write(Unknown Source)
        at org.apache.solr.util.SimplePostTool.pipe(SimplePostTool.java:281)
        at org.apache.solr.util.SimplePostTool.postData(SimplePostTool.java:247)

        at org.apache.solr.util.SimplePostTool.postFile(SimplePostTool.java:213)

        at org.apache.solr.util.SimplePostTool.postFiles(SimplePostTool.java:152
)
        at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java:112)

C:\solr\example\exampledocs>

       
---------------------------------
Pinpoint customers who are looking for what you sell.
Reply | Threaded
Open this post in threaded view
|

Re: can't post.sh/post.jar

Ryan McKinley

If you are dealing with such large files, you need to make sure the JVM
has a big enough heap.  Try starting java with -mx100m (-mx2G if you
have it)

java -mx100m -jar post.jar flix.xml

The solr server also needs to be started with enough memory...

ryan


michael ravits wrote:

> hello solrs!
>
> I get the following error on windows when trying to index an ~60mb xml file with post.jar.
>
> Also couldn't get post.sh to work - anyone successfully ran it on windows?
>
> C:\solr\example\exampledocs>java -jar post.jar  flix.xml
> SimplePostTool: version 1.2
> SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, othe
> r encodings are not currently supported
> SimplePostTool: POSTing files to http://localhost:8983/solr/update..
> SimplePostTool: POSTing file flix.xml
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Unknown Source)
>         at java.io.ByteArrayOutputStream.write(Unknown Source)
>         at sun.net.www.http.PosterOutputStream.write(Unknown Source)
>         at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
>         at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
>         at sun.nio.cs.StreamEncoder.write(Unknown Source)
>         at java.io.OutputStreamWriter.write(Unknown Source)
>         at org.apache.solr.util.SimplePostTool.pipe(SimplePostTool.java:281)
>         at org.apache.solr.util.SimplePostTool.postData(SimplePostTool.java:247)
>
>         at org.apache.solr.util.SimplePostTool.postFile(SimplePostTool.java:213)
>
>         at org.apache.solr.util.SimplePostTool.postFiles(SimplePostTool.java:152
> )
>         at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java:112)
>
> C:\solr\example\exampledocs>
>
>        
> ---------------------------------
> Pinpoint customers who are looking for what you sell.

Reply | Threaded
Open this post in threaded view
|

Re: can't post.sh/post.jar

Thierry Collogne
In reply to this post by Michael Ravits
Michael,

You can start the process with more memory assigned to your JVM Like this :

         java -Xms512m -Xmx512m -jar post.jar *.xml

This will start the process with an initial memory of 512 MB and a maximum
memory use of 512 MB.

This is a little explanation about the memory parameters

-Xms    initial java heap size
-Xmx    maximum java heap size
-Xmn    the size of the heap for the young generation

Hope this helps,

Thierry

On 18/06/07, michael ravits <[hidden email]> wrote:

>
> hello solrs!
>
> I get the following error on windows when trying to index an ~60mb xml
> file with post.jar.
>
> Also couldn't get post.sh to work - anyone successfully ran it on windows?
>
> C:\solr\example\exampledocs>java -jar post.jar  flix.xml
> SimplePostTool: version 1.2
> SimplePostTool: WARNING: Make sure your XML documents are encoded in
> UTF-8, othe
> r encodings are not currently supported
> SimplePostTool: POSTing files to http://localhost:8983/solr/update..
> SimplePostTool: POSTing file flix.xml
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Unknown Source)
>         at java.io.ByteArrayOutputStream.write(Unknown Source)
>         at sun.net.www.http.PosterOutputStream.write(Unknown Source)
>         at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
>         at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
>         at sun.nio.cs.StreamEncoder.write(Unknown Source)
>         at java.io.OutputStreamWriter.write(Unknown Source)
>         at org.apache.solr.util.SimplePostTool.pipe(SimplePostTool.java
> :281)
>         at org.apache.solr.util.SimplePostTool.postData(
> SimplePostTool.java:247)
>
>         at org.apache.solr.util.SimplePostTool.postFile(
> SimplePostTool.java:213)
>
>         at org.apache.solr.util.SimplePostTool.postFiles(
> SimplePostTool.java:152
> )
>         at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java
> :112)
>
> C:\solr\example\exampledocs>
>
>
> ---------------------------------
> Pinpoint customers who are looking for what you sell.