problem on running fullimport

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

problem on running fullimport

swap
Hi,

I am using the full import option with the data-config file as mentioned
below

<dataConfig>
   <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://xxxx/xxx" user="xxx" password="xx"  />
    <document >
            <entity name="yyy" query="select studentName from test1">
            <field column="studentName" name="studentName" />
            </entity>
    </document>
</dataConfig>


on running the full-import option I am getting the error mentioned below.I
had already included the dataimport.properties file in my conf file.help me
to get the issue resolved

<response>
-
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">334</int>
</lst>
-
<lst name="initArgs">
-
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</lst>
<str name="command">full-import</str>
<str name="mode">debug</str>
<null name="documents"/>
-
<lst name="verbose-output">
-
<lst name="entity:test1">
-
<lst name="document#1">
<str name="query">select studentName from test1</str>
-
<str name="EXCEPTION">
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
execute query: select studentName from test1 Processing Document # 1
    at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
    at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
    at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
    at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
    at
org.apache.solr.handler.dataimport.DebugLogger$2.getData(DebugLogger.java:184)
    at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
    at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
    at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
    at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
    at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242)
    at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
    at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:331)
    at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389)
    at
org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:203)
    at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
    at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
    at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
    at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
    at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
    at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
    at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
    at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
    at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
    at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
    at org.mortbay.jetty.Server.handle(Server.java:285)
    at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
    at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
    at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
    at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: java.sql.SQLException: Illegal value for setFetchSize().
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
    at com.mysql.jdbc.StatementImpl.setFetchSize(StatementImpl.java:2496)
    at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:242)
    ... 33 more
</str>
<str name="time-taken">0:0:0.50</str>
</lst>
</lst>
</lst>
<str name="status">idle</str>
<str name="importResponse">Configuration Re-loaded sucessfully</str>
-
<lst name="statusMessages">
<str name="Time Elapsed">0:0:0.299</str>
<str name="Total Requests made to DataSource">1</str>
<str name="Total Rows Fetched">0</str>
<str name="Total Documents Processed">0</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2010-10-15 16:42:21</str>
<str name="">Indexing failed. Rolled back all changes.</str>
<str name="Rolledback">2010-10-15 16:42:21</str>
</lst>
-
<str name="WARNING">
This response format is experimental.  It is likely to change in the future.
</str>
</response>

--
Regards
Swapnil Dubey
Reply | Threaded
Open this post in threaded view
|

Re: problem on running fullimport

Ken Stanley
On Fri, Oct 15, 2010 at 7:42 AM, swapnil dubey <[hidden email]>wrote:

> Hi,
>
> I am using the full import option with the data-config file as mentioned
> below
>
> <dataConfig>
>   <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://xxxx/xxx" user="xxx" password="xx"  />
>    <document >
>            <entity name="yyy" query="select studentName from test1">
>            <field column="studentName" name="studentName" />
>            </entity>
>    </document>
> </dataConfig>
>
>
> on running the full-import option I am getting the error mentioned below.I
> had already included the dataimport.properties file in my conf file.help me
> to get the issue resolved
>
> <response>
> -
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">334</int>
> </lst>
> -
> <lst name="initArgs">
> -
> <lst name="defaults">
> <str name="config">data-config.xml</str>
> </lst>
> </lst>
> <str name="command">full-import</str>
> <str name="mode">debug</str>
> <null name="documents"/>
> -
> <lst name="verbose-output">
> -
> <lst name="entity:test1">
> -
> <lst name="document#1">
> <str name="query">select studentName from test1</str>
> -
> <str name="EXCEPTION">
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> execute query: select studentName from test1 Processing Document # 1
> ...
>
> --
> Regards
> Swapnil Dubey
>

Swapnil,

Everything looks fine, except that in your entity definition you forgot to
define which datasource you wish to use. So if you add the
'dataSource="JdbcDataSource"' that should get rid of your exception. As a
reminder, the DataImportHandler wiki (
http://wiki.apache.org/solr/DataImportHandler) on Apache's website is very
helpful with learning how to use the DIH properly. It has helped me with
having a printed copy beside me for easy and quick reference.

- Ken
Reply | Threaded
Open this post in threaded view
|

Re: problem on running fullimport

Alexey Serba
In reply to this post by swap
" Caused by: java.sql.SQLException: Illegal value for setFetchSize(). "

Try to add batchSize="-1" to your data source declaration

http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F

On Fri, Oct 15, 2010 at 3:42 PM, swapnil dubey <[hidden email]> wrote:

> Hi,
>
> I am using the full import option with the data-config file as mentioned
> below
>
> <dataConfig>
>   <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://xxxx/xxx" user="xxx" password="xx"  />
>    <document >
>            <entity name="yyy" query="select studentName from test1">
>            <field column="studentName" name="studentName" />
>            </entity>
>    </document>
> </dataConfig>
>
>
> on running the full-import option I am getting the error mentioned below.I
> had already included the dataimport.properties file in my conf file.help me
> to get the issue resolved
>
> <response>
> -
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">334</int>
> </lst>
> -
> <lst name="initArgs">
> -
> <lst name="defaults">
> <str name="config">data-config.xml</str>
> </lst>
> </lst>
> <str name="command">full-import</str>
> <str name="mode">debug</str>
> <null name="documents"/>
> -
> <lst name="verbose-output">
> -
> <lst name="entity:test1">
> -
> <lst name="document#1">
> <str name="query">select studentName from test1</str>
> -
> <str name="EXCEPTION">
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> execute query: select studentName from test1 Processing Document # 1
>    at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
>    at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
>    at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
>    at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
>    at
> org.apache.solr.handler.dataimport.DebugLogger$2.getData(DebugLogger.java:184)
>    at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
>    at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
>    at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
>    at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
>    at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242)
>    at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
>    at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:331)
>    at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389)
>    at
> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:203)
>    at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>    at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>    at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>    at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>    at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>    at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
>    at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
>    at
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>    at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
>    at org.mortbay.jetty.Server.handle(Server.java:285)
>    at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
>    at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821)
>    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
>    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
>    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>    at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>    at
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
> Caused by: java.sql.SQLException: Illegal value for setFetchSize().
>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
>    at com.mysql.jdbc.StatementImpl.setFetchSize(StatementImpl.java:2496)
>    at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:242)
>    ... 33 more
> </str>
> <str name="time-taken">0:0:0.50</str>
> </lst>
> </lst>
> </lst>
> <str name="status">idle</str>
> <str name="importResponse">Configuration Re-loaded sucessfully</str>
> -
> <lst name="statusMessages">
> <str name="Time Elapsed">0:0:0.299</str>
> <str name="Total Requests made to DataSource">1</str>
> <str name="Total Rows Fetched">0</str>
> <str name="Total Documents Processed">0</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2010-10-15 16:42:21</str>
> <str name="">Indexing failed. Rolled back all changes.</str>
> <str name="Rolledback">2010-10-15 16:42:21</str>
> </lst>
> -
> <str name="WARNING">
> This response format is experimental.  It is likely to change in the future.
> </str>
> </response>
>
> --
> Regards
> Swapnil Dubey
>
pla
Reply | Threaded
Open this post in threaded view
|

Re: problem on running fullimport

pla
Thanks Alexey Serba. I encountered the java.sql.SQLException: Illegal value for setFetchSize() error after upgrading one of my servers to MySQL version 5.5.22.

PLA
Reply | Threaded
Open this post in threaded view
|

RE: problem on running fullimport

Dyer, James
On your <dataSource /> tag, specify "batchSize" with a value that your db/driver allows.  The hardcoded default is 500.  If you set it to -1 it converts it to Integer.MIN_VALUE.  See http://wiki.apache.org/solr/DataImportHandler#Configuring_JdbcDataSource , which recommends using this -1 value in the case of errors.

When executing queries, JdbcDataSource calls Statement.setFetchSize(batchSize) .  Based on the java.sql.Statement api documentation, you can set this to 0 to retain your db/driver's default.  See http://docs.oracle.com/javase/6/docs/api/java/sql/Statement.html#setFetchSize%28int%29

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-----Original Message-----
From: pla [mailto:[hidden email]]
Sent: Thursday, May 24, 2012 9:10 AM
To: [hidden email]
Subject: Re: problem on running fullimport

Thanks Alexey Serba. I encountered the *java.sql.SQLException: Illegal value
for setFetchSize()* error after upgrading one of my servers to MySQL version
5.5.22.

PLA

--
View this message in context: http://lucene.472066.n3.nabble.com/problem-on-running-fullimport-tp1707206p3985924.html
Sent from the Solr - User mailing list archive at Nabble.com.