DIH keeps felling during full-import

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

DIH keeps felling during full-import

Software Dev
I'm receiving the following exception when trying to perform a
full-import (~30 hours). Any idea on ways I could fix this?

Is there an easy way to use DIH to break apart a full-import into
multiple pieces? IE 3 mini-imports instead of 1 large import?

Thanks.




Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
closeConnection
SEVERE: Ignoring Error when closing connection
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Communications link failure during rollback(). Transaction resolution
unknown.
     at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
Source)
     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
     at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
     at com.mysql.jdbc.Util.getInstance(Util.java:382)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
     at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
     at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
     at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
     at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
     at
org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
     at
org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:165)
     at
org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
     at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
     at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
     at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
closeConnection
SEVERE: Ignoring Error when closing connection
java.sql.SQLException: Streaming result set
com.mysql.jdbc.RowDataDynamic@1a797305 is still active. No statements
may be issued when any streaming result sets are open and in use on a
given connection. Ensure that you have called .close() on any active
streaming result sets before attempting more queries.
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
     at
com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2724)
     at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1895)
     at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2140)
     at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2620)
     at
com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4854)
     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4737)
     at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
     at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
     at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
     at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
     at
org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
     at
org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
     at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
     at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
     at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Feb 7, 2011 7:03:29 AM org.apache.solr.handler.dataimport.JdbcDataSource
closeConnection
SEVERE: Ignoring Error when closing connection
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Communications link failure during rollback(). Transaction resolution
unknown.
     at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
Source)
     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
     at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
     at com.mysql.jdbc.Util.getInstance(Util.java:382)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
     at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
     at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
     at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
     at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
     at
org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
     at
org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
     at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
     at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
     at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Feb 7, 2011 7:03:29 AM org.apache.solr.handler.dataimport.JdbcDataSource
closeConnection
SEVERE: Ignoring Error when closing connection
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Communications link failure during rollback(). Transaction resolution
unknown.
     at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
Source)
     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
     at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
     at com.mysql.jdbc.Util.getInstance(Util.java:382)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
     at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
     at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
     at
org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
     at
org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
     at
org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
     at
org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:165)
     at
org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
     at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
     at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
     at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)

Reply | Threaded
Open this post in threaded view
|

Re: DIH keeps failing during full-import

Software Dev
Typo in subject

On 2/7/11 7:59 AM, Mark wrote:

> I'm receiving the following exception when trying to perform a
> full-import (~30 hours). Any idea on ways I could fix this?
>
> Is there an easy way to use DIH to break apart a full-import into
> multiple pieces? IE 3 mini-imports instead of 1 large import?
>
> Thanks.
>
>
>
>
> Feb 7, 2011 5:52:33 AM
> org.apache.solr.handler.dataimport.JdbcDataSource closeConnection
> SEVERE: Ignoring Error when closing connection
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> Communications link failure during rollback(). Transaction resolution
> unknown.
>     at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
> Source)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>     at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>     at com.mysql.jdbc.Util.getInstance(Util.java:382)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>     at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>     at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>     at
> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>     at
> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>     at
> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>     at
> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:165)
>     at
> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>     at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>     at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>     at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
> Feb 7, 2011 5:52:33 AM
> org.apache.solr.handler.dataimport.JdbcDataSource closeConnection
> SEVERE: Ignoring Error when closing connection
> java.sql.SQLException: Streaming result set
> com.mysql.jdbc.RowDataDynamic@1a797305 is still active. No statements
> may be issued when any streaming result sets are open and in use on a
> given connection. Ensure that you have called .close() on any active
> streaming result sets before attempting more queries.
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
>     at
> com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2724)
>     at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1895)
>     at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2140)
>     at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2620)
>     at
> com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4854)
>     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4737)
>     at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>     at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>     at
> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>     at
> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>     at
> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>     at
> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>     at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>     at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>     at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
> Feb 7, 2011 7:03:29 AM
> org.apache.solr.handler.dataimport.JdbcDataSource closeConnection
> SEVERE: Ignoring Error when closing connection
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> Communications link failure during rollback(). Transaction resolution
> unknown.
>     at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
> Source)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>     at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>     at com.mysql.jdbc.Util.getInstance(Util.java:382)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>     at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>     at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>     at
> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>     at
> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>     at
> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>     at
> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>     at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>     at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>     at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
> Feb 7, 2011 7:03:29 AM
> org.apache.solr.handler.dataimport.JdbcDataSource closeConnection
> SEVERE: Ignoring Error when closing connection
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> Communications link failure during rollback(). Transaction resolution
> unknown.
>     at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
> Source)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>     at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>     at com.mysql.jdbc.Util.getInstance(Util.java:382)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>     at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>     at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>     at
> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>     at
> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>     at
> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>     at
> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:165)
>     at
> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>     at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>     at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>     at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>
Reply | Threaded
Open this post in threaded view
|

Re: DIH keeps felling during full-import

Gora Mohanty-3
In reply to this post by Software Dev
On Mon, Feb 7, 2011 at 9:29 PM, Mark <[hidden email]> wrote:

> I'm receiving the following exception when trying to perform a full-import
> (~30 hours). Any idea on ways I could fix this?
>
> Is there an easy way to use DIH to break apart a full-import into multiple
> pieces? IE 3 mini-imports instead of 1 large import?
>
> Thanks.
>
>
>
>
> Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
> closeConnection
> SEVERE: Ignoring Error when closing connection
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> Communications link failure during rollback(). Transaction resolution
> unknown.
[...]

This looks like a network issue, or some other failure in communicating
with the mysql database. Is that a possibility? Also, how many records
are you importing, what is the data size, what is the quality of the network
connection, etc.?

One way to break up the number of records imported at a time is to
shard your data at at the database level, but the advisability of this
option depends on whether there is a more fundamental issue.

Regards,
Gora
Reply | Threaded
Open this post in threaded view
|

Re: DIH keeps felling during full-import

Software Dev
Full import is around 6M documents which when completed totals around
30GB in size.

Im guessing it could be a database connectivity problem because I also
see these types of errors on delta-imports which could be anywhere from
20K to 300K records.

On 2/7/11 8:15 AM, Gora Mohanty wrote:

> On Mon, Feb 7, 2011 at 9:29 PM, Mark<[hidden email]>  wrote:
>> I'm receiving the following exception when trying to perform a full-import
>> (~30 hours). Any idea on ways I could fix this?
>>
>> Is there an easy way to use DIH to break apart a full-import into multiple
>> pieces? IE 3 mini-imports instead of 1 large import?
>>
>> Thanks.
>>
>>
>>
>>
>> Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>> Communications link failure during rollback(). Transaction resolution
>> unknown.
> [...]
>
> This looks like a network issue, or some other failure in communicating
> with the mysql database. Is that a possibility? Also, how many records
> are you importing, what is the data size, what is the quality of the network
> connection, etc.?
>
> One way to break up the number of records imported at a time is to
> shard your data at at the database level, but the advisability of this
> option depends on whether there is a more fundamental issue.
>
> Regards,
> Gora
Reply | Threaded
Open this post in threaded view
|

Re: DIH keeps felling during full-import

Gora Mohanty-3
On Mon, Feb 7, 2011 at 10:15 PM, Mark <[hidden email]> wrote:
> Full import is around 6M documents which when completed totals around 30GB
> in size.
>
> Im guessing it could be a database connectivity problem because I also see
> these types of errors on delta-imports which could be anywhere from 20K to
> 300K records.
[...[

In that case, it might be advisable to start by trying to fix these.
mysql, as well as most any modern database, ought to be able
to deal with the sizes that you mention above, so my first guess
would be issues with network connectivity.

Is this an internal network, or does it go over the Internet? In
either case, how good is the network supposed to be? Is
there any application monitoring the network?

Regards,
Gora
Reply | Threaded
Open this post in threaded view
|

Re: DIH keeps failing during full-import

Lance Norskog-2
In reply to this post by Software Dev
It is not reasonable to expect a database session to work over 30
hours, let alone an app/database operation.

If you can mark a database record as successfully indexed, the
incremental feature can be used to only index non-marked records.

SOLR-1499 offers a way to check Solr with a sorted query on every
field; you could use to find the most recent indexed record.  There is
no general way of doing this.

On Mon, Feb 7, 2011 at 7:59 AM, Mark <[hidden email]> wrote:

> Typo in subject
>
> On 2/7/11 7:59 AM, Mark wrote:
>>
>> I'm receiving the following exception when trying to perform a full-import
>> (~30 hours). Any idea on ways I could fix this?
>>
>> Is there an easy way to use DIH to break apart a full-import into multiple
>> pieces? IE 3 mini-imports instead of 1 large import?
>>
>> Thanks.
>>
>>
>>
>>
>> Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>> Communications link failure during rollback(). Transaction resolution
>> unknown.
>>    at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
>> Source)
>>    at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>    at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>>    at com.mysql.jdbc.Util.getInstance(Util.java:382)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>>    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>>    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>>    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:165)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> java.sql.SQLException: Streaming result set
>> com.mysql.jdbc.RowDataDynamic@1a797305 is still active. No statements may be
>> issued when any streaming result sets are open and in use on a given
>> connection. Ensure that you have called .close() on any active streaming
>> result sets before attempting more queries.
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
>>    at
>> com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2724)
>>    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1895)
>>    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2140)
>>    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2620)
>>    at
>> com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4854)
>>    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4737)
>>    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>>    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> Feb 7, 2011 7:03:29 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>> Communications link failure during rollback(). Transaction resolution
>> unknown.
>>    at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
>> Source)
>>    at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>    at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>>    at com.mysql.jdbc.Util.getInstance(Util.java:382)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>>    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>>    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>>    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> Feb 7, 2011 7:03:29 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>> Communications link failure during rollback(). Transaction resolution
>> unknown.
>>    at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
>> Source)
>>    at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>    at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>>    at com.mysql.jdbc.Util.getInstance(Util.java:382)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>>    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>>    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>>    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:165)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>>
>



--
Lance Norskog
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: DIH keeps failing during full-import

Erick Erickson
In reply to this post by Software Dev
You're probably better off in this instance creating your own
process based on SolrJ and your jdbc-driver-of-choice. DIH
doesn't provide much in the way of fine-grained control over
all aspects of the process, and at +30 hours I suspect you
want some better control.

FWIW, SolrJ is not very hard at all to use for this kind of thing.

Best
Erick

On Mon, Feb 7, 2011 at 10:59 AM, Mark <[hidden email]> wrote:

> Typo in subject
>
> On 2/7/11 7:59 AM, Mark wrote:
>
>> I'm receiving the following exception when trying to perform a full-import
>> (~30 hours). Any idea on ways I could fix this?
>>
>> Is there an easy way to use DIH to break apart a full-import into multiple
>> pieces? IE 3 mini-imports instead of 1 large import?
>>
>> Thanks.
>>
>>
>>
>>
>> Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>> Communications link failure during rollback(). Transaction resolution
>> unknown.
>>    at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
>> Source)
>>    at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>    at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>>    at com.mysql.jdbc.Util.getInstance(Util.java:382)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>>    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>>    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>>    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:165)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> java.sql.SQLException: Streaming result set
>> com.mysql.jdbc.RowDataDynamic@1a797305 is still active. No statements may
>> be issued when any streaming result sets are open and in use on a given
>> connection. Ensure that you have called .close() on any active streaming
>> result sets before attempting more queries.
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
>>    at
>> com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2724)
>>    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1895)
>>    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2140)
>>    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2620)
>>    at
>> com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4854)
>>    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4737)
>>    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>>    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> Feb 7, 2011 7:03:29 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>> Communications link failure during rollback(). Transaction resolution
>> unknown.
>>    at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
>> Source)
>>    at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>    at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>>    at com.mysql.jdbc.Util.getInstance(Util.java:382)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>>    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>>    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>>    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> Feb 7, 2011 7:03:29 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> closeConnection
>> SEVERE: Ignoring Error when closing connection
>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>> Communications link failure during rollback(). Transaction resolution
>> unknown.
>>    at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown
>> Source)
>>    at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>    at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
>>    at com.mysql.jdbc.Util.getInstance(Util.java:382)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
>>    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
>>    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4751)
>>    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4345)
>>    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1564)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.closeConnection(JdbcDataSource.java:399)
>>    at
>> org.apache.solr.handler.dataimport.JdbcDataSource.close(JdbcDataSource.java:390)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:174)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig$Entity.clearCache(DataConfig.java:165)
>>    at
>> org.apache.solr.handler.dataimport.DataConfig.clearCaches(DataConfig.java:332)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:360)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>>    at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>>
>>