FW: Solr 4 to Solr7 migration DIH behavior change

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

FW: Solr 4 to Solr7 migration DIH behavior change

Shashank Bellary

Hi Folks

I migrated from Solr 4 to 7.5 and I see an issue with the way DIH is working. I use `JdbcDataSource` and here the config file is attached

1) I started seeing OutOfMemory issue since MySQL JDBC driver has that issue of not respecting `batchSize` (though Solr4 didn't show this behavior). So, I added `batchSize=-1` for that

2) After adding that I'm running into ResultSet closed exception as shown below while fetching the child entity

 
getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ; ':org.apache.solr.handler.dataimport.DataImportHandlerException: java.sql.SQLException: Operation not allowed after ResultSet closed
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Operation not allowed after ResultSet closed
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
... 13 more
 

Is this a known issue? How do I fix this, any help is greatly appreciated.

 

Thanks

Shashank

This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.

serviceprofile-data-import.xml (31K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Solr 4 to Solr7 migration DIH behavior change

Shashank Bellary

 

Hi Folks

I migrated from Solr 4 to 7.5 and I see an issue with the way DIH is working. I use `JdbcDataSource` and here the config file is attached

1) I started seeing OutOfMemory issue since MySQL JDBC driver has that issue of not respecting `batchSize` (though Solr4 didn't show this behavior). So, I added `batchSize=-1` for that

2) After adding that I'm running into ResultSet closed exception as shown below while fetching the child entity

 
getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ; ':org.apache.solr.handler.dataimport.DataImportHandlerException: java.sql.SQLException: Operation not allowed after ResultSet closed
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Operation not allowed after ResultSet closed
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
... 13 more
 

Is this a known issue? How do I fix this, any help is greatly appreciated.

 

Thanks

Shashank

This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.

serviceprofile-data-import.xml (31K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Jörn Franke
Did you update the java version to 8? Did you upgrade the MySQL driver to the latest version?

> Am 22.11.2019 um 20:43 schrieb Shashank Bellary <[hidden email]>:
>
> 
>  
> Hi Folks
> I migrated from Solr 4 to 7.5 and I see an issue with the way DIH is working. I use `JdbcDataSource` and here the config file is attached
> 1) I started seeing OutOfMemory issue since MySQL JDBC driver has that issue of not respecting `batchSize` (though Solr4 didn't show this behavior). So, I added `batchSize=-1` for that
> 2) After adding that I'm running into ResultSet closed exception as shown below while fetching the child entity
>  
> getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ; ':org.apache.solr.handler.dataimport.DataImportHandlerException: java.sql.SQLException: Operation not allowed after ResultSet closed
> at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
> at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
> at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
> at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
> at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
> at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
> at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
> at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
> at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
> at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
> at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
> at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.sql.SQLException: Operation not allowed after ResultSet closed
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
> at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
> at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
> at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
> at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
> at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
> ... 13 more
>  
> Is this a known issue? How do I fix this, any help is greatly appreciated.
>  
> Thanks
> Shashank
> This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
> <serviceprofile-data-import.xml>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Shashank Bellary
Yes, I'm on java 8 and mysql driver 5.1.13

Thanks

On 11/22/19, 3:00 PM, "Jörn Franke" <[hidden email]> wrote:

    Note - This message originated from outside Care.com - Please use caution before opening attachments, clicking on links or sharing information.


    Did you update the java version to 8? Did you upgrade the MySQL driver to the latest version?

    > Am 22.11.2019 um 20:43 schrieb Shashank Bellary <[hidden email]>:
    >
    >
    >
    > Hi Folks
    > I migrated from Solr 4 to 7.5 and I see an issue with the way DIH is working. I use `JdbcDataSource` and here the config file is attached
    > 1) I started seeing OutOfMemory issue since MySQL JDBC driver has that issue of not respecting `batchSize` (though Solr4 didn't show this behavior). So, I added `batchSize=-1` for that
    > 2) After adding that I'm running into ResultSet closed exception as shown below while fetching the child entity
    >
    > getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ; ':org.apache.solr.handler.dataimport.DataImportHandlerException: java.sql.SQLException: Operation not allowed after ResultSet closed
    > at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
    > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
    > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
    > at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
    > at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
    > at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
    > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
    > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
    > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
    > at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
    > at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
    > at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
    > at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
    > at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
    > at java.lang.Thread.run(Thread.java:748)
    > Caused by: java.sql.SQLException: Operation not allowed after ResultSet closed
    > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
    > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
    > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
    > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
    > at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
    > at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
    > at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
    > at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
    > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
    > ... 13 more
    >
    > Is this a known issue? How do I fix this, any help is greatly appreciated.
    >
    > Thanks
    > Shashank
    > This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
    > <serviceprofile-data-import.xml>


This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Shashank Bellary
I've not upgraded mysql driver to latest, I can you explain whats the issue here?
thanks

On 11/22/19, 4:40 PM, "Shashank Bellary" <[hidden email]> wrote:

    Note - This message originated from outside Care.com - Please use caution before opening attachments, clicking on links or sharing information.


    Yes, I'm on java 8 and mysql driver 5.1.13

    Thanks

    On 11/22/19, 3:00 PM, "Jörn Franke" <[hidden email]> wrote:

        Note - This message originated from outside Care.com - Please use caution before opening attachments, clicking on links or sharing information.


        Did you update the java version to 8? Did you upgrade the MySQL driver to the latest version?

        > Am 22.11.2019 um 20:43 schrieb Shashank Bellary <[hidden email]>:
        >
        >
        >
        > Hi Folks
        > I migrated from Solr 4 to 7.5 and I see an issue with the way DIH is working. I use `JdbcDataSource` and here the config file is attached
        > 1) I started seeing OutOfMemory issue since MySQL JDBC driver has that issue of not respecting `batchSize` (though Solr4 didn't show this behavior). So, I added `batchSize=-1` for that
        > 2) After adding that I'm running into ResultSet closed exception as shown below while fetching the child entity
        >
        > getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ; ':org.apache.solr.handler.dataimport.DataImportHandlerException: java.sql.SQLException: Operation not allowed after ResultSet closed
        > at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
        > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
        > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
        > at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
        > at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
        > at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
        > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
        > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
        > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
        > at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
        > at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
        > at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
        > at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
        > at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
        > at java.lang.Thread.run(Thread.java:748)
        > Caused by: java.sql.SQLException: Operation not allowed after ResultSet closed
        > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
        > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
        > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
        > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
        > at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
        > at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
        > at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
        > at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
        > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
        > ... 13 more
        >
        > Is this a known issue? How do I fix this, any help is greatly appreciated.
        >
        > Thanks
        > Shashank
        > This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
        > <serviceprofile-data-import.xml>


    This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.


This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Shashank Bellary
In reply to this post by Jörn Franke
Any thoughts guys? I tried with mysql driver v8 also, still no luck

On 11/22/19, 3:00 PM, "Jörn Franke" <[hidden email]> wrote:

    Note - This message originated from outside Care.com - Please use caution before opening attachments, clicking on links or sharing information.


    Did you update the java version to 8? Did you upgrade the MySQL driver to the latest version?

    > Am 22.11.2019 um 20:43 schrieb Shashank Bellary <[hidden email]>:
    >
    >
    >
    > Hi Folks
    > I migrated from Solr 4 to 7.5 and I see an issue with the way DIH is working. I use `JdbcDataSource` and here the config file is attached
    > 1) I started seeing OutOfMemory issue since MySQL JDBC driver has that issue of not respecting `batchSize` (though Solr4 didn't show this behavior). So, I added `batchSize=-1` for that
    > 2) After adding that I'm running into ResultSet closed exception as shown below while fetching the child entity
    >
    > getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ; ':org.apache.solr.handler.dataimport.DataImportHandlerException: java.sql.SQLException: Operation not allowed after ResultSet closed
    > at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
    > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
    > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
    > at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
    > at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
    > at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
    > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
    > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
    > at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
    > at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
    > at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
    > at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
    > at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
    > at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
    > at java.lang.Thread.run(Thread.java:748)
    > Caused by: java.sql.SQLException: Operation not allowed after ResultSet closed
    > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
    > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
    > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
    > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
    > at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
    > at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
    > at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
    > at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
    > at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
    > ... 13 more
    >
    > Is this a known issue? How do I fix this, any help is greatly appreciated.
    >
    > Thanks
    > Shashank
    > This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
    > <serviceprofile-data-import.xml>


This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Mikhail Khludnev-2
Hello, Shashank.
The error seems similar, but I didin't find an old issue with such error.
I've found only one abandoned thread in the mailing list. ='17' seems
suspicious to me, usually it should be done via prepared statement. Have no
thoughts, maybe you can share you data config?

On Sun, Nov 24, 2019 at 10:40 PM Shashank Bellary <[hidden email]> wrote:

> Any thoughts guys? I tried with mysql driver v8 also, still no luck
>
> On 11/22/19, 3:00 PM, "Jörn Franke" <[hidden email]> wrote:
>
>     Note - This message originated from outside Care.com - Please use
> caution before opening attachments, clicking on links or sharing
> information.
>
>
>     Did you update the java version to 8? Did you upgrade the MySQL driver
> to the latest version?
>
>     > Am 22.11.2019 um 20:43 schrieb Shashank Bellary <[hidden email]>:
>     >
>     >
>     >
>     > Hi Folks
>     > I migrated from Solr 4 to 7.5 and I see an issue with the way DIH is
> working. I use `JdbcDataSource` and here the config file is attached
>     > 1) I started seeing OutOfMemory issue since MySQL JDBC driver has
> that issue of not respecting `batchSize` (though Solr4 didn't show this
> behavior). So, I added `batchSize=-1` for that
>     > 2) After adding that I'm running into ResultSet closed exception as
> shown below while fetching the child entity
>     >
>     > getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM
> SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ;
> ':org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.sql.SQLException: Operation not allowed after ResultSet closed
>     > at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
>     > at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
>     > at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
>     > at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
>     > at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
>     > at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
>     > at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
>     > at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
>     > at
> org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
>     > at java.lang.Thread.run(Thread.java:748)
>     > Caused by: java.sql.SQLException: Operation not allowed after
> ResultSet closed
>     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
>     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
>     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
>     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
>     > at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
>     > at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
>     > at
> com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
>     > at
> com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
>     > at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
>     > ... 13 more
>     >
>     > Is this a known issue? How do I fix this, any help is greatly
> appreciated.
>     >
>     > Thanks
>     > Shashank
>     > This email is intended for the person(s) to whom it is addressed and
> may contain information that is PRIVILEGED or CONFIDENTIAL. Any
> unauthorized use, distribution, copying, or disclosure by any person other
> than the addressee(s) is strictly prohibited. If you have received this
> email in error, please notify the sender immediately by return email and
> delete the message and any attachments from your system.
>     > <serviceprofile-data-import.xml>
>
>
> This email is intended for the person(s) to whom it is addressed and may
> contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized
> use, distribution, copying, or disclosure by any person other than the
> addressee(s) is strictly prohibited. If you have received this email in
> error, please notify the sender immediately by return email and delete the
> message and any attachments from your system.
>


--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Shashank Bellary
Thanks Mikhail, data config is on the thread above. I’ll share again if you can’t find it

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Mikhail Khludnev <[hidden email]>
Sent: Sunday, November 24, 2019 2:51:40 PM
To: solr-user <[hidden email]>
Subject: Re: Solr 4 to Solr7 migration DIH behavior change

Note - This message originated from outside Care.com - Please use caution before opening attachments, clicking on links or sharing information.


Hello, Shashank.
The error seems similar, but I didin't find an old issue with such error.
I've found only one abandoned thread in the mailing list. ='17' seems
suspicious to me, usually it should be done via prepared statement. Have no
thoughts, maybe you can share you data config?

On Sun, Nov 24, 2019 at 10:40 PM Shashank Bellary <[hidden email]> wrote:

> Any thoughts guys? I tried with mysql driver v8 also, still no luck
>
> On 11/22/19, 3:00 PM, "Jörn Franke" <[hidden email]> wrote:
>
>     Note - This message originated from outside Care.com - Please use
> caution before opening attachments, clicking on links or sharing
> information.
>
>
>     Did you update the java version to 8? Did you upgrade the MySQL driver
> to the latest version?
>
>     > Am 22.11.2019 um 20:43 schrieb Shashank Bellary <[hidden email]>:
>     >
>     >
>     >
>     > Hi Folks
>     > I migrated from Solr 4 to 7.5 and I see an issue with the way DIH is
> working. I use `JdbcDataSource` and here the config file is attached
>     > 1) I started seeing OutOfMemory issue since MySQL JDBC driver has
> that issue of not respecting `batchSize` (though Solr4 didn't show this
> behavior). So, I added `batchSize=-1` for that
>     > 2) After adding that I'm running into ResultSet closed exception as
> shown below while fetching the child entity
>     >
>     > getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM
> SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ;
> ':org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.sql.SQLException: Operation not allowed after ResultSet closed
>     > at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
>     > at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
>     > at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
>     > at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
>     > at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
>     > at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
>     > at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
>     > at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
>     > at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
>     > at
> org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
>     > at java.lang.Thread.run(Thread.java:748)
>     > Caused by: java.sql.SQLException: Operation not allowed after
> ResultSet closed
>     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
>     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
>     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
>     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
>     > at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
>     > at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
>     > at
> com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
>     > at
> com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
>     > at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
>     > ... 13 more
>     >
>     > Is this a known issue? How do I fix this, any help is greatly
> appreciated.
>     >
>     > Thanks
>     > Shashank
>     > This email is intended for the person(s) to whom it is addressed and
> may contain information that is PRIVILEGED or CONFIDENTIAL. Any
> unauthorized use, distribution, copying, or disclosure by any person other
> than the addressee(s) is strictly prohibited. If you have received this
> email in error, please notify the sender immediately by return email and
> delete the message and any attachments from your system.
>     > <serviceprofile-data-import.xml>
>
>
> This email is intended for the person(s) to whom it is addressed and may
> contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized
> use, distribution, copying, or disclosure by any person other than the
> addressee(s) is strictly prohibited. If you have received this email in
> error, please notify the sender immediately by return email and delete the
> message and any attachments from your system.
>


--
Sincerely yours
Mikhail Khludnev
This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Mikhail Khludnev-2
Config makes sense to me. The only unusual thing is tailing semicolons in
SQL statements. What if you drop them?

On Sun, Nov 24, 2019 at 11:30 PM Shashank Bellary <[hidden email]> wrote:

> Thanks Mikhail, data config is on the thread above. I’ll share again if
> you can’t find it
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Mikhail Khludnev <[hidden email]>
> Sent: Sunday, November 24, 2019 2:51:40 PM
> To: solr-user <[hidden email]>
> Subject: Re: Solr 4 to Solr7 migration DIH behavior change
>
> Note - This message originated from outside Care.com - Please use caution
> before opening attachments, clicking on links or sharing information.
>
>
> Hello, Shashank.
> The error seems similar, but I didin't find an old issue with such error.
> I've found only one abandoned thread in the mailing list. ='17' seems
> suspicious to me, usually it should be done via prepared statement. Have no
> thoughts, maybe you can share you data config?
>
> On Sun, Nov 24, 2019 at 10:40 PM Shashank Bellary <[hidden email]>
> wrote:
>
> > Any thoughts guys? I tried with mysql driver v8 also, still no luck
> >
> > On 11/22/19, 3:00 PM, "Jörn Franke" <[hidden email]> wrote:
> >
> >     Note - This message originated from outside Care.com - Please use
> > caution before opening attachments, clicking on links or sharing
> > information.
> >
> >
> >     Did you update the java version to 8? Did you upgrade the MySQL
> driver
> > to the latest version?
> >
> >     > Am 22.11.2019 um 20:43 schrieb Shashank Bellary <[hidden email]
> >:
> >     >
> >     >
> >     >
> >     > Hi Folks
> >     > I migrated from Solr 4 to 7.5 and I see an issue with the way DIH
> is
> > working. I use `JdbcDataSource` and here the config file is attached
> >     > 1) I started seeing OutOfMemory issue since MySQL JDBC driver has
> > that issue of not respecting `batchSize` (though Solr4 didn't show this
> > behavior). So, I added `batchSize=-1` for that
> >     > 2) After adding that I'm running into ResultSet closed exception as
> > shown below while fetching the child entity
> >     >
> >     > getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM
> > SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ;
> > ':org.apache.solr.handler.dataimport.DataImportHandlerException:
> > java.sql.SQLException: Operation not allowed after ResultSet closed
> >     > at
> >
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
> >     > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
> >     > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
> >     > at
> >
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
> >     > at
> >
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
> >     > at
> >
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
> >     > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
> >     > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
> >     > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
> >     > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
> >     > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
> >     > at
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
> >     > at
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
> >     > at
> >
> org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
> >     > at java.lang.Thread.run(Thread.java:748)
> >     > Caused by: java.sql.SQLException: Operation not allowed after
> > ResultSet closed
> >     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
> >     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
> >     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
> >     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
> >     > at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
> >     > at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
> >     > at
> > com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
> >     > at
> > com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
> >     > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
> >     > ... 13 more
> >     >
> >     > Is this a known issue? How do I fix this, any help is greatly
> > appreciated.
> >     >
> >     > Thanks
> >     > Shashank
> >     > This email is intended for the person(s) to whom it is addressed
> and
> > may contain information that is PRIVILEGED or CONFIDENTIAL. Any
> > unauthorized use, distribution, copying, or disclosure by any person
> other
> > than the addressee(s) is strictly prohibited. If you have received this
> > email in error, please notify the sender immediately by return email and
> > delete the message and any attachments from your system.
> >     > <serviceprofile-data-import.xml>
> >
> >
> > This email is intended for the person(s) to whom it is addressed and may
> > contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized
> > use, distribution, copying, or disclosure by any person other than the
> > addressee(s) is strictly prohibited. If you have received this email in
> > error, please notify the sender immediately by return email and delete
> the
> > message and any attachments from your system.
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> This email is intended for the person(s) to whom it is addressed and may
> contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized
> use, distribution, copying, or disclosure by any person other than the
> addressee(s) is strictly prohibited. If you have received this email in
> error, please notify the sender immediately by return email and delete the
> message and any attachments from your system.
>


--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Shashank Bellary
That didn't make any difference. However, I upgraded to 8.0.15 version of mysql-jdbc driver which fixed the problem of result set closing but then data import got really slow and getting stuck without any logs in solr logs. I'll debug more from DB perspective, in the meantime any pointers would be helpful

On 11/24/19, 3:51 PM, "Mikhail Khludnev" <[hidden email]> wrote:

    Note - This message originated from outside Care.com - Please use caution before opening attachments, clicking on links or sharing information.


    Config makes sense to me. The only unusual thing is tailing semicolons in
    SQL statements. What if you drop them?

    On Sun, Nov 24, 2019 at 11:30 PM Shashank Bellary <[hidden email]> wrote:

    > Thanks Mikhail, data config is on the thread above. I’ll share again if
    > you can’t find it
    >
    > Get Outlook for iOS<https://aka.ms/o0ukef>
    > ________________________________
    > From: Mikhail Khludnev <[hidden email]>
    > Sent: Sunday, November 24, 2019 2:51:40 PM
    > To: solr-user <[hidden email]>
    > Subject: Re: Solr 4 to Solr7 migration DIH behavior change
    >
    > Note - This message originated from outside Care.com - Please use caution
    > before opening attachments, clicking on links or sharing information.
    >
    >
    > Hello, Shashank.
    > The error seems similar, but I didin't find an old issue with such error.
    > I've found only one abandoned thread in the mailing list. ='17' seems
    > suspicious to me, usually it should be done via prepared statement. Have no
    > thoughts, maybe you can share you data config?
    >
    > On Sun, Nov 24, 2019 at 10:40 PM Shashank Bellary <[hidden email]>
    > wrote:
    >
    > > Any thoughts guys? I tried with mysql driver v8 also, still no luck
    > >
    > > On 11/22/19, 3:00 PM, "Jörn Franke" <[hidden email]> wrote:
    > >
    > >     Note - This message originated from outside Care.com - Please use
    > > caution before opening attachments, clicking on links or sharing
    > > information.
    > >
    > >
    > >     Did you update the java version to 8? Did you upgrade the MySQL
    > driver
    > > to the latest version?
    > >
    > >     > Am 22.11.2019 um 20:43 schrieb Shashank Bellary <[hidden email]
    > >:
    > >     >
    > >     >
    > >     >
    > >     > Hi Folks
    > >     > I migrated from Solr 4 to 7.5 and I see an issue with the way DIH
    > is
    > > working. I use `JdbcDataSource` and here the config file is attached
    > >     > 1) I started seeing OutOfMemory issue since MySQL JDBC driver has
    > > that issue of not respecting `batchSize` (though Solr4 didn't show this
    > > behavior). So, I added `batchSize=-1` for that
    > >     > 2) After adding that I'm running into ResultSet closed exception as
    > > shown below while fetching the child entity
    > >     >
    > >     > getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM
    > > SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID = '17' ;
    > > ':org.apache.solr.handler.dataimport.DataImportHandlerException:
    > > java.sql.SQLException: Operation not allowed after ResultSet closed
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
    > >     > at java.lang.Thread.run(Thread.java:748)
    > >     > Caused by: java.sql.SQLException: Operation not allowed after
    > > ResultSet closed
    > >     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
    > >     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
    > >     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
    > >     > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
    > >     > at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
    > >     > at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
    > >     > at
    > > com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
    > >     > at
    > > com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
    > >     > at
    > >
    > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
    > >     > ... 13 more
    > >     >
    > >     > Is this a known issue? How do I fix this, any help is greatly
    > > appreciated.
    > >     >
    > >     > Thanks
    > >     > Shashank
    > >     > This email is intended for the person(s) to whom it is addressed
    > and
    > > may contain information that is PRIVILEGED or CONFIDENTIAL. Any
    > > unauthorized use, distribution, copying, or disclosure by any person
    > other
    > > than the addressee(s) is strictly prohibited. If you have received this
    > > email in error, please notify the sender immediately by return email and
    > > delete the message and any attachments from your system.
    > >     > <serviceprofile-data-import.xml>
    > >
    > >
    > > This email is intended for the person(s) to whom it is addressed and may
    > > contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized
    > > use, distribution, copying, or disclosure by any person other than the
    > > addressee(s) is strictly prohibited. If you have received this email in
    > > error, please notify the sender immediately by return email and delete
    > the
    > > message and any attachments from your system.
    > >
    >
    >
    > --
    > Sincerely yours
    > Mikhail Khludnev
    > This email is intended for the person(s) to whom it is addressed and may
    > contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized
    > use, distribution, copying, or disclosure by any person other than the
    > addressee(s) is strictly prohibited. If you have received this email in
    > error, please notify the sender immediately by return email and delete the
    > message and any attachments from your system.
    >


    --
    Sincerely yours
    Mikhail Khludnev


This email is intended for the person(s) to whom it is addressed and may contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized use, distribution, copying, or disclosure by any person other than the addressee(s) is strictly prohibited. If you have received this email in error, please notify the sender immediately by return email and delete the message and any attachments from your system.
Reply | Threaded
Open this post in threaded view
|

Re: Solr 4 to Solr7 migration DIH behavior change

Mikhail Khludnev-2
It's worth to increase log level for DIH categories in SolrAmin. It's quite
useful usually.

On Mon, Nov 25, 2019 at 10:19 PM Shashank Bellary <[hidden email]> wrote:

> That didn't make any difference. However, I upgraded to 8.0.15 version of
> mysql-jdbc driver which fixed the problem of result set closing but then
> data import got really slow and getting stuck without any logs in solr
> logs. I'll debug more from DB perspective, in the meantime any pointers
> would be helpful
>
> On 11/24/19, 3:51 PM, "Mikhail Khludnev" <[hidden email]> wrote:
>
>     Note - This message originated from outside Care.com - Please use
> caution before opening attachments, clicking on links or sharing
> information.
>
>
>     Config makes sense to me. The only unusual thing is tailing semicolons
> in
>     SQL statements. What if you drop them?
>
>     On Sun, Nov 24, 2019 at 11:30 PM Shashank Bellary <[hidden email]>
> wrote:
>
>     > Thanks Mikhail, data config is on the thread above. I’ll share again
> if
>     > you can’t find it
>     >
>     > Get Outlook for iOS<https://aka.ms/o0ukef>
>     > ________________________________
>     > From: Mikhail Khludnev <[hidden email]>
>     > Sent: Sunday, November 24, 2019 2:51:40 PM
>     > To: solr-user <[hidden email]>
>     > Subject: Re: Solr 4 to Solr7 migration DIH behavior change
>     >
>     > Note - This message originated from outside Care.com - Please use
> caution
>     > before opening attachments, clicking on links or sharing information.
>     >
>     >
>     > Hello, Shashank.
>     > The error seems similar, but I didin't find an old issue with such
> error.
>     > I've found only one abandoned thread in the mailing list. ='17' seems
>     > suspicious to me, usually it should be done via prepared statement.
> Have no
>     > thoughts, maybe you can share you data config?
>     >
>     > On Sun, Nov 24, 2019 at 10:40 PM Shashank Bellary <[hidden email]
> >
>     > wrote:
>     >
>     > > Any thoughts guys? I tried with mysql driver v8 also, still no luck
>     > >
>     > > On 11/22/19, 3:00 PM, "Jörn Franke" <[hidden email]> wrote:
>     > >
>     > >     Note - This message originated from outside Care.com - Please
> use
>     > > caution before opening attachments, clicking on links or sharing
>     > > information.
>     > >
>     > >
>     > >     Did you update the java version to 8? Did you upgrade the MySQL
>     > driver
>     > > to the latest version?
>     > >
>     > >     > Am 22.11.2019 um 20:43 schrieb Shashank Bellary <
> [hidden email]
>     > >:
>     > >     >
>     > >     >
>     > >     >
>     > >     > Hi Folks
>     > >     > I migrated from Solr 4 to 7.5 and I see an issue with the
> way DIH
>     > is
>     > > working. I use `JdbcDataSource` and here the config file is
> attached
>     > >     > 1) I started seeing OutOfMemory issue since MySQL JDBC
> driver has
>     > > that issue of not respecting `batchSize` (though Solr4 didn't show
> this
>     > > behavior). So, I added `batchSize=-1` for that
>     > >     > 2) After adding that I'm running into ResultSet closed
> exception as
>     > > shown below while fetching the child entity
>     > >     >
>     > >     > getNext() failed for query ' SELECT REVIEW AS REVIEWS FROM
>     > > SOLR_SITTER_SERVICE_PROFILE_REVIEWS WHERE SERVICE_PROFILE_ID =
> '17' ;
>     > > ':org.apache.solr.handler.dataimport.DataImportHandlerException:
>     > > java.sql.SQLException: Operation not allowed after ResultSet closed
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:517)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:33)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
>     > >     > at java.lang.Thread.run(Thread.java:748)
>     > >     > Caused by: java.sql.SQLException: Operation not allowed after
>     > > ResultSet closed
>     > >     > at
> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
>     > >     > at
> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
>     > >     > at
> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
>     > >     > at
> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
>     > >     > at
> com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:794)
>     > >     > at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7145)
>     > >     > at
>     > >
> com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2078)
>     > >     > at
>     > >
> com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2062)
>     > >     > at
>     > >
>     >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)
>     > >     > ... 13 more
>     > >     >
>     > >     > Is this a known issue? How do I fix this, any help is greatly
>     > > appreciated.
>     > >     >
>     > >     > Thanks
>     > >     > Shashank
>     > >     > This email is intended for the person(s) to whom it is
> addressed
>     > and
>     > > may contain information that is PRIVILEGED or CONFIDENTIAL. Any
>     > > unauthorized use, distribution, copying, or disclosure by any
> person
>     > other
>     > > than the addressee(s) is strictly prohibited. If you have received
> this
>     > > email in error, please notify the sender immediately by return
> email and
>     > > delete the message and any attachments from your system.
>     > >     > <serviceprofile-data-import.xml>
>     > >
>     > >
>     > > This email is intended for the person(s) to whom it is addressed
> and may
>     > > contain information that is PRIVILEGED or CONFIDENTIAL. Any
> unauthorized
>     > > use, distribution, copying, or disclosure by any person other than
> the
>     > > addressee(s) is strictly prohibited. If you have received this
> email in
>     > > error, please notify the sender immediately by return email and
> delete
>     > the
>     > > message and any attachments from your system.
>     > >
>     >
>     >
>     > --
>     > Sincerely yours
>     > Mikhail Khludnev
>     > This email is intended for the person(s) to whom it is addressed and
> may
>     > contain information that is PRIVILEGED or CONFIDENTIAL. Any
> unauthorized
>     > use, distribution, copying, or disclosure by any person other than
> the
>     > addressee(s) is strictly prohibited. If you have received this email
> in
>     > error, please notify the sender immediately by return email and
> delete the
>     > message and any attachments from your system.
>     >
>
>
>     --
>     Sincerely yours
>     Mikhail Khludnev
>
>
> This email is intended for the person(s) to whom it is addressed and may
> contain information that is PRIVILEGED or CONFIDENTIAL. Any unauthorized
> use, distribution, copying, or disclosure by any person other than the
> addressee(s) is strictly prohibited. If you have received this email in
> error, please notify the sender immediately by return email and delete the
> message and any attachments from your system.
>


--
Sincerely yours
Mikhail Khludnev