Issue with delta import

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Issue with delta import

bhargava ravali koganti
Hi,

I'm trying to integrate Solr and Cassandra. I"m facing problem with delta
import. For every 10 minutes I'm running deltaquery using cron job. If any
changes in the data based on last index time, it has to fetch the data(as
far as my knowledge), however, it keeps fetching the whole data
irrespective of changes.

My problem:
https://stackoverflow.com/questions/45304803/deltaimport-fetches-all-the-data

Looking forward to hear from you.

Thanks,
Bhargava Ravali Koganti
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with delta import

Sujay Bawaskar-2
can you please try ${dih.last_index_time} instead of
${dataimporter.last_index_time}.

On Wed, Jul 26, 2017 at 2:33 PM, bhargava ravali koganti <
[hidden email]> wrote:

> Hi,
>
> I'm trying to integrate Solr and Cassandra. I"m facing problem with delta
> import. For every 10 minutes I'm running deltaquery using cron job. If any
> changes in the data based on last index time, it has to fetch the data(as
> far as my knowledge), however, it keeps fetching the whole data
> irrespective of changes.
>
> My problem:
> https://stackoverflow.com/questions/45304803/deltaimport-fetches-all-the-
> data
>
> Looking forward to hear from you.
>
> Thanks,
> Bhargava Ravali Koganti
>



--
Thanks,
Sujay P Bawaskar
M:+91-77091 53669
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with delta import

bhargava ravali koganti
Tried it had no impact.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with delta import

vrindavda
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with delta import

bhargava ravali koganti
Does this way even handle the deletes?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with delta import

vrindavda
yes.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with delta import

bhargava ravali koganti
I'm getting error like this

Exception while processing: test document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT * FROM person WHERE 'false' != 'false' OR last_modified > '2017-08-09 15:48:02' Processing Document # 1
        at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:318)
        at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:279)
        at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:54)
        at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
        at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
        at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
        at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
        at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
        at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
        at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: com.datastax.driver.core.exceptions.SyntaxError: line 1:27 no viable alternative at input 'false' (SELECT * FROM person WHERE ['fals]...)
        at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:58)
        at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:24)
        at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
        at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)
        at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:43)
        at com.dbschema.CassandraPreparedStatement.executeQuery(CassandraPreparedStatement.java:73)
        at com.dbschema.CassandraPreparedStatement.execute(CassandraPreparedStatement.java:114)
        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.executeStatement(JdbcDataSource.java:338)
        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:313)
        ... 12 more
Caused by: com.datastax.driver.core.exceptions.SyntaxError: line 1:27 no viable alternative at input 'false' (SELECT * FROM person WHERE ['fals]...)
        at com.datastax.driver.core.Responses$Error.asException(Responses.java:144)
        at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
        at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:186)
        at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:50)
        at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:817)
        at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651)
        at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1077)
        at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1000)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:946)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:831)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:322)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:703)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:748)<quote author="vrindavda">
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with delta import

Rick Leir-2
It says here

https://cwiki.apache.org/confluence/display/solr/Parallel+SQL+Interface#ParallelSQLInterface-SolrSQLSyntax

that you need to escape the FROM.

Did you try it without the

'false' != 'false' OR


On 2017-08-16 04:07 AM, bhargava ravali koganti wrote:

> I'm getting error like this
>
> Exception while processing: test document : SolrInputDocument(fields:
> []):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> execute query: SELECT * FROM person WHERE 'false' != 'false' OR
> last_modified > '2017-08-09 15:48:02' Processing Document # 1
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:318)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:279)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:54)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
> Caused by: com.datastax.driver.core.exceptions.SyntaxError: line 1:27 no
> viable alternative at input 'false' (SELECT * FROM person WHERE ['fals]...)
> at
> com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:58)
> at

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with delta import

bhargava ravali koganti
Yes it does work in that case. And one more thing is, I'm use my database as cassandra not sql. Probably, that must be the problem.
Loading...