Sql server data import

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Sql server data import

Verthosa
Hello, i managed to set up a connection to my sql server to import data into
Solr. The idea is to import filetables but for now i first want to get it
working using regular tables. So i created

*data-config.xml*
<dataConfig>  
  <dataSource type="JdbcDataSource"    
     driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"    
   
url="jdbc:sqlserver://localhost;databaseName=inConnexion_Tenant2;integratedSecurity=true"
/>  
  <document>  
   <entity name="Dashboard" pk="id" query="SELECT Id,PublicId FROM
foundation.Shops">  
         <field column="Id" name="Id"/>  
     <field column="PublicId" name="PublicId" />
   </entity>  
  </document>  
  </dataConfig>  

*schema.xml*
i added
<field name="Id" type="string" indexed="true" stored="true" required="true"
multiValued="false" />
<field name="PublicId" type="string" indexed="true" stored="true"
multiValued="false"/>

and changed uniqueKey entry to
<uniqueKey>Id</uniqueKey>

When i want to import my data (which is just data like Id: 5, PublicId:
"test"), i get the following error in the logging.

Error creating document : SolrInputDocument(fields: [PublicId=10065,&#8203;
Id=117])


I tried all sorts of things but can't get it fixed. Is anyone want to give
me a hand?

thanks in advance!




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Sql server data import

Alexandre Rafalovitch
Which version of Solr is it? Because we have not used schema.xml for a
very long time. It has been managed-schema instead.

Also, have you tried using DIH example that uses database and
modifying it just enough to read data from your database. Even if it
has a lot of extra junk, this would test half of the pipeline, which
you can then transfer to the clean setup.

Regards,
   Alex.
On Fri, 9 Nov 2018 at 08:09, Verthosa <[hidden email]> wrote:

>
> Hello, i managed to set up a connection to my sql server to import data into
> Solr. The idea is to import filetables but for now i first want to get it
> working using regular tables. So i created
>
> *data-config.xml*
> <dataConfig>
>   <dataSource type="JdbcDataSource"
>      driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
>
> url="jdbc:sqlserver://localhost;databaseName=inConnexion_Tenant2;integratedSecurity=true"
> />
>   <document>
>    <entity name="Dashboard" pk="id" query="SELECT Id,PublicId FROM
> foundation.Shops">
>          <field column="Id" name="Id"/>
>      <field column="PublicId" name="PublicId" />
>    </entity>
>   </document>
>   </dataConfig>
>
> *schema.xml*
> i added
> <field name="Id" type="string" indexed="true" stored="true" required="true"
> multiValued="false" />
> <field name="PublicId" type="string" indexed="true" stored="true"
> multiValued="false"/>
>
> and changed uniqueKey entry to
> <uniqueKey>Id</uniqueKey>
>
> When i want to import my data (which is just data like Id: 5, PublicId:
> "test"), i get the following error in the logging.
>
> Error creating document : SolrInputDocument(fields: [PublicId=10065,&#8203;
> Id=117])
>
>
> I tried all sorts of things but can't get it fixed. Is anyone want to give
> me a hand?
>
> thanks in advance!
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

RE: Sql server data import

Gu, Steve (CDC/DDPHSS/OS) (CTR)
In reply to this post by Verthosa
What is "&#8203;"  in the PublicId?  Is it part of the data?  Did you check if the special characters in your data cause the problem?

Steve

###########
Error creating document : SolrInputDocument(fields: [PublicId=10065,&#8203;
Id=117])

-----Original Message-----
From: Verthosa <[hidden email]>
Sent: Friday, November 9, 2018 7:51 AM
To: [hidden email]
Subject: Sql server data import

Hello, i managed to set up a connection to my sql server to import data into Solr. The idea is to import filetables but for now i first want to get it working using regular tables. So i created

*data-config.xml*
<dataConfig>  
  <dataSource type="JdbcDataSource"    
     driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"    
   
url="jdbc:sqlserver://localhost;databaseName=inConnexion_Tenant2;integratedSecurity=true"
/>  
  <document>  
   <entity name="Dashboard" pk="id" query="SELECT Id,PublicId FROM
foundation.Shops">  
         <field column="Id" name="Id"/>  
     <field column="PublicId" name="PublicId" />
   </entity>  
  </document>  
  </dataConfig>  

*schema.xml*
i added
<field name="Id" type="string" indexed="true" stored="true" required="true"
multiValued="false" />
<field name="PublicId" type="string" indexed="true" stored="true"
multiValued="false"/>

and changed uniqueKey entry to
<uniqueKey>Id</uniqueKey>

When i want to import my data (which is just data like Id: 5, PublicId:
"test"), i get the following error in the logging.

Error creating document : SolrInputDocument(fields: [PublicId=10065,&#8203;
Id=117])


I tried all sorts of things but can't get it fixed. Is anyone want to give me a hand?

thanks in advance!




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Sql server data import

Verthosa
In reply to this post by Alexandre Rafalovitch
Hello, i managed to fix the problem. I'm using Solr 7.5.0. My problem was
that in the server logs i got "This Indexschema is not mutable" (i did not
know about the logs folder, so i just found out 5 minutes ago). I fixed it
by modifying solrconfig.xml to

<updateRequestProcessorChain
name=&quot;add-unknown-fields-to-the-schema&quot;
default=&quot;${update.autoCreateFields:&lt;b>false*}"
                           
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
    <processor class="solr.LogUpdateProcessorFactory"/>
    <processor class="solr.DistributedUpdateProcessorFactory"/>
    <processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>

Since then the indexing is done correctly. I even got the blob fields
indexation working now ! Thanks for your reply, everything is fixed for now.




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Sql server data import

Erick Erickson
Ok, what that means is you're letting Solr do its best to figure out
what fields you should have in the schema and how they're defined.
Almost invariably, you can do better by explicitly defining the fields
you need in your schema rather than enabling add-unknown----. It's
fine for getting started, but not advised for production.

Best,
Erick
On Fri, Nov 9, 2018 at 7:52 AM Verthosa <[hidden email]> wrote:

>
> Hello, i managed to fix the problem. I'm using Solr 7.5.0. My problem was
> that in the server logs i got "This Indexschema is not mutable" (i did not
> know about the logs folder, so i just found out 5 minutes ago). I fixed it
> by modifying solrconfig.xml to
>
> <updateRequestProcessorChain
> name=&quot;add-unknown-fields-to-the-schema&quot;
> default=&quot;${update.autoCreateFields:&lt;b>false*}"
>
> processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
>     <processor class="solr.LogUpdateProcessorFactory"/>
>     <processor class="solr.DistributedUpdateProcessorFactory"/>
>     <processor class="solr.RunUpdateProcessorFactory"/>
> </updateRequestProcessorChain>
>
> Since then the indexing is done correctly. I even got the blob fields
> indexation working now ! Thanks for your reply, everything is fixed for now.
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html