Modify data-conf.xml on the runtime

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Modify data-conf.xml on the runtime

rameshkjes
Hi,

I am doing configuration of solr with the xml and pdf dataset, it works
perfect. But, I want to modify few things:
Such as, we can see below, "baseDir" and "filePrefix" is being defined
manually. I want this to be defined on the runtime. Consider that I have GUI
and user is specifying the directories for datasets. So, I have separate
java file that does this job. But, I don't know how to use data of that
class to configure "baseDir" and "filePrefix". Could anyone help me that how
can i access that data from java class and fill in the data-conf.xml.

These are few fields of my data-conf.xml file.

<entity name="file" processor="FileListEntityProcessor"
                        baseDir="C:/Users/kuma_rm/github/ML/dataset/XmlMetaData/"
                        fileName=".*xml" rootEntity="false" datasource="null"
                        transformer="pathos.utils.transformer.FilePathTransformer">

                         
    <field column="filePath"
filePrefix="C:/Users/kuma_rm/github/ML/dataset/PdfMetaData/"
    fileSuffix=".pdf" oldFileSuffix=".xml" srcColName="file"/>



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Modify data-conf.xml on the runtime

Shawn Heisey-2
On 4/11/2018 9:21 AM, rameshkjes wrote:
> I am doing configuration of solr with the xml and pdf dataset, it works
> perfect. But, I want to modify few things:
> Such as, we can see below, "baseDir" and "filePrefix" is being defined
> manually. I want this to be defined on the runtime.

The way I would do this is to have DIH grab that information from URL
parameters on the request to start the import.

So your URL would look like this:

?command=full-import&baseDir=somePath&filePathPrefix=someOtherPath

And your config would look like this:

<entity name="file" processor="FileListEntityProcessor"
  baseDir="${dih.request.baseDir}"
  fileName=".*xml" rootEntity="false" datasource="null"
  transformer="pathos.utils.transformer.FilePathTransformer">
  <field column="filePath"
    filePrefix="/${dih.request.filePathPrefix/}"

I *think* that the property substitution has to happen inside the quotes
like I have shown, but I'm not completely positive.  I'm using that
functionality for parts of an SQL query, which is definitely inside the
quotes.

Be aware that you will need to run the values in the URL parameters
through an encoder function to make sure they're properly encoded.  For
instance, the value you're using for "filePrefix" would actually look
like this on the URL:

C%3a%2fUsers%2fkuma_rm%2fgithub%2fML%2fdataset%2fPdfMetaData%2f

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Modify data-conf.xml on the runtime

rameshkjes
I am using datasource as "FileDataSource", how can i grab that information
from url?  



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Modify data-conf.xml on the runtime

rameshkjes
In reply to this post by Shawn Heisey-2
How data will be imported using dih, if path of data is not provided? What I
want is to provide the link of dataset files during runtime.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Modify data-conf.xml on the runtime

Shawn Heisey
In reply to this post by rameshkjes
On 4/13/2018 2:54 AM, rameshkjes wrote:
> I am using datasource as "FileDataSource", how can i grab that information
> from url?

I'm pretty sure that you can provide pretty much ANY information in the
DIH config file from a URL parameter, using the ${dih.request.XXXX} syntax.

To answer your other question you sent in a different message: With the
standard syntax, imports are going to fail if you don't supply all the
parameters.  I think you can specify a default value in that syntax like
this, which will get used if the property isn't defined:

${dih.request.XXXX:defaultValue}

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Modify data-conf.xml on the runtime

rameshkjes
Actually I am trying to approach this problem from another way.
I am taking user input from gui which is direcotory of dataset, and saving
that path in properties file. Since I am using Maven, so I am able to access
that path in my pom file using properties tag. So, now is it possible to use
that properties variable from pom.xml to the data-conf.xml? do you think, it
is right way?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Modify data-conf.xml on the runtime

Shawn Heisey-2
On 4/25/2018 4:12 AM, rameshkjes wrote:
> Actually I am trying to approach this problem from another way.
> I am taking user input from gui which is direcotory of dataset, and saving
> that path in properties file. Since I am using Maven, so I am able to access
> that path in my pom file using properties tag. So, now is it possible to use
> that properties variable from pom.xml to the data-conf.xml? do you think, it
> is right way?

Maven is used to build a program from source.  Maven is not involved
when running Solr unless you're doing something non-standard.  It's
difficult for this list to support a non-standard setup.

What are you trying to do?  Perhaps there is a way to do it that doesn't
involve using unexpected tools.

If you must use Maven, you may be on your own in trying to get it to
work with Solr.  I have never heard of such an integration before except
at build time.  The maven build for Lucene/Solr is not the standard
build tool.  The standard build uses ant.

Thanks,
Shawn