Some newby questions ...

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Some newby questions ...

Christian Dannemann
Hi Everyone,

I've managed to successfully install solr on my server, and it's running
and I have created a core and indexed some 1000 pdf files... Searches work,
and so far, so good.

There are two issues that I can't resolve even though I have tried for a
good 8 hours, so I feel I can hopefully reach out having tried myself
before asking stupid questions.

Environment:

   - Ubuntu OpenJDK 64-Bit Server VM 11.0.4
   11.0.4+11-post-Ubuntu-1ubuntu218.04.3
   - Processors8
   - Args-DSTOP.KEY=solrrocks-DSTOP.PORT=9539-Djetty.home=/opt/solr/server
   -Djetty.port=10539-Dlog4j.configurationFile=file:/var/solr/log4j2.xml
   -Dsolr.data.home=
   -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
   -Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=10539
   -Dsolr.log.dir=/var/solr/logs-Dsolr.log.muteconsole
   -Dsolr.solr.home=/var/solr/data-Duser.timezone=UTC-XX:+AlwaysPreTouch
   -XX:+ParallelRefProcEnabled-XX:+PerfDisableSharedMem-XX:+UseG1GC
   -XX:+UseLargePages-XX:MaxGCPauseMillis=250-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
   10539 /var/solr/logs
   -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
   -Xms512m-Xmx512m-Xss256k


and my core: CWD:
/opt/solr-8.3.0/serverInstance:/var/solr/data/meeting_minutesData:
/var/solr/data/meeting_minutes/dataIndex:
/var/solr/data/meeting_minutes/data/indexImpl:
org.apache.solr.core.NRTCachingDirectoryFactory
Issue 1: I want to secure my server with basic authentication (that's why
I'm running on port 10539 at the moment, but that's not security ...

I've put a file security.json in
/opt/solr/server/solr/configsets/_default/conf, but that doesn't do
anything. No request for a password when I access the web interface or do
searches
Any help on that would be highly appreciated.
Issue 2: I would like to index a lot of emails that reside on a local imap
server (dovecot). I have managed to configure that, and my
configuration shows up in the "dataimport" menu on my core
(meeting_minutes) in the web interface: <dataConfig> <document> <entity
processor="MailEntityProcessor" user="christian" password="******" host=
"localhost" protocol="imap" fetchMailsSince="2010-06-30 00:00:00" batchSize=
"200" folders="Secretary" processAttachement="false" name="mail_entity"/> </
document> </dataConfig>

(the password does of course appear in cleartext in my config file ....)
I can execute the command from the web interface, and get the message

Last Update: 21:08:18
Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0

My dovecot server shows nothing in the logs, and I can change the password
and don't get any error message - solr always claims it executed, and that
there were 0 Fetched etc.
I manually have logged into the dovecot server with imap, and the folder
"Secretary" does indeed show up when I list the folders. It appears solr
doesn't even try to connect to my dovecot server.

I can't use the debug option, as apparently, I have to
enable.dih.dataConfigParam has to be set to true, but I don't have the
slightest clue where and how to set that. (It took me quite a while to
figure out that changing the default port had to be done in /etc/default/
solr.in.sh, rather than in solrconfig.xml...)

The performance is great, and I love the stuff that works, but I find it
very hard to get solr going, configure it, and get my imap dataConfig going.

Any help is greatly appreciated!

Best regards,

Christian
Reply | Threaded
Open this post in threaded view
|

Re: Some newby questions ...

Christian Dannemann
... sorry, forgot to post solr.log:

2019-11-26 21:24:11.242 INFO  (qtp1884155890-24) [   x:meeting_minutes]
o.a.s.h.d.DataImporter Loading DIH Configuration: mail-data-config.xml
2019-11-26 21:24:11.245 INFO  (qtp1884155890-24) [   x:meeting_minutes]
o.a.s.h.d.c.DIHConfiguration id is a required field in SolrSchema . But not
found in DataConfig
2019-11-26 21:24:11.245 INFO  (qtp1884155890-24) [   x:meeting_minutes]
o.a.s.h.d.DataImporter Data Configuration loaded successfully
2019-11-26 21:24:11.246 INFO  (qtp1884155890-24) [   x:meeting_minutes]
o.a.s.c.S.Request [meeting_minutes]  webapp=/solr path=/dataimport
params={core=meeting_minutes&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=full-import&_=1574803444168&verbose=false}
status=0 QTime=4
2019-11-26 21:24:11.246 INFO  (Thread-18) [   ] o.a.s.h.d.DataImporter
Starting Full Import
2019-11-26 21:24:11.252 WARN  (Thread-18) [   ]
o.a.s.h.d.SimplePropertiesWriter Unable to read: dataimport.properties
2019-11-26 21:24:11.257 INFO  (Thread-18) [   ]
o.a.s.u.p.LogUpdateProcessorFactory [meeting_minutes]  webapp=/solr
path=/dataimport
params={core=meeting_minutes&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=full-import&_=1574803444168&verbose=false}
status=0 QTime=4{} 0 15
2019-11-26 21:24:11.294 INFO  (qtp1884155890-42) [   x:meeting_minutes]
o.a.s.c.S.Request [meeting_minutes]  webapp=/solr path=/dataimport
params={indent=on&wt=json&command=status&_=1574803444168} status=0 QTime=0

Best regards,

Christian

On Tue, 26 Nov 2019 at 21:17, Christian Dannemann <[hidden email]>
wrote:

> Hi Everyone,
>
> I've managed to successfully install solr on my server, and it's running
> and I have created a core and indexed some 1000 pdf files... Searches work,
> and so far, so good.
>
> There are two issues that I can't resolve even though I have tried for a
> good 8 hours, so I feel I can hopefully reach out having tried myself
> before asking stupid questions.
>
> Environment:
>
>    - Ubuntu OpenJDK 64-Bit Server VM 11.0.4
>    11.0.4+11-post-Ubuntu-1ubuntu218.04.3
>    - Processors8
>    - Args-DSTOP.KEY=solrrocks-DSTOP.PORT=9539-Djetty.home=/opt/solr/server
>    -Djetty.port=10539-Dlog4j.configurationFile=file:/var/solr/log4j2.xml
>    -Dsolr.data.home=
>    -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
>    -Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=10539
>    -Dsolr.log.dir=/var/solr/logs-Dsolr.log.muteconsole
>    -Dsolr.solr.home=/var/solr/data-Duser.timezone=UTC-XX:+AlwaysPreTouch
>    -XX:+ParallelRefProcEnabled-XX:+PerfDisableSharedMem-XX:+UseG1GC
>    -XX:+UseLargePages-XX:MaxGCPauseMillis=250-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
>    10539 /var/solr/logs
>    -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
>    -Xms512m-Xmx512m-Xss256k
>
>
> and my core: CWD:
> /opt/solr-8.3.0/serverInstance:/var/solr/data/meeting_minutesData:
> /var/solr/data/meeting_minutes/dataIndex:
> /var/solr/data/meeting_minutes/data/indexImpl:
> org.apache.solr.core.NRTCachingDirectoryFactory
> Issue 1: I want to secure my server with basic authentication (that's why
> I'm running on port 10539 at the moment, but that's not security ...
>
> I've put a file security.json in
> /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> anything. No request for a password when I access the web interface or do
> searches
> Any help on that would be highly appreciated.
> Issue 2: I would like to index a lot of emails that reside on a local imap
> server (dovecot). I have managed to configure that, and my
> configuration shows up in the "dataimport" menu on my core
> (meeting_minutes) in the web interface: <dataConfig> <document> <entity
> processor="MailEntityProcessor" user="christian" password="******" host=
> "localhost" protocol="imap" fetchMailsSince="2010-06-30 00:00:00"
> batchSize="200" folders="Secretary" processAttachement="false" name=
> "mail_entity"/> </document> </dataConfig>
>
> (the password does of course appear in cleartext in my config file ....)
> I can execute the command from the web interface, and get the message
>
> Last Update: 21:08:18
> Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0
>
> My dovecot server shows nothing in the logs, and I can change the password
> and don't get any error message - solr always claims it executed, and that
> there were 0 Fetched etc.
> I manually have logged into the dovecot server with imap, and the folder
> "Secretary" does indeed show up when I list the folders. It appears solr
> doesn't even try to connect to my dovecot server.
>
> I can't use the debug option, as apparently, I have to
> enable.dih.dataConfigParam has to be set to true, but I don't have the
> slightest clue where and how to set that. (It took me quite a while to
> figure out that changing the default port had to be done in /etc/default/
> solr.in.sh, rather than in solrconfig.xml...)
>
> The performance is great, and I love the stuff that works, but I find it
> very hard to get solr going, configure it, and get my imap dataConfig going.
>
> Any help is greatly appreciated!
>
> Best regards,
>
> Christian
>
>
>
>
>
>
>
>

--
Christian Dannemann
Managing Director
Merus Software Ltd
http://merus.eu
DDI:+44 1453 708610
Reply | Threaded
Open this post in threaded view
|

Re: Some newby questions ...

Jan Høydahl / Cominvent
In reply to this post by Christian Dannemann
Try

bin/solr auth enable -blockUnknown true -credentials user:password -z local host:11539

That should enable basic auth for you given you run in cloud mode (-c switch)

Jan Høydahl

> 26. nov. 2019 kl. 22:18 skrev Christian Dannemann <[hidden email]>:
>
> Hi Everyone,
>
> I've managed to successfully install solr on my server, and it's running
> and I have created a core and indexed some 1000 pdf files... Searches work,
> and so far, so good.
>
> There are two issues that I can't resolve even though I have tried for a
> good 8 hours, so I feel I can hopefully reach out having tried myself
> before asking stupid questions.
>
> Environment:
>
>   - Ubuntu OpenJDK 64-Bit Server VM 11.0.4
>   11.0.4+11-post-Ubuntu-1ubuntu218.04.3
>   - Processors8
>   - Args-DSTOP.KEY=solrrocks-DSTOP.PORT=9539-Djetty.home=/opt/solr/server
>   -Djetty.port=10539-Dlog4j.configurationFile=file:/var/solr/log4j2.xml
>   -Dsolr.data.home=
>   -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
>   -Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=10539
>   -Dsolr.log.dir=/var/solr/logs-Dsolr.log.muteconsole
>   -Dsolr.solr.home=/var/solr/data-Duser.timezone=UTC-XX:+AlwaysPreTouch
>   -XX:+ParallelRefProcEnabled-XX:+PerfDisableSharedMem-XX:+UseG1GC
>   -XX:+UseLargePages-XX:MaxGCPauseMillis=250-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
>   10539 /var/solr/logs
>   -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
>   -Xms512m-Xmx512m-Xss256k
>
>
> and my core: CWD:
> /opt/solr-8.3.0/serverInstance:/var/solr/data/meeting_minutesData:
> /var/solr/data/meeting_minutes/dataIndex:
> /var/solr/data/meeting_minutes/data/indexImpl:
> org.apache.solr.core.NRTCachingDirectoryFactory
> Issue 1: I want to secure my server with basic authentication (that's why
> I'm running on port 10539 at the moment, but that's not security ...
>
> I've put a file security.json in
> /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> anything. No request for a password when I access the web interface or do
> searches
> Any help on that would be highly appreciated.
> Issue 2: I would like to index a lot of emails that reside on a local imap
> server (dovecot). I have managed to configure that, and my
> configuration shows up in the "dataimport" menu on my core
> (meeting_minutes) in the web interface: <dataConfig> <document> <entity
> processor="MailEntityProcessor" user="christian" password="******" host=
> "localhost" protocol="imap" fetchMailsSince="2010-06-30 00:00:00" batchSize=
> "200" folders="Secretary" processAttachement="false" name="mail_entity"/> </
> document> </dataConfig>
>
> (the password does of course appear in cleartext in my config file ....)
> I can execute the command from the web interface, and get the message
>
> Last Update: 21:08:18
> Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0
>
> My dovecot server shows nothing in the logs, and I can change the password
> and don't get any error message - solr always claims it executed, and that
> there were 0 Fetched etc.
> I manually have logged into the dovecot server with imap, and the folder
> "Secretary" does indeed show up when I list the folders. It appears solr
> doesn't even try to connect to my dovecot server.
>
> I can't use the debug option, as apparently, I have to
> enable.dih.dataConfigParam has to be set to true, but I don't have the
> slightest clue where and how to set that. (It took me quite a while to
> figure out that changing the default port had to be done in /etc/default/
> solr.in.sh, rather than in solrconfig.xml...)
>
> The performance is great, and I love the stuff that works, but I find it
> very hard to get solr going, configure it, and get my imap dataConfig going.
>
> Any help is greatly appreciated!
>
> Best regards,
>
> Christian
Reply | Threaded
Open this post in threaded view
|

Re: Some newby questions ...

Christian Dannemann
Hi Jan,

I'm afraid I don't run in cloud mode, and I get
Failed to parse command-line arguments due to: Missing argument for option:
solrIncludeFile
usage: org.apache.solr.util.SolrCLI

Best regards,

Christian

On Tue, 26 Nov 2019 at 21:30, Jan Høydahl <[hidden email]> wrote:

> Try
>
> bin/solr auth enable -blockUnknown true -credentials user:password -z
> local host:11539
>
> That should enable basic auth for you given you run in cloud mode (-c
> switch)
>
> Jan Høydahl
>
> > 26. nov. 2019 kl. 22:18 skrev Christian Dannemann <[hidden email]>:
> >
> > Hi Everyone,
> >
> > I've managed to successfully install solr on my server, and it's running
> > and I have created a core and indexed some 1000 pdf files... Searches
> work,
> > and so far, so good.
> >
> > There are two issues that I can't resolve even though I have tried for a
> > good 8 hours, so I feel I can hopefully reach out having tried myself
> > before asking stupid questions.
> >
> > Environment:
> >
> >   - Ubuntu OpenJDK 64-Bit Server VM 11.0.4
> >   11.0.4+11-post-Ubuntu-1ubuntu218.04.3
> >   - Processors8
> >   - Args-DSTOP.KEY=solrrocks-DSTOP.PORT=9539-Djetty.home=/opt/solr/server
> >   -Djetty.port=10539-Dlog4j.configurationFile=file:/var/solr/log4j2.xml
> >   -Dsolr.data.home=
> >   -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
> >   -Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=10539
> >   -Dsolr.log.dir=/var/solr/logs-Dsolr.log.muteconsole
> >   -Dsolr.solr.home=/var/solr/data-Duser.timezone=UTC-XX:+AlwaysPreTouch
> >   -XX:+ParallelRefProcEnabled-XX:+PerfDisableSharedMem-XX:+UseG1GC
> >
>  -XX:+UseLargePages-XX:MaxGCPauseMillis=250-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
> >   10539 /var/solr/logs
> >
>  -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
> >   -Xms512m-Xmx512m-Xss256k
> >
> >
> > and my core: CWD:
> > /opt/solr-8.3.0/serverInstance:/var/solr/data/meeting_minutesData:
> > /var/solr/data/meeting_minutes/dataIndex:
> > /var/solr/data/meeting_minutes/data/indexImpl:
> > org.apache.solr.core.NRTCachingDirectoryFactory
> > Issue 1: I want to secure my server with basic authentication (that's why
> > I'm running on port 10539 at the moment, but that's not security ...
> >
> > I've put a file security.json in
> > /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> > anything. No request for a password when I access the web interface or do
> > searches
> > Any help on that would be highly appreciated.
> > Issue 2: I would like to index a lot of emails that reside on a local
> imap
> > server (dovecot). I have managed to configure that, and my
> > configuration shows up in the "dataimport" menu on my core
> > (meeting_minutes) in the web interface: <dataConfig> <document> <entity
> > processor="MailEntityProcessor" user="christian" password="******" host=
> > "localhost" protocol="imap" fetchMailsSince="2010-06-30 00:00:00"
> batchSize=
> > "200" folders="Secretary" processAttachement="false"
> name="mail_entity"/> </
> > document> </dataConfig>
> >
> > (the password does of course appear in cleartext in my config file ....)
> > I can execute the command from the web interface, and get the message
> >
> > Last Update: 21:08:18
> > Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0
> >
> > My dovecot server shows nothing in the logs, and I can change the
> password
> > and don't get any error message - solr always claims it executed, and
> that
> > there were 0 Fetched etc.
> > I manually have logged into the dovecot server with imap, and the folder
> > "Secretary" does indeed show up when I list the folders. It appears solr
> > doesn't even try to connect to my dovecot server.
> >
> > I can't use the debug option, as apparently, I have to
> > enable.dih.dataConfigParam has to be set to true, but I don't have the
> > slightest clue where and how to set that. (It took me quite a while to
> > figure out that changing the default port had to be done in /etc/default/
> > solr.in.sh, rather than in solrconfig.xml...)
> >
> > The performance is great, and I love the stuff that works, but I find it
> > very hard to get solr going, configure it, and get my imap dataConfig
> going.
> >
> > Any help is greatly appreciated!
> >
> > Best regards,
> >
> > Christian
>


--
Christian Dannemann
Managing Director
Merus Software Ltd
http://merus.eu
DDI:+44 1453 708610
Reply | Threaded
Open this post in threaded view
|

Re: Some newby questions ...

Shawn Heisey-2
In reply to this post by Christian Dannemann
On 11/26/2019 2:17 PM, Christian Dannemann wrote:
> Issue 1: I want to secure my server with basic authentication (that's why
> I'm running on port 10539 at the moment, but that's not security ...
>
> I've put a file security.json in
> /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> anything.

Your solr home appears to be /var/solr/data ... that is where you need
to place the security.json file for a setup that is not running SolrCloud.

https://lucene.apache.org/solr/guide/8_3/authentication-and-authorization-plugins.html#in-standalone-mode

If you install your Solr server in a network location where it cannot be
reached by people you cannot trust, there is usually no need for
security measures like authentication.

> Issue 2: I would like to index a lot of emails that reside on a local imap
> server (dovecot).

I've got no idea how to use the dataimport imap capability.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Some newby questions ...

Christian Dannemann
Hmmm... I tried that as well, but it doesn't pick up the security.json
settings.

I run this instance on a computer that is on the internet, so just changing
the port is asking for trouble.

Looks like nobody knows how to import imap data ....

Best regards,

Christian

On Tue, 26 Nov 2019 at 23:55, Shawn Heisey <[hidden email]> wrote:

> On 11/26/2019 2:17 PM, Christian Dannemann wrote:
> > Issue 1: I want to secure my server with basic authentication (that's why
> > I'm running on port 10539 at the moment, but that's not security ...
> >
> > I've put a file security.json in
> > /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> > anything.
>
> Your solr home appears to be /var/solr/data ... that is where you need
> to place the security.json file for a setup that is not running SolrCloud.
>
>
> https://lucene.apache.org/solr/guide/8_3/authentication-and-authorization-plugins.html#in-standalone-mode
>
> If you install your Solr server in a network location where it cannot be
> reached by people you cannot trust, there is usually no need for
> security measures like authentication.
>
> > Issue 2: I would like to index a lot of emails that reside on a local
> imap
> > server (dovecot).
>
> I've got no idea how to use the dataimport imap capability.
>
> Thanks,
> Shawn
>


--
Christian Dannemann
Managing Director
Merus Software Ltd
http://merus.eu
DDI:+44 1453 708610