Problem with Password Decryption in Data Import Handler

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with Password Decryption in Data Import Handler

Jamie Jackson
Hi Folks,

(Using Solr 5.5.3.)

As far as I know, the only place where encrypted password use is documented
is in
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
under the "Configuring the DIH Configuration File", in a comment in the
sample XML file:

<!--
Alternately the password can be encrypted as follows. This is the value
obtained as a result of the command
openssl enc -aes-128-cbc -a -salt -in pwd.txt
password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
WHen the password is encrypted, you must provide an extra attribute
encryptKeyFile="/location/of/encryptionkey"
This file should a text file with a single line containing the
encrypt/decrypt password
-->

Anyway, I can encrypt just fine:

$ openssl enc -aes-128-cbc -a -salt -in stgps.txt
enter aes-128-cbc encryption password:
Verifying - enter aes-128-cbc encryption password:
U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=


I can also decrypt just fine from the command line.

However, if I use the encrypted password and encryptKeyFile in the config
file, I end up with an error: "String length must be a multiple of four."

https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9

How do I get this working?

Thanks,
Jamie
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Kevin Risden-2
I haven't tried this but is it possible there is a new line at the end in
the file?

If you did something like echo "" > file.txt then there would be a new
line. Use echo -n "" > file.txt

Also you should be able to check how many characters are in the file.

Kevin Risden

On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]> wrote:

> Hi Folks,
>
> (Using Solr 5.5.3.)
>
> As far as I know, the only place where encrypted password use is documented
> is in
> https://cwiki.apache.org/confluence/display/solr/
> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> under the "Configuring the DIH Configuration File", in a comment in the
> sample XML file:
>
> <!--
> Alternately the password can be encrypted as follows. This is the value
> obtained as a result of the command
> openssl enc -aes-128-cbc -a -salt -in pwd.txt
> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
> WHen the password is encrypted, you must provide an extra attribute
> encryptKeyFile="/location/of/encryptionkey"
> This file should a text file with a single line containing the
> encrypt/decrypt password
> -->
>
> Anyway, I can encrypt just fine:
>
> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> enter aes-128-cbc encryption password:
> Verifying - enter aes-128-cbc encryption password:
> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>
>
> I can also decrypt just fine from the command line.
>
> However, if I use the encrypted password and encryptKeyFile in the config
> file, I end up with an error: "String length must be a multiple of four."
>
> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
>
> How do I get this working?
>
> Thanks,
> Jamie
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Alexandre Rafalovitch
How long is the encryption key (file content)? Because the code I am
looking at seems to expect it to be at most 100 characters.

Regards,
   Alex.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/


On 6 October 2016 at 23:26, Kevin Risden <[hidden email]> wrote:

> I haven't tried this but is it possible there is a new line at the end in
> the file?
>
> If you did something like echo "" > file.txt then there would be a new
> line. Use echo -n "" > file.txt
>
> Also you should be able to check how many characters are in the file.
>
> Kevin Risden
>
> On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]> wrote:
>
>> Hi Folks,
>>
>> (Using Solr 5.5.3.)
>>
>> As far as I know, the only place where encrypted password use is documented
>> is in
>> https://cwiki.apache.org/confluence/display/solr/
>> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
>> under the "Configuring the DIH Configuration File", in a comment in the
>> sample XML file:
>>
>> <!--
>> Alternately the password can be encrypted as follows. This is the value
>> obtained as a result of the command
>> openssl enc -aes-128-cbc -a -salt -in pwd.txt
>> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
>> WHen the password is encrypted, you must provide an extra attribute
>> encryptKeyFile="/location/of/encryptionkey"
>> This file should a text file with a single line containing the
>> encrypt/decrypt password
>> -->
>>
>> Anyway, I can encrypt just fine:
>>
>> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
>> enter aes-128-cbc encryption password:
>> Verifying - enter aes-128-cbc encryption password:
>> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>>
>>
>> I can also decrypt just fine from the command line.
>>
>> However, if I use the encrypted password and encryptKeyFile in the config
>> file, I end up with an error: "String length must be a multiple of four."
>>
>> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
>>
>> How do I get this working?
>>
>> Thanks,
>> Jamie
>>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Jamie Jackson
In reply to this post by Kevin Risden-2
I tried it both ways yesterday--with a newline and without.

On Thu, Oct 6, 2016 at 12:26 PM, Kevin Risden <[hidden email]>
wrote:

> I haven't tried this but is it possible there is a new line at the end in
> the file?
>
> If you did something like echo "" > file.txt then there would be a new
> line. Use echo -n "" > file.txt
>
> Also you should be able to check how many characters are in the file.
>
> Kevin Risden
>
> On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>
> wrote:
>
> > Hi Folks,
> >
> > (Using Solr 5.5.3.)
> >
> > As far as I know, the only place where encrypted password use is
> documented
> > is in
> > https://cwiki.apache.org/confluence/display/solr/
> > Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> > under the "Configuring the DIH Configuration File", in a comment in the
> > sample XML file:
> >
> > <!--
> > Alternately the password can be encrypted as follows. This is the value
> > obtained as a result of the command
> > openssl enc -aes-128-cbc -a -salt -in pwd.txt
> > password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
> > WHen the password is encrypted, you must provide an extra attribute
> > encryptKeyFile="/location/of/encryptionkey"
> > This file should a text file with a single line containing the
> > encrypt/decrypt password
> > -->
> >
> > Anyway, I can encrypt just fine:
> >
> > $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> > enter aes-128-cbc encryption password:
> > Verifying - enter aes-128-cbc encryption password:
> > U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >
> >
> > I can also decrypt just fine from the command line.
> >
> > However, if I use the encrypted password and encryptKeyFile in the config
> > file, I end up with an error: "String length must be a multiple of four."
> >
> > https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
> >
> > How do I get this working?
> >
> > Thanks,
> > Jamie
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Jamie Jackson
In reply to this post by Alexandre Rafalovitch
It happens to be ten characters.

On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <[hidden email]>
wrote:

> How long is the encryption key (file content)? Because the code I am
> looking at seems to expect it to be at most 100 characters.
>
> Regards,
>    Alex.
> ----
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]> wrote:
> > I haven't tried this but is it possible there is a new line at the end in
> > the file?
> >
> > If you did something like echo "" > file.txt then there would be a new
> > line. Use echo -n "" > file.txt
> >
> > Also you should be able to check how many characters are in the file.
> >
> > Kevin Risden
> >
> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>
> wrote:
> >
> >> Hi Folks,
> >>
> >> (Using Solr 5.5.3.)
> >>
> >> As far as I know, the only place where encrypted password use is
> documented
> >> is in
> >> https://cwiki.apache.org/confluence/display/solr/
> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> >> under the "Configuring the DIH Configuration File", in a comment in the
> >> sample XML file:
> >>
> >> <!--
> >> Alternately the password can be encrypted as follows. This is the value
> >> obtained as a result of the command
> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt
> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
> >> WHen the password is encrypted, you must provide an extra attribute
> >> encryptKeyFile="/location/of/encryptionkey"
> >> This file should a text file with a single line containing the
> >> encrypt/decrypt password
> >> -->
> >>
> >> Anyway, I can encrypt just fine:
> >>
> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> >> enter aes-128-cbc encryption password:
> >> Verifying - enter aes-128-cbc encryption password:
> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >>
> >>
> >> I can also decrypt just fine from the command line.
> >>
> >> However, if I use the encrypted password and encryptKeyFile in the
> config
> >> file, I end up with an error: "String length must be a multiple of
> four."
> >>
> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
> >>
> >> How do I get this working?
> >>
> >> Thanks,
> >> Jamie
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Jamie Jackson
I'm at a brick wall. Here's the latest status:

Here are some sample commands that I'm using:

*Create the encryptKeyFile and encrypted password:*


encrypter_password='this_is_my_encrypter_password'
plain_db_pw='Oakton153'

cd /var/docker/solr_stage2/credentials/
echo -n "${encrypter_password}" > encpwd.txt
echo -n "${plain_db_pwd}" > plaindbpwd.txt
openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
"${encrypter_password}"

rm plaindbpwd.txt

That generated this as the password, by the way:

U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=

*Configure DIH configuration:*

<dataConfig>

<dataSource
driver="org.mariadb.jdbc.Driver"
url="jdbc:mysql://local.mysite.com:3306/mysite"
user="root"
password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
encryptKeyFile="/opt/solr/credentials/encpwd.txt"
/>
...


By the way, /var/docker/solr_stage2/credentials/ is mapped to
/opt/solr/credentials/ in the docker container, so that's why the paths
*seem* different (but aren't, really).


*Authentication error when data import is run:*

Exception while processing: question document :
SolrInputDocument(fields:
[]):org.apache.solr.handler.dataimport.DataImportHandlerException:
Unable to execute query:     select     'foo' as bar;    Processing
Document # 1
        at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:323)
        at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283)
        at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52)
        at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
        at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
        at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
        at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
        at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
        at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
        at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
connect: Access denied for user 'root'@'ICZ2002912' (using password:
NO)
        at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
        at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:71)
        at org.mariadb.jdbc.Driver.connect(Driver.java:109)
        at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:192)
        at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
        at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:503)
        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:313)
        ... 12 more
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
not connect: Access denied for user 'root'@'ICZ2002912' (using
password: NO)
        at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:524)
        at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:472)
        at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:374)
        at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:763)
        at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)
        at org.mariadb.jdbc.Driver.connect(Driver.java:104)
        ... 16 more



On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <[hidden email]> wrote:

> It happens to be ten characters.
>
> On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <[hidden email]
> > wrote:
>
>> How long is the encryption key (file content)? Because the code I am
>> looking at seems to expect it to be at most 100 characters.
>>
>> Regards,
>>    Alex.
>> ----
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]>
>> wrote:
>> > I haven't tried this but is it possible there is a new line at the end
>> in
>> > the file?
>> >
>> > If you did something like echo "" > file.txt then there would be a new
>> > line. Use echo -n "" > file.txt
>> >
>> > Also you should be able to check how many characters are in the file.
>> >
>> > Kevin Risden
>> >
>> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>
>> wrote:
>> >
>> >> Hi Folks,
>> >>
>> >> (Using Solr 5.5.3.)
>> >>
>> >> As far as I know, the only place where encrypted password use is
>> documented
>> >> is in
>> >> https://cwiki.apache.org/confluence/display/solr/
>> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
>> >> under the "Configuring the DIH Configuration File", in a comment in the
>> >> sample XML file:
>> >>
>> >> <!--
>> >> Alternately the password can be encrypted as follows. This is the value
>> >> obtained as a result of the command
>> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt
>> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
>> >> WHen the password is encrypted, you must provide an extra attribute
>> >> encryptKeyFile="/location/of/encryptionkey"
>> >> This file should a text file with a single line containing the
>> >> encrypt/decrypt password
>> >> -->
>> >>
>> >> Anyway, I can encrypt just fine:
>> >>
>> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
>> >> enter aes-128-cbc encryption password:
>> >> Verifying - enter aes-128-cbc encryption password:
>> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>> >>
>> >>
>> >> I can also decrypt just fine from the command line.
>> >>
>> >> However, if I use the encrypted password and encryptKeyFile in the
>> config
>> >> file, I end up with an error: "String length must be a multiple of
>> four."
>> >>
>> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
>> >>
>> >> How do I get this working?
>> >>
>> >> Thanks,
>> >> Jamie
>> >>
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Fuad Efendi
This is the root of the problem:
"Access denied for user 'root'@'ICZ2002912' (using password: NO) “


First of all, ensure that plain (non-encrypted) password settings work for you.

Check that you can connect using MySQL client from ICZ2002912 to your MySQL & Co. instance

I suspect you need to allow MySQL & Co. to accept connections from ICZ2002912. Plus, check DNS resolution, etc. 


Thanks,


--
Fuad Efendi
(416) 993-2060
http://www.tokenizer.ca
Recommender Systems


On November 2, 2016 at 2:37:08 PM, Jamie Jackson ([hidden email]) wrote:

I'm at a brick wall. Here's the latest status:  

Here are some sample commands that I'm using:  

*Create the encryptKeyFile and encrypted password:*  


encrypter_password='this_is_my_encrypter_password'  
plain_db_pw='Oakton153'  

cd /var/docker/solr_stage2/credentials/  
echo -n "${encrypter_password}" > encpwd.txt  
echo -n "${plain_db_pwd}" > plaindbpwd.txt  
openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k  
"${encrypter_password}"  

rm plaindbpwd.txt  

That generated this as the password, by the way:  

U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=  

*Configure DIH configuration:*  

<dataConfig>  

<dataSource  
driver="org.mariadb.jdbc.Driver"  
url="jdbc:mysql://local.mysite.com:3306/mysite"  
user="root"  
password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="  
encryptKeyFile="/opt/solr/credentials/encpwd.txt"  
/>  
...  


By the way, /var/docker/solr_stage2/credentials/ is mapped to  
/opt/solr/credentials/ in the docker container, so that's why the paths  
*seem* different (but aren't, really).  


*Authentication error when data import is run:*  

Exception while processing: question document :  
SolrInputDocument(fields:  
[]):org.apache.solr.handler.dataimport.DataImportHandlerException:  
Unable to execute query: select 'foo' as bar; Processing  
Document # 1  
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)  
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:323)  
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283)  
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52)  
at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)  
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)  
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)  
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)  
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)  
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)  
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)  
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)  
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)  
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)  
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not  
connect: Access denied for user 'root'@'ICZ2002912' (using password:  
NO)  
at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)  
at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:71)  
at org.mariadb.jdbc.Driver.connect(Driver.java:109)  
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:192)  
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)  
at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:503)  
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:313)  
... 12 more  
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could  
not connect: Access denied for user 'root'@'ICZ2002912' (using  
password: NO)  
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:524)  
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:472)  
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:374)  
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:763)  
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)  
at org.mariadb.jdbc.Driver.connect(Driver.java:104)  
... 16 more  



On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <[hidden email]> wrote:  

> It happens to be ten characters.  
>  
> On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <[hidden email]  
> > wrote:  
>  
>> How long is the encryption key (file content)? Because the code I am  
>> looking at seems to expect it to be at most 100 characters.  
>>  
>> Regards,  
>> Alex.  
>> ----  
>> Newsletter and resources for Solr beginners and intermediates:  
>> http://www.solr-start.com/ 
>>  
>>  
>> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]>  
>> wrote:  
>> > I haven't tried this but is it possible there is a new line at the end  
>> in  
>> > the file?  
>> >  
>> > If you did something like echo "" > file.txt then there would be a new  
>> > line. Use echo -n "" > file.txt  
>> >  
>> > Also you should be able to check how many characters are in the file.  
>> >  
>> > Kevin Risden  
>> >  
>> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>  
>> wrote:  
>> >  
>> >> Hi Folks,  
>> >>  
>> >> (Using Solr 5.5.3.)  
>> >>  
>> >> As far as I know, the only place where encrypted password use is  
>> documented  
>> >> is in  
>> >> https://cwiki.apache.org/confluence/display/solr/ 
>> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,  
>> >> under the "Configuring the DIH Configuration File", in a comment in the  
>> >> sample XML file:  
>> >>  
>> >> <!--  
>> >> Alternately the password can be encrypted as follows. This is the value  
>> >> obtained as a result of the command  
>> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt  
>> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="  
>> >> WHen the password is encrypted, you must provide an extra attribute  
>> >> encryptKeyFile="/location/of/encryptionkey"  
>> >> This file should a text file with a single line containing the  
>> >> encrypt/decrypt password  
>> >> -->  
>> >>  
>> >> Anyway, I can encrypt just fine:  
>> >>  
>> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt  
>> >> enter aes-128-cbc encryption password:  
>> >> Verifying - enter aes-128-cbc encryption password:  
>> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=  
>> >>  
>> >>  
>> >> I can also decrypt just fine from the command line.  
>> >>  
>> >> However, if I use the encrypted password and encryptKeyFile in the  
>> config  
>> >> file, I end up with an error: "String length must be a multiple of  
>> four."  
>> >>  
>> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9 
>> >>  
>> >> How do I get this working?  
>> >>  
>> >> Thanks,  
>> >> Jamie  
>> >>  
>>  
>  
>  
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Fuad Efendi
In MySQL, this command will explicitly allow to connect from remote ICZ2002912 host, check MySQL documentation:

GRANT ALL ON mysite.* TO 'root’@'ICZ2002912' IDENTIFIED BY ‘Oakton123’;



On November 2, 2016 at 4:41:48 PM, Fuad Efendi ([hidden email]) wrote:

This is the root of the problem:
"Access denied for user 'root'@'ICZ2002912' (using password: NO) “


First of all, ensure that plain (non-encrypted) password settings work for you.

Check that you can connect using MySQL client from ICZ2002912 to your MySQL & Co. instance

I suspect you need to allow MySQL & Co. to accept connections from ICZ2002912. Plus, check DNS resolution, etc. 


Thanks,


--
Fuad Efendi
(416) 993-2060
http://www.tokenizer.ca
Recommender Systems


On November 2, 2016 at 2:37:08 PM, Jamie Jackson ([hidden email]) wrote:

I'm at a brick wall. Here's the latest status:

Here are some sample commands that I'm using:

*Create the encryptKeyFile and encrypted password:*


encrypter_password='this_is_my_encrypter_password'
plain_db_pw='Oakton153'

cd /var/docker/solr_stage2/credentials/
echo -n "${encrypter_password}" > encpwd.txt
echo -n "${plain_db_pwd}" > plaindbpwd.txt
openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
"${encrypter_password}"

rm plaindbpwd.txt

That generated this as the password, by the way:

U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=

*Configure DIH configuration:*

<dataConfig>

<dataSource
driver="org.mariadb.jdbc.Driver"
url="jdbc:mysql://local.mysite.com:3306/mysite"
user="root"
password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
encryptKeyFile="/opt/solr/credentials/encpwd.txt"
/>
...


By the way, /var/docker/solr_stage2/credentials/ is mapped to
/opt/solr/credentials/ in the docker container, so that's why the paths
*seem* different (but aren't, really).


*Authentication error when data import is run:*

Exception while processing: question document :
SolrInputDocument(fields:
[]):org.apache.solr.handler.dataimport.DataImportHandlerException:
Unable to execute query: select 'foo' as bar; Processing
Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:323)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283)
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
connect: Access denied for user 'root'@'ICZ2002912' (using password:
NO)
at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:71)
at org.mariadb.jdbc.Driver.connect(Driver.java:109)
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:192)
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:503)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:313)
... 12 more
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
not connect: Access denied for user 'root'@'ICZ2002912' (using
password: NO)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:524)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:472)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:374)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:763)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)
at org.mariadb.jdbc.Driver.connect(Driver.java:104)
... 16 more



On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <[hidden email]> wrote:

> It happens to be ten characters.
>
> On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <[hidden email]
> > wrote:
>
>> How long is the encryption key (file content)? Because the code I am
>> looking at seems to expect it to be at most 100 characters.
>>
>> Regards,
>> Alex.
>> ----
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]>
>> wrote:
>> > I haven't tried this but is it possible there is a new line at the end
>> in
>> > the file?
>> >
>> > If you did something like echo "" > file.txt then there would be a new
>> > line. Use echo -n "" > file.txt
>> >
>> > Also you should be able to check how many characters are in the file.
>> >
>> > Kevin Risden
>> >
>> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>
>> wrote:
>> >
>> >> Hi Folks,
>> >>
>> >> (Using Solr 5.5.3.)
>> >>
>> >> As far as I know, the only place where encrypted password use is
>> documented
>> >> is in
>> >> https://cwiki.apache.org/confluence/display/solr/
>> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
>> >> under the "Configuring the DIH Configuration File", in a comment in the
>> >> sample XML file:
>> >>
>> >> <!--
>> >> Alternately the password can be encrypted as follows. This is the value
>> >> obtained as a result of the command
>> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt
>> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
>> >> WHen the password is encrypted, you must provide an extra attribute
>> >> encryptKeyFile="/location/of/encryptionkey"
>> >> This file should a text file with a single line containing the
>> >> encrypt/decrypt password
>> >> -->
>> >>
>> >> Anyway, I can encrypt just fine:
>> >>
>> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
>> >> enter aes-128-cbc encryption password:
>> >> Verifying - enter aes-128-cbc encryption password:
>> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>> >>
>> >>
>> >> I can also decrypt just fine from the command line.
>> >>
>> >> However, if I use the encrypted password and encryptKeyFile in the
>> config
>> >> file, I end up with an error: "String length must be a multiple of
>> four."
>> >>
>> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4d9
>> >>
>> >> How do I get this working?
>> >>
>> >> Thanks,
>> >> Jamie
>> >>
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Jamie Jackson
I should have mentioned that I verified connectivity with plain passwords:

From the same machine that Solr's running on:

solr@000650cbdd5e:/opt/solr$ mysql -uroot -pOakton153 -h local.mysite.com
mysite -e "select 'foo' as bar;"
+-----+
| bar |
+-----+
| foo |
+-----+

Also, if I add the plain-text password to the config, it connects fine:

<dataSource
  driver="org.mariadb.jdbc.Driver"
  url="jdbc:mysql://local.mysite.com:3306/mysite"
  user="root"
  password="Oakton153"
/>


So that is why I claim to have a problem with encryptKeyFile, specifically,
because I've eliminated general connectivity/authentication problems.

Thanks,
Jamie

On Wed, Nov 2, 2016 at 4:58 PM, Fuad Efendi <[hidden email]> wrote:

> In MySQL, this command will explicitly allow to connect from
> remote ICZ2002912 host, check MySQL documentation:
>
> GRANT ALL ON mysite.* TO 'root’@'ICZ2002912' IDENTIFIED BY ‘Oakton123’;
>
>
>
> On November 2, 2016 at 4:41:48 PM, Fuad Efendi ([hidden email]) wrote:
>
> This is the root of the problem:
> "Access denied for user 'root'@'ICZ2002912' (using password: NO) “
>
>
> First of all, ensure that plain (non-encrypted) password settings work for
> you.
>
> Check that you can connect using MySQL client from ICZ2002912 to your
> MySQL & Co. instance
>
> I suspect you need to allow MySQL & Co. to accept connections
> from ICZ2002912. Plus, check DNS resolution, etc.
>
>
> Thanks,
>
>
> --
> Fuad Efendi
> (416) 993-2060
> http://www.tokenizer.ca
> Recommender Systems
>
>
> On November 2, 2016 at 2:37:08 PM, Jamie Jackson ([hidden email])
> wrote:
>
> I'm at a brick wall. Here's the latest status:
>
> Here are some sample commands that I'm using:
>
> *Create the encryptKeyFile and encrypted password:*
>
>
> encrypter_password='this_is_my_encrypter_password'
> plain_db_pw='Oakton153'
>
> cd /var/docker/solr_stage2/credentials/
> echo -n "${encrypter_password}" > encpwd.txt
> echo -n "${plain_db_pwd}" > plaindbpwd.txt
> openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
> "${encrypter_password}"
>
> rm plaindbpwd.txt
>
> That generated this as the password, by the way:
>
> U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
>
> *Configure DIH configuration:*
>
> <dataConfig>
>
> <dataSource
> driver="org.mariadb.jdbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> encryptKeyFile="/opt/solr/credentials/encpwd.txt"
> />
> ...
>
>
> By the way, /var/docker/solr_stage2/credentials/ is mapped to
> /opt/solr/credentials/ in the docker container, so that's why the paths
> *seem* different (but aren't, really).
>
>
> *Authentication error when data import is run:*
>
> Exception while processing: question document :
> SolrInputDocument(fields:
> []):org.apache.solr.handler.dataimport.DataImportHandlerException:
> Unable to execute query: select 'foo' as bar; Processing
> Document # 1
> at org.apache.solr.handler.dataimport.DataImportHandlerException.
> wrapAndThrow(DataImportHandlerException.java:69)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.<init>(JdbcDataSource.java:323)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:283)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:52)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> initQuery(SqlEntityProcessor.java:59)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> nextRow(SqlEntityProcessor.java:73)
> at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(
> EntityProcessorWrapper.java:244)
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:475)
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:414)
> at org.apache.solr.handler.dataimport.DocBuilder.
> doFullDump(DocBuilder.java:329)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:232)
> at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:416)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:480)
> at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:461)
> Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
> connect: Access denied for user 'root'@'ICZ2002912' (using password:
> NO)
> at org.mariadb.jdbc.internal.util.ExceptionMapper.get(
> ExceptionMapper.java:123)
> at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(
> ExceptionMapper.java:71)
> at org.mariadb.jdbc.Driver.connect(Driver.java:109)
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> call(JdbcDataSource.java:192)
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> call(JdbcDataSource.java:172)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> getConnection(JdbcDataSource.java:503)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.<init>(JdbcDataSource.java:313)
> ... 12 more
> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
> not connect: Access denied for user 'root'@'ICZ2002912' (using
> password: NO)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> authentication(AbstractConnectProtocol.java:524)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> handleConnectionPhases(AbstractConnectProtocol.java:472)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(
> AbstractConnectProtocol.java:374)
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> connectWithoutProxy(AbstractConnectProtocol.java:763)
> at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)
> at org.mariadb.jdbc.Driver.connect(Driver.java:104)
> ... 16 more
>
>
>
> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <[hidden email]>
> wrote:
>
> > It happens to be ten characters.
> >
> > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <
> [hidden email]
> > > wrote:
> >
> >> How long is the encryption key (file content)? Because the code I am
> >> looking at seems to expect it to be at most 100 characters.
> >>
> >> Regards,
> >> Alex.
> >> ----
> >> Newsletter and resources for Solr beginners and intermediates:
> >> http://www.solr-start.com/
> >>
> >>
> >> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]>
> >> wrote:
> >> > I haven't tried this but is it possible there is a new line at the end
> >> in
> >> > the file?
> >> >
> >> > If you did something like echo "" > file.txt then there would be a new
> >> > line. Use echo -n "" > file.txt
> >> >
> >> > Also you should be able to check how many characters are in the file.
> >> >
> >> > Kevin Risden
> >> >
> >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>
> >> wrote:
> >> >
> >> >> Hi Folks,
> >> >>
> >> >> (Using Solr 5.5.3.)
> >> >>
> >> >> As far as I know, the only place where encrypted password use is
> >> documented
> >> >> is in
> >> >> https://cwiki.apache.org/confluence/display/solr/
> >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> >> >> under the "Configuring the DIH Configuration File", in a comment in
> the
> >> >> sample XML file:
> >> >>
> >> >> <!--
> >> >> Alternately the password can be encrypted as follows. This is the
> value
> >> >> obtained as a result of the command
> >> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt
> >> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
> >> >> WHen the password is encrypted, you must provide an extra attribute
> >> >> encryptKeyFile="/location/of/encryptionkey"
> >> >> This file should a text file with a single line containing the
> >> >> encrypt/decrypt password
> >> >> -->
> >> >>
> >> >> Anyway, I can encrypt just fine:
> >> >>
> >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> >> >> enter aes-128-cbc encryption password:
> >> >> Verifying - enter aes-128-cbc encryption password:
> >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >> >>
> >> >>
> >> >> I can also decrypt just fine from the command line.
> >> >>
> >> >> However, if I use the encrypted password and encryptKeyFile in the
> >> config
> >> >> file, I end up with an error: "String length must be a multiple of
> >> four."
> >> >>
> >> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4
> d9
> >> >>
> >> >> How do I get this working?
> >> >>
> >> >> Thanks,
> >> >> Jamie
> >> >>
> >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Fuad Efendi
Then I can only guess that in current configuration decrypted password is empty string.

Try to manually replace some characters in encpwd.txt file to see if you get different errors; try to delete this file completely to see if you get different errors. Try to add new line in this file; try to change password in config file.



On November 2, 2016 at 5:23:33 PM, Jamie Jackson ([hidden email]) wrote:

I should have mentioned that I verified connectivity with plain passwords:  

From the same machine that Solr's running on:  

solr@000650cbdd5e:/opt/solr$ mysql -uroot -pOakton153 -h local.mysite.com  
mysite -e "select 'foo' as bar;"  
+-----+  
| bar |  
+-----+  
| foo |  
+-----+  

Also, if I add the plain-text password to the config, it connects fine:  

<dataSource  
driver="org.mariadb.jdbc.Driver"  
url="jdbc:mysql://local.mysite.com:3306/mysite"  
user="root"  
password="Oakton153"  
/>  


So that is why I claim to have a problem with encryptKeyFile, specifically,  
because I've eliminated general connectivity/authentication problems.  

Thanks,  
Jamie  

On Wed, Nov 2, 2016 at 4:58 PM, Fuad Efendi <[hidden email]> wrote:  

> In MySQL, this command will explicitly allow to connect from  
> remote ICZ2002912 host, check MySQL documentation:  
>  
> GRANT ALL ON mysite.* TO 'root’@'ICZ2002912' IDENTIFIED BY ‘Oakton123’;  
>  
>  
>  
> On November 2, 2016 at 4:41:48 PM, Fuad Efendi ([hidden email]) wrote:  
>  
> This is the root of the problem:  
> "Access denied for user 'root'@'ICZ2002912' (using password: NO) “  
>  
>  
> First of all, ensure that plain (non-encrypted) password settings work for  
> you.  
>  
> Check that you can connect using MySQL client from ICZ2002912 to your  
> MySQL & Co. instance  
>  
> I suspect you need to allow MySQL & Co. to accept connections  
> from ICZ2002912. Plus, check DNS resolution, etc.  
>  
>  
> Thanks,  
>  
>  
> --  
> Fuad Efendi  
> (416) 993-2060  
> http://www.tokenizer.ca 
> Recommender Systems  
>  
>  
> On November 2, 2016 at 2:37:08 PM, Jamie Jackson ([hidden email])  
> wrote:  
>  
> I'm at a brick wall. Here's the latest status:  
>  
> Here are some sample commands that I'm using:  
>  
> *Create the encryptKeyFile and encrypted password:*  
>  
>  
> encrypter_password='this_is_my_encrypter_password'  
> plain_db_pw='Oakton153'  
>  
> cd /var/docker/solr_stage2/credentials/  
> echo -n "${encrypter_password}" > encpwd.txt  
> echo -n "${plain_db_pwd}" > plaindbpwd.txt  
> openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k  
> "${encrypter_password}"  
>  
> rm plaindbpwd.txt  
>  
> That generated this as the password, by the way:  
>  
> U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=  
>  
> *Configure DIH configuration:*  
>  
> <dataConfig>  
>  
> <dataSource  
> driver="org.mariadb.jdbc.Driver"  
> url="jdbc:mysql://local.mysite.com:3306/mysite"  
> user="root"  
> password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="  
> encryptKeyFile="/opt/solr/credentials/encpwd.txt"  
> />  
> ...  
>  
>  
> By the way, /var/docker/solr_stage2/credentials/ is mapped to  
> /opt/solr/credentials/ in the docker container, so that's why the paths  
> *seem* different (but aren't, really).  
>  
>  
> *Authentication error when data import is run:*  
>  
> Exception while processing: question document :  
> SolrInputDocument(fields:  
> []):org.apache.solr.handler.dataimport.DataImportHandlerException:  
> Unable to execute query: select 'foo' as bar; Processing  
> Document # 1  
> at org.apache.solr.handler.dataimport.DataImportHandlerException.  
> wrapAndThrow(DataImportHandlerException.java:69)  
> at org.apache.solr.handler.dataimport.JdbcDataSource$  
> ResultSetIterator.<init>(JdbcDataSource.java:323)  
> at org.apache.solr.handler.dataimport.JdbcDataSource.  
> getData(JdbcDataSource.java:283)  
> at org.apache.solr.handler.dataimport.JdbcDataSource.  
> getData(JdbcDataSource.java:52)  
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.  
> initQuery(SqlEntityProcessor.java:59)  
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.  
> nextRow(SqlEntityProcessor.java:73)  
> at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(  
> EntityProcessorWrapper.java:244)  
> at org.apache.solr.handler.dataimport.DocBuilder.  
> buildDocument(DocBuilder.java:475)  
> at org.apache.solr.handler.dataimport.DocBuilder.  
> buildDocument(DocBuilder.java:414)  
> at org.apache.solr.handler.dataimport.DocBuilder.  
> doFullDump(DocBuilder.java:329)  
> at org.apache.solr.handler.dataimport.DocBuilder.execute(  
> DocBuilder.java:232)  
> at org.apache.solr.handler.dataimport.DataImporter.  
> doFullImport(DataImporter.java:416)  
> at org.apache.solr.handler.dataimport.DataImporter.  
> runCmd(DataImporter.java:480)  
> at org.apache.solr.handler.dataimport.DataImporter$1.run(  
> DataImporter.java:461)  
> Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not  
> connect: Access denied for user 'root'@'ICZ2002912' (using password:  
> NO)  
> at org.mariadb.jdbc.internal.util.ExceptionMapper.get(  
> ExceptionMapper.java:123)  
> at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(  
> ExceptionMapper.java:71)  
> at org.mariadb.jdbc.Driver.connect(Driver.java:109)  
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.  
> call(JdbcDataSource.java:192)  
> at org.apache.solr.handler.dataimport.JdbcDataSource$1.  
> call(JdbcDataSource.java:172)  
> at org.apache.solr.handler.dataimport.JdbcDataSource.  
> getConnection(JdbcDataSource.java:503)  
> at org.apache.solr.handler.dataimport.JdbcDataSource$  
> ResultSetIterator.<init>(JdbcDataSource.java:313)  
> ... 12 more  
> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could  
> not connect: Access denied for user 'root'@'ICZ2002912' (using  
> password: NO)  
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.  
> authentication(AbstractConnectProtocol.java:524)  
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.  
> handleConnectionPhases(AbstractConnectProtocol.java:472)  
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(  
> AbstractConnectProtocol.java:374)  
> at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.  
> connectWithoutProxy(AbstractConnectProtocol.java:763)  
> at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)  
> at org.mariadb.jdbc.Driver.connect(Driver.java:104)  
> ... 16 more  
>  
>  
>  
> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <[hidden email]>  
> wrote:  
>  
> > It happens to be ten characters.  
> >  
> > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <  
> [hidden email]  
> > > wrote:  
> >  
> >> How long is the encryption key (file content)? Because the code I am  
> >> looking at seems to expect it to be at most 100 characters.  
> >>  
> >> Regards,  
> >> Alex.  
> >> ----  
> >> Newsletter and resources for Solr beginners and intermediates:  
> >> http://www.solr-start.com/ 
> >>  
> >>  
> >> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]>  
> >> wrote:  
> >> > I haven't tried this but is it possible there is a new line at the end  
> >> in  
> >> > the file?  
> >> >  
> >> > If you did something like echo "" > file.txt then there would be a new  
> >> > line. Use echo -n "" > file.txt  
> >> >  
> >> > Also you should be able to check how many characters are in the file.  
> >> >  
> >> > Kevin Risden  
> >> >  
> >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>  
> >> wrote:  
> >> >  
> >> >> Hi Folks,  
> >> >>  
> >> >> (Using Solr 5.5.3.)  
> >> >>  
> >> >> As far as I know, the only place where encrypted password use is  
> >> documented  
> >> >> is in  
> >> >> https://cwiki.apache.org/confluence/display/solr/ 
> >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,  
> >> >> under the "Configuring the DIH Configuration File", in a comment in  
> the  
> >> >> sample XML file:  
> >> >>  
> >> >> <!--  
> >> >> Alternately the password can be encrypted as follows. This is the  
> value  
> >> >> obtained as a result of the command  
> >> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt  
> >> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="  
> >> >> WHen the password is encrypted, you must provide an extra attribute  
> >> >> encryptKeyFile="/location/of/encryptionkey"  
> >> >> This file should a text file with a single line containing the  
> >> >> encrypt/decrypt password  
> >> >> -->  
> >> >>  
> >> >> Anyway, I can encrypt just fine:  
> >> >>  
> >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt  
> >> >> enter aes-128-cbc encryption password:  
> >> >> Verifying - enter aes-128-cbc encryption password:  
> >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=  
> >> >>  
> >> >>  
> >> >> I can also decrypt just fine from the command line.  
> >> >>  
> >> >> However, if I use the encrypted password and encryptKeyFile in the  
> >> config  
> >> >> file, I end up with an error: "String length must be a multiple of  
> >> four."  
> >> >>  
> >> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4 
> d9  
> >> >>  
> >> >> How do I get this working?  
> >> >>  
> >> >> Thanks,  
> >> >> Jamie  
> >> >>  
> >>  
> >  
> >  
>  
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Jamie Jackson
You were right, Fuad. There was a flaw in my script (inconsistent naming of
the `plain_db_pwd` variable.

Thanks for figuring that out.

For posterity, here's the fixed script:

################################################################################
encrypt_key=your_encryption_key
plain_db_pwd=your_db_password
cred_dir=/your/credentials/directory

cd "${cred_dir}
echo -n "${encrypt_key}" > encrypt.key
echo -n "${plain_db_pwd}" | openssl enc -aes-128-cbc -a -salt -k
"${encrypt_key}"
#==============================================================================#

Then, in the DIH config:
 encryptKeyFile="/your/credentials/directory/encrypt.key"

I have another, semi-related, issue that I'll bring up in another thread.

Thanks,
Jamie


On Wed, Nov 2, 2016 at 6:26 PM, Fuad Efendi <[hidden email]> wrote:

> Then I can only guess that in current configuration decrypted password is
> empty string.
>
> Try to manually replace some characters in encpwd.txt file to see if you
> get different errors; try to delete this file completely to see if you get
> different errors. Try to add new line in this file; try to change password
> in config file.
>
>
>
> On November 2, 2016 at 5:23:33 PM, Jamie Jackson ([hidden email])
> wrote:
>
> I should have mentioned that I verified connectivity with plain passwords:
>
> From the same machine that Solr's running on:
>
> solr@000650cbdd5e:/opt/solr$ mysql -uroot -pOakton153 -h local.mysite.com
> mysite -e "select 'foo' as bar;"
> +-----+
> | bar |
> +-----+
> | foo |
> +-----+
>
> Also, if I add the plain-text password to the config, it connects fine:
>
> <dataSource
> driver="org.mariadb.jdbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="Oakton153"
> />
>
>
> So that is why I claim to have a problem with encryptKeyFile, specifically,
> because I've eliminated general connectivity/authentication problems.
>
> Thanks,
> Jamie
>
> On Wed, Nov 2, 2016 at 4:58 PM, Fuad Efendi <[hidden email]> wrote:
>
> > In MySQL, this command will explicitly allow to connect from
> > remote ICZ2002912 host, check MySQL documentation:
> >
> > GRANT ALL ON mysite.* TO 'root’@'ICZ2002912' IDENTIFIED BY ‘Oakton123’;
> >
> >
> >
> > On November 2, 2016 at 4:41:48 PM, Fuad Efendi ([hidden email]) wrote:
> >
> > This is the root of the problem:
> > "Access denied for user 'root'@'ICZ2002912' (using password: NO) “
> >
> >
> > First of all, ensure that plain (non-encrypted) password settings work
> for
> > you.
> >
> > Check that you can connect using MySQL client from ICZ2002912 to your
> > MySQL & Co. instance
> >
> > I suspect you need to allow MySQL & Co. to accept connections
> > from ICZ2002912. Plus, check DNS resolution, etc.
> >
> >
> > Thanks,
> >
> >
> > --
> > Fuad Efendi
> > (416) 993-2060
> > http://www.tokenizer.ca
> > Recommender Systems
> >
> >
> > On November 2, 2016 at 2:37:08 PM, Jamie Jackson ([hidden email])
> > wrote:
> >
> > I'm at a brick wall. Here's the latest status:
> >
> > Here are some sample commands that I'm using:
> >
> > *Create the encryptKeyFile and encrypted password:*
> >
> >
> > encrypter_password='this_is_my_encrypter_password'
> > plain_db_pw='Oakton153'
> >
> > cd /var/docker/solr_stage2/credentials/
> > echo -n "${encrypter_password}" > encpwd.txt
> > echo -n "${plain_db_pwd}" > plaindbpwd.txt
> > openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
> > "${encrypter_password}"
> >
> > rm plaindbpwd.txt
> >
> > That generated this as the password, by the way:
> >
> > U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
> >
> > *Configure DIH configuration:*
> >
> > <dataConfig>
> >
> > <dataSource
> > driver="org.mariadb.jdbc.Driver"
> > url="jdbc:mysql://local.mysite.com:3306/mysite"
> > user="root"
> > password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> > encryptKeyFile="/opt/solr/credentials/encpwd.txt"
> > />
> > ...
> >
> >
> > By the way, /var/docker/solr_stage2/credentials/ is mapped to
> > /opt/solr/credentials/ in the docker container, so that's why the paths
> > *seem* different (but aren't, really).
> >
> >
> > *Authentication error when data import is run:*
> >
> > Exception while processing: question document :
> > SolrInputDocument(fields:
> > []):org.apache.solr.handler.dataimport.DataImportHandlerException:
> > Unable to execute query: select 'foo' as bar; Processing
> > Document # 1
> > at org.apache.solr.handler.dataimport.DataImportHandlerException.
> > wrapAndThrow(DataImportHandlerException.java:69)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > ResultSetIterator.<init>(JdbcDataSource.java:323)
> > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > getData(JdbcDataSource.java:283)
> > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > getData(JdbcDataSource.java:52)
> > at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> > initQuery(SqlEntityProcessor.java:59)
> > at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> > nextRow(SqlEntityProcessor.java:73)
> > at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(
> > EntityProcessorWrapper.java:244)
> > at org.apache.solr.handler.dataimport.DocBuilder.
> > buildDocument(DocBuilder.java:475)
> > at org.apache.solr.handler.dataimport.DocBuilder.
> > buildDocument(DocBuilder.java:414)
> > at org.apache.solr.handler.dataimport.DocBuilder.
> > doFullDump(DocBuilder.java:329)
> > at org.apache.solr.handler.dataimport.DocBuilder.execute(
> > DocBuilder.java:232)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > doFullImport(DataImporter.java:416)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > runCmd(DataImporter.java:480)
> > at org.apache.solr.handler.dataimport.DataImporter$1.run(
> > DataImporter.java:461)
> > Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
> > connect: Access denied for user 'root'@'ICZ2002912' (using password:
> > NO)
> > at org.mariadb.jdbc.internal.util.ExceptionMapper.get(
> > ExceptionMapper.java:123)
> > at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(
> > ExceptionMapper.java:71)
> > at org.mariadb.jdbc.Driver.connect(Driver.java:109)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> > call(JdbcDataSource.java:192)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> > call(JdbcDataSource.java:172)
> > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > getConnection(JdbcDataSource.java:503)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > ResultSetIterator.<init>(JdbcDataSource.java:313)
> > ... 12 more
> > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
> > not connect: Access denied for user 'root'@'ICZ2002912' (using
> > password: NO)
> > at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> > authentication(AbstractConnectProtocol.java:524)
> > at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> > handleConnectionPhases(AbstractConnectProtocol.java:472)
> > at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(
> > AbstractConnectProtocol.java:374)
> > at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> > connectWithoutProxy(AbstractConnectProtocol.java:763)
> > at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:469)
> > at org.mariadb.jdbc.Driver.connect(Driver.java:104)
> > ... 16 more
> >
> >
> >
> > On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <[hidden email]>
> > wrote:
> >
> > > It happens to be ten characters.
> > >
> > > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <
> > [hidden email]
> > > > wrote:
> > >
> > >> How long is the encryption key (file content)? Because the code I am
> > >> looking at seems to expect it to be at most 100 characters.
> > >>
> > >> Regards,
> > >> Alex.
> > >> ----
> > >> Newsletter and resources for Solr beginners and intermediates:
> > >> http://www.solr-start.com/
> > >>
> > >>
> > >> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]>
> > >> wrote:
> > >> > I haven't tried this but is it possible there is a new line at the
> end
> > >> in
> > >> > the file?
> > >> >
> > >> > If you did something like echo "" > file.txt then there would be a
> new
> > >> > line. Use echo -n "" > file.txt
> > >> >
> > >> > Also you should be able to check how many characters are in the
> file.
> > >> >
> > >> > Kevin Risden
> > >> >
> > >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]
> >
> > >> wrote:
> > >> >
> > >> >> Hi Folks,
> > >> >>
> > >> >> (Using Solr 5.5.3.)
> > >> >>
> > >> >> As far as I know, the only place where encrypted password use is
> > >> documented
> > >> >> is in
> > >> >> https://cwiki.apache.org/confluence/display/solr/
> > >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> > >> >> under the "Configuring the DIH Configuration File", in a comment in
> > the
> > >> >> sample XML file:
> > >> >>
> > >> >> <!--
> > >> >> Alternately the password can be encrypted as follows. This is the
> > value
> > >> >> obtained as a result of the command
> > >> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt
> > >> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
> > >> >> WHen the password is encrypted, you must provide an extra attribute
> > >> >> encryptKeyFile="/location/of/encryptionkey"
> > >> >> This file should a text file with a single line containing the
> > >> >> encrypt/decrypt password
> > >> >> -->
> > >> >>
> > >> >> Anyway, I can encrypt just fine:
> > >> >>
> > >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> > >> >> enter aes-128-cbc encryption password:
> > >> >> Verifying - enter aes-128-cbc encryption password:
> > >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> > >> >>
> > >> >>
> > >> >> I can also decrypt just fine from the command line.
> > >> >>
> > >> >> However, if I use the encrypted password and encryptKeyFile in the
> > >> config
> > >> >> file, I end up with an error: "String length must be a multiple of
> > >> four."
> > >> >>
> > >> >> https://gist.github.com/jamiejackson/
> 3852dacb03432328ea187d43ade5e4
> > d9
> > >> >>
> > >> >> How do I get this working?
> > >> >>
> > >> >> Thanks,
> > >> >> Jamie
> > >> >>
> > >>
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Billnbell
In reply to this post by Jamie Jackson
I cannot get it to work either.

Here are my steps. I took the key from the Patch in
https://issues.apache.org/jira/secure/attachment/12730862/SOLR-4392.patch.

echo
U2FsdGVkX19Gz7q7/4jj3Wsin7801TlFbob1PBT2YEacbPEUARDiuV5zGSAwU4Sz7upXDEPIQPU48oY1fBWM6Q==
> pass.enc

openssl aes-128-cbc -d -a -salt -in pass.enc

I typed: Password

enter aes-128-cbc decryption password:

SomeRandomEncryptedTextUsingAES128

I cannot find a test case in the latest v5.5.3 code.? It seems like openssl
command is wrong?

So it worked for that. Not sure if the code changed, but after doing this I
get in solr.log:


2016-11-03 12:06:20.139 INFO  (Thread-127) [   x:autosuggestfull]
o.a.s.u.p.LogUpdateProcessorFactory [autosuggestfull]  webapp=/solr
path=/dataimport
params={debug=false&optimize=false&indent=true&commit=false&clean=false&wt=json&command=full-import&entity=spec&verbose=false}
status=0 QTime=19{} 0 64

2016-11-03 12:06:20.140 ERROR (Thread-127) [   x:autosuggestfull]
o.a.s.h.d.DataImporter Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Error
decoding password Processing Document # 1

        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)

        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)

        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)

        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)

Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Error
decoding password Processing Document # 1

        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)

        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)

        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)

        ... 3 more

Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
Error decoding password Processing Document # 1

        at
org.apache.solr.handler.dataimport.JdbcDataSource.decryptPwd(JdbcDataSource.java:131)

        at
org.apache.solr.handler.dataimport.JdbcDataSource.init(JdbcDataSource.java:74)

        at
org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:389)

        at
org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:100)

        at
org.apache.solr.handler.dataimport.SqlEntityProcessor.init(SqlEntityProcessor.java:53)

        at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:75)

        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:433)

        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)

        ... 5 more

Caused by: java.lang.IllegalStateException: Bad password, algorithm, mode
or padding; no salt, wrong number of iterations or corrupted ciphertext.

        at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:249)

        at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:195)

        at
org.apache.solr.handler.dataimport.JdbcDataSource.decryptPwd(JdbcDataSource.java:129)

        ... 12 more

Caused by: javax.crypto.BadPaddingException: Given final block not properly
padded

        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:975)

        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:833)

        at
com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)

        at javax.crypto.Cipher.doFinal(Cipher.java:2165)

        at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:245)

        ... 14 more


2016-11-03 12:06:20.140 INFO  (Thread-127) [   x:autosuggestfull]
o.a.s.u.DirectUpdateHandler2 start rollback{}

2016-11-03 12:06:20.140 INFO  (Thread-127) [   x:autosuggestfull]
o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter...
core=autosuggestfull

2016-11-03 12:06:20.154 INFO  (Thread-127) [   x:autosuggestfull]
o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1



On Wed, Nov 2, 2016 at 12:21 PM, Jamie Jackson <[hidden email]> wrote:

> I'm at a brick wall. Here's the latest status:
>
> Here are some sample commands that I'm using:
>
> *Create the encryptKeyFile and encrypted password:*
>
>
> encrypter_password='this_is_my_encrypter_password'
> plain_db_pw='Oakton153'
>
> cd /var/docker/solr_stage2/credentials/
> echo -n "${encrypter_password}" > encpwd.txt
> echo -n "${plain_db_pwd}" > plaindbpwd.txt
> openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
> "${encrypter_password}"
>
> rm plaindbpwd.txt
>
> That generated this as the password, by the way:
>
> U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
>
> *Configure DIH configuration:*
>
> <dataConfig>
>
> <dataSource
> driver="org.mariadb.jdbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> encryptKeyFile="/opt/solr/credentials/encpwd.txt"
> />
> ...
>
>
> By the way, /var/docker/solr_stage2/credentials/ is mapped to
> /opt/solr/credentials/ in the docker container, so that's why the paths
> *seem* different (but aren't, really).
>
>
> *Authentication error when data import is run:*
>
> Exception while processing: question document :
> SolrInputDocument(fields:
> []):org.apache.solr.handler.dataimport.DataImportHandlerException:
> Unable to execute query:     select     'foo' as bar;    Processing
> Document # 1
>         at org.apache.solr.handler.dataimport.DataImportHandlerException.
> wrapAndThrow(DataImportHandlerException.java:69)
>         at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.<init>(JdbcDataSource.java:323)
>         at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:283)
>         at org.apache.solr.handler.dataimport.JdbcDataSource.
> getData(JdbcDataSource.java:52)
>         at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> initQuery(SqlEntityProcessor.java:59)
>         at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> nextRow(SqlEntityProcessor.java:73)
>         at org.apache.solr.handler.dataimport.EntityProcessorWrapper.
> nextRow(EntityProcessorWrapper.java:244)
>         at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:475)
>         at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:414)
>         at org.apache.solr.handler.dataimport.DocBuilder.
> doFullDump(DocBuilder.java:329)
>         at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:232)
>         at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:416)
>         at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:480)
>         at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:461)
> Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
> connect: Access denied for user 'root'@'ICZ2002912' (using password:
> NO)
>         at org.mariadb.jdbc.internal.util.ExceptionMapper.get(
> ExceptionMapper.java:123)
>         at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(
> ExceptionMapper.java:71)
>         at org.mariadb.jdbc.Driver.connect(Driver.java:109)
>         at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> call(JdbcDataSource.java:192)
>         at org.apache.solr.handler.dataimport.JdbcDataSource$1.
> call(JdbcDataSource.java:172)
>         at org.apache.solr.handler.dataimport.JdbcDataSource.
> getConnection(JdbcDataSource.java:503)
>         at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.<init>(JdbcDataSource.java:313)
>         ... 12 more
> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
> not connect: Access denied for user 'root'@'ICZ2002912' (using
> password: NO)
>         at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> authentication(AbstractConnectProtocol.java:524)
>         at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> handleConnectionPhases(AbstractConnectProtocol.java:472)
>         at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> connect(AbstractConnectProtocol.java:374)
>         at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.
> connectWithoutProxy(AbstractConnectProtocol.java:763)
>         at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(
> Utils.java:469)
>         at org.mariadb.jdbc.Driver.connect(Driver.java:104)
>         ... 16 more
>
>
>
> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <[hidden email]>
> wrote:
>
> > It happens to be ten characters.
> >
> > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <
> [hidden email]
> > > wrote:
> >
> >> How long is the encryption key (file content)? Because the code I am
> >> looking at seems to expect it to be at most 100 characters.
> >>
> >> Regards,
> >>    Alex.
> >> ----
> >> Newsletter and resources for Solr beginners and intermediates:
> >> http://www.solr-start.com/
> >>
> >>
> >> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]>
> >> wrote:
> >> > I haven't tried this but is it possible there is a new line at the end
> >> in
> >> > the file?
> >> >
> >> > If you did something like echo "" > file.txt then there would be a new
> >> > line. Use echo -n "" > file.txt
> >> >
> >> > Also you should be able to check how many characters are in the file.
> >> >
> >> > Kevin Risden
> >> >
> >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>
> >> wrote:
> >> >
> >> >> Hi Folks,
> >> >>
> >> >> (Using Solr 5.5.3.)
> >> >>
> >> >> As far as I know, the only place where encrypted password use is
> >> documented
> >> >> is in
> >> >> https://cwiki.apache.org/confluence/display/solr/
> >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
> >> >> under the "Configuring the DIH Configuration File", in a comment in
> the
> >> >> sample XML file:
> >> >>
> >> >> <!--
> >> >> Alternately the password can be encrypted as follows. This is the
> value
> >> >> obtained as a result of the command
> >> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt
> >> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
> >> >> WHen the password is encrypted, you must provide an extra attribute
> >> >> encryptKeyFile="/location/of/encryptionkey"
> >> >> This file should a text file with a single line containing the
> >> >> encrypt/decrypt password
> >> >> -->
> >> >>
> >> >> Anyway, I can encrypt just fine:
> >> >>
> >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
> >> >> enter aes-128-cbc encryption password:
> >> >> Verifying - enter aes-128-cbc encryption password:
> >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
> >> >>
> >> >>
> >> >> I can also decrypt just fine from the command line.
> >> >>
> >> >> However, if I use the encrypted password and encryptKeyFile in the
> >> config
> >> >> file, I end up with an error: "String length must be a multiple of
> >> four."
> >> >>
> >> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d43ade5e4
> d9
> >> >>
> >> >> How do I get this working?
> >> >>
> >> >> Thanks,
> >> >> Jamie
> >> >>
> >>
> >
> >
>



--
Bill Bell
[hidden email]
cell 720-256-8076
Reply | Threaded
Open this post in threaded view
|

Re: Problem with Password Decryption in Data Import Handler

Billnbell
OK it was

echo -n "${encrypt_key}" > encrypt.key



On Thu, Nov 3, 2016 at 12:20 PM, William Bell <[hidden email]> wrote:

> I cannot get it to work either.
>
> Here are my steps. I took the key from the Patch in
> https://issues.apache.org/jira/secure/attachment/12730862/SOLR-4392.patch.
>
> echo U2FsdGVkX19Gz7q7/4jj3Wsin7801TlFbob1PBT2YEacbPE
> UARDiuV5zGSAwU4Sz7upXDEPIQPU48oY1fBWM6Q== > pass.enc
>
> openssl aes-128-cbc -d -a -salt -in pass.enc
>
> I typed: Password
>
> enter aes-128-cbc decryption password:
>
> SomeRandomEncryptedTextUsingAES128
>
> I cannot find a test case in the latest v5.5.3 code.? It seems like
> openssl command is wrong?
>
> So it worked for that. Not sure if the code changed, but after doing this
> I get in solr.log:
>
>
> 2016-11-03 12:06:20.139 INFO  (Thread-127) [   x:autosuggestfull]
> o.a.s.u.p.LogUpdateProcessorFactory [autosuggestfull]  webapp=/solr
> path=/dataimport params={debug=false&optimize=false&indent=true&commit=
> false&clean=false&wt=json&command=full-import&entity=spec&verbose=false}
> status=0 QTime=19{} 0 64
>
> 2016-11-03 12:06:20.140 ERROR (Thread-127) [   x:autosuggestfull]
> o.a.s.h.d.DataImporter Full Import failed:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException:
> Error decoding password Processing Document # 1
>
>         at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:270)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:416)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:480)
>
>         at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:461)
>
> Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException:
> Error decoding password Processing Document # 1
>
>         at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:416)
>
>         at org.apache.solr.handler.dataimport.DocBuilder.
> doFullDump(DocBuilder.java:329)
>
>         at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:232)
>
>         ... 3 more
>
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> Error decoding password Processing Document # 1
>
>         at org.apache.solr.handler.dataimport.JdbcDataSource.
> decryptPwd(JdbcDataSource.java:131)
>
>         at org.apache.solr.handler.dataimport.JdbcDataSource.
> init(JdbcDataSource.java:74)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> getDataSourceInstance(DataImporter.java:389)
>
>         at org.apache.solr.handler.dataimport.ContextImpl.
> getDataSource(ContextImpl.java:100)
>
>         at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> init(SqlEntityProcessor.java:53)
>
>         at org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(
> EntityProcessorWrapper.java:75)
>
>         at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:433)
>
>         at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:414)
>
>         ... 5 more
>
> Caused by: java.lang.IllegalStateException: Bad password, algorithm, mode
> or padding; no salt, wrong number of iterations or corrupted ciphertext.
>
>         at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:249)
>
>         at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:195)
>
>         at org.apache.solr.handler.dataimport.JdbcDataSource.
> decryptPwd(JdbcDataSource.java:129)
>
>         ... 12 more
>
> Caused by: javax.crypto.BadPaddingException: Given final block not
> properly padded
>
>         at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:975)
>
>         at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:833)
>
>         at com.sun.crypto.provider.AESCipher.engineDoFinal(
> AESCipher.java:446)
>
>         at javax.crypto.Cipher.doFinal(Cipher.java:2165)
>
>         at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:245)
>
>         ... 14 more
>
>
> 2016-11-03 12:06:20.140 INFO  (Thread-127) [   x:autosuggestfull]
> o.a.s.u.DirectUpdateHandler2 start rollback{}
>
> 2016-11-03 12:06:20.140 INFO  (Thread-127) [   x:autosuggestfull]
> o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter...
> core=autosuggestfull
>
> 2016-11-03 12:06:20.154 INFO  (Thread-127) [   x:autosuggestfull]
> o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
>
>
>
> On Wed, Nov 2, 2016 at 12:21 PM, Jamie Jackson <[hidden email]>
> wrote:
>
>> I'm at a brick wall. Here's the latest status:
>>
>> Here are some sample commands that I'm using:
>>
>> *Create the encryptKeyFile and encrypted password:*
>>
>>
>> encrypter_password='this_is_my_encrypter_password'
>> plain_db_pw='Oakton153'
>>
>> cd /var/docker/solr_stage2/credentials/
>> echo -n "${encrypter_password}" > encpwd.txt
>> echo -n "${plain_db_pwd}" > plaindbpwd.txt
>> openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
>> "${encrypter_password}"
>>
>> rm plaindbpwd.txt
>>
>> That generated this as the password, by the way:
>>
>> U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
>>
>> *Configure DIH configuration:*
>>
>> <dataConfig>
>>
>> <dataSource
>> driver="org.mariadb.jdbc.Driver"
>> url="jdbc:mysql://local.mysite.com:3306/mysite"
>> user="root"
>> password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
>> encryptKeyFile="/opt/solr/credentials/encpwd.txt"
>> />
>> ...
>>
>>
>> By the way, /var/docker/solr_stage2/credentials/ is mapped to
>> /opt/solr/credentials/ in the docker container, so that's why the paths
>> *seem* different (but aren't, really).
>>
>>
>> *Authentication error when data import is run:*
>>
>> Exception while processing: question document :
>> SolrInputDocument(fields:
>> []):org.apache.solr.handler.dataimport.DataImportHandlerException:
>> Unable to execute query:     select     'foo' as bar;    Processing
>> Document # 1
>>         at org.apache.solr.handler.dataimport.DataImportHandlerExceptio
>> n.wrapAndThrow(DataImportHandlerException.java:69)
>>         at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetI
>> terator.<init>(JdbcDataSource.java:323)
>>         at org.apache.solr.handler.dataimport.JdbcDataSource.getData(
>> JdbcDataSource.java:283)
>>         at org.apache.solr.handler.dataimport.JdbcDataSource.getData(
>> JdbcDataSource.java:52)
>>         at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQu
>> ery(SqlEntityProcessor.java:59)
>>         at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRo
>> w(SqlEntityProcessor.java:73)
>>         at org.apache.solr.handler.dataimport.EntityProcessorWrapper.ne
>> xtRow(EntityProcessorWrapper.java:244)
>>         at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(
>> DocBuilder.java:475)
>>         at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(
>> DocBuilder.java:414)
>>         at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(
>> DocBuilder.java:329)
>>         at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBui
>> lder.java:232)
>>         at org.apache.solr.handler.dataimport.DataImporter.doFullImport
>> (DataImporter.java:416)
>>         at org.apache.solr.handler.dataimport.DataImporter.runCmd(
>> DataImporter.java:480)
>>         at org.apache.solr.handler.dataimport.DataImporter$1.run(DataIm
>> porter.java:461)
>> Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not
>> connect: Access denied for user 'root'@'ICZ2002912' (using password:
>> NO)
>>         at org.mariadb.jdbc.internal.util.ExceptionMapper.get(Exception
>> Mapper.java:123)
>>         at org.mariadb.jdbc.internal.util.ExceptionMapper.throwExceptio
>> n(ExceptionMapper.java:71)
>>         at org.mariadb.jdbc.Driver.connect(Driver.java:109)
>>         at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(
>> JdbcDataSource.java:192)
>>         at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(
>> JdbcDataSource.java:172)
>>         at org.apache.solr.handler.dataimport.JdbcDataSource.getConnect
>> ion(JdbcDataSource.java:503)
>>         at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetI
>> terator.<init>(JdbcDataSource.java:313)
>>         ... 12 more
>> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could
>> not connect: Access denied for user 'root'@'ICZ2002912' (using
>> password: NO)
>>         at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.a
>> uthentication(AbstractConnectProtocol.java:524)
>>         at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.h
>> andleConnectionPhases(AbstractConnectProtocol.java:472)
>>         at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.c
>> onnect(AbstractConnectProtocol.java:374)
>>         at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.c
>> onnectWithoutProxy(AbstractConnectProtocol.java:763)
>>         at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.
>> java:469)
>>         at org.mariadb.jdbc.Driver.connect(Driver.java:104)
>>         ... 16 more
>>
>>
>>
>> On Thu, Oct 6, 2016 at 2:42 PM, Jamie Jackson <[hidden email]>
>> wrote:
>>
>> > It happens to be ten characters.
>> >
>> > On Thu, Oct 6, 2016 at 12:44 PM, Alexandre Rafalovitch <
>> [hidden email]
>> > > wrote:
>> >
>> >> How long is the encryption key (file content)? Because the code I am
>> >> looking at seems to expect it to be at most 100 characters.
>> >>
>> >> Regards,
>> >>    Alex.
>> >> ----
>> >> Newsletter and resources for Solr beginners and intermediates:
>> >> http://www.solr-start.com/
>> >>
>> >>
>> >> On 6 October 2016 at 23:26, Kevin Risden <[hidden email]>
>> >> wrote:
>> >> > I haven't tried this but is it possible there is a new line at the
>> end
>> >> in
>> >> > the file?
>> >> >
>> >> > If you did something like echo "" > file.txt then there would be a
>> new
>> >> > line. Use echo -n "" > file.txt
>> >> >
>> >> > Also you should be able to check how many characters are in the file.
>> >> >
>> >> > Kevin Risden
>> >> >
>> >> > On Wed, Oct 5, 2016 at 5:00 PM, Jamie Jackson <[hidden email]>
>> >> wrote:
>> >> >
>> >> >> Hi Folks,
>> >> >>
>> >> >> (Using Solr 5.5.3.)
>> >> >>
>> >> >> As far as I know, the only place where encrypted password use is
>> >> documented
>> >> >> is in
>> >> >> https://cwiki.apache.org/confluence/display/solr/
>> >> >> Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler,
>> >> >> under the "Configuring the DIH Configuration File", in a comment in
>> the
>> >> >> sample XML file:
>> >> >>
>> >> >> <!--
>> >> >> Alternately the password can be encrypted as follows. This is the
>> value
>> >> >> obtained as a result of the command
>> >> >> openssl enc -aes-128-cbc -a -salt -in pwd.txt
>> >> >> password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o="
>> >> >> WHen the password is encrypted, you must provide an extra attribute
>> >> >> encryptKeyFile="/location/of/encryptionkey"
>> >> >> This file should a text file with a single line containing the
>> >> >> encrypt/decrypt password
>> >> >> -->
>> >> >>
>> >> >> Anyway, I can encrypt just fine:
>> >> >>
>> >> >> $ openssl enc -aes-128-cbc -a -salt -in stgps.txt
>> >> >> enter aes-128-cbc encryption password:
>> >> >> Verifying - enter aes-128-cbc encryption password:
>> >> >> U2FsdGVkX1+VtVoQtmEREvB5qZjn3131+N4jRXmjyIY=
>> >> >>
>> >> >>
>> >> >> I can also decrypt just fine from the command line.
>> >> >>
>> >> >> However, if I use the encrypted password and encryptKeyFile in the
>> >> config
>> >> >> file, I end up with an error: "String length must be a multiple of
>> >> four."
>> >> >>
>> >> >> https://gist.github.com/jamiejackson/3852dacb03432328ea187d4
>> 3ade5e4d9
>> >> >>
>> >> >> How do I get this working?
>> >> >>
>> >> >> Thanks,
>> >> >> Jamie
>> >> >>
>> >>
>> >
>> >
>>
>
>
>
> --
> Bill Bell
> [hidden email]
> cell 720-256-8076
>



--
Bill Bell
[hidden email]
cell 720-256-8076